Object recognition of building elements

Abstract

A computer-implemented method for identifying individual building elements and / or piles comprising a plurality of individual building elements, whereby the individual building elements in particular are intended for the construction of a building structure made from several of the individual building elements, whereby the method comprises the steps of: A1 Taking an image of an individual building element and / or a pile to be identified; A2 Performing an image analysis of the image to extract a shape of the building element and / or the pile in the image; A3 Optionally, comparison of the extracted shape with one or more predefined shapes, which are defined by predefined data models of building elements and / or piles; A4 Optionally, evaluating whether the extracted shape corresponds to one of the predefined shapes and, optionally, obtaining a deviation between the extracted shape and the corresponding predefined shape; A5 Optionally, repeating the steps with a next building element or pile to be identified.

Description

[0001] Object recognition of building elements

[0002] Technical field

[0003] The invention relates to a method for identifying individual building elements and / or piles comprising a plurality of individual building elements, whereby the individual building elements in particular are intended for the construction of a building structure made from several of the individual building elements. Further aspects of the invention are directed to producing individual building elements and / or piles comprising a plurality of individual building elements, as well as to a method for producing a building structure, especially a 3D reinforcement and / or formwork structure, from multiple individual building elements with a robotic system. A still further aspect of the invention is directed to a system that is configured for performing one or more of these methods.

[0004] Prior art

[0005] In the dynamic landscape of modern construction, the integration of automation has ushered in a new era of efficiency, precision, and sustainability. Automated building processes represent a paradigm shift in the way structures are conceived, designed, and erected. This transformative approach leverages cutting-edge technologies to streamline construction workflows, minimize errors, and enhance overall project outcomes.

[0006] Automation in building processes involves the use of advanced robotics and digital technologies to perform tasks traditionally carried out by human labor. This encompasses a wide array of activities, including site preparation, material handling, assembly, and even project management. The aim is not only to expedite the construction timeline but also to elevate the quality of the built environment.

[0007] However, as with any transformative technology, the adoption of automated building processes comes with its own set of challenges. In particular, when increasing automation in construction processes, there is typically an increased need for automatic identification, verification, and locating of construction elements and materials.

[0008] In this regards, for example, the article "An Image-Based Steel Rebar Size Estimation and Counting Method Using a Convolutional Neural Network Combined with Homography" (Buildings 2021, 11(10), 463) describes an automated system to estimate the size of, and count, steel rebars in bale packing using computer vision techniques based on a convolutional neural network (CNN).

[0009] As a further example, the article "Depth-camera-based rebar detection and digital reconstruction for robotic concrete spraying" (Construction Robotics 2021, 5, 191) describes accurate digital reconstruction of steel rebar meshes using a set of industrial depth cameras. A construction example under investigation in this paper is robotic concrete spraying, where material is sprayed onto double-curved single layered rebar meshes. Before the spraying process can start, the location and geometry of the rebar mesh needs to be accurately know. We present an automatic image-based processing approach of depth images for grid point extraction at an accuracy of a few mm.

[0010] However, there is still a need for improved solutions that allow for further improving automatic building processes.

[0011] Disclosure of the invention

[0012] It is an object of the present invention to provide solutions for simplifying and / or improving automatic building processes. Especially, the solution should allow for automatic identification, verification, locating, placing and / or assembling of building elements. Also, preferably, the solution should enable efficient, precise and / or safe methods for producing building elements and building structures.

[0013] Surprisingly it was found that these objects can be achieved by the features of claim 1. Especially preferred implementations and embodiments are the subject of the further claims and are set out in the following description.

[0014] A first aspect, the present invention is directed to a computer-implemented method for identifying individual building elements and / or piles comprising a plurality of individual building elements, whereby the individual building elements in particular are intended for the construction of a building structure made from several of the individual building elements. The method comprises the steps of:

[0015] A1 Taking an image of an individual building element and / or a pile to be identified;

[0016] A2 Performing an image analysis of the image to extract a shape of the building element and / or the pile in the image;

[0017] A3 Optionally, comparison of the extracted shape with one or more predefined shapes, which are defined by predefined data models of building elements and / or piles;

[0018] A4 Optionally, evaluating whether the extracted shape corresponds to one of the predefined shapes and, optionally, obtaining a deviation between the extracted shape and the corresponding predefined shape;

[0019] A5 Optionally, repeating the steps with a next building element or pile to be identified

[0020] In contrast to other methods known from prior art, which are e.g. based on identification of tags attached to the building elements or the determination of isolated parameters such as diameters of rebars, the inventive method is based on shape recognition, which greatly improves efficiency, precision, reliability and possible applications.

[0021] Especially, the inventive method allows for identifying building elements or piles of such elements automatically in a very efficient, precise and reliable manner. Specifically, the shape recognition can be implemented such that the building element or piles of a plurality of building elements can be recognized essentially independently of the orientation of these items. Therefore, the elements do not need to be arranged in a specific manner and / or at a specific location.

[0022] Also, since the shape of the elements or piles is determined, gripping positions for robot-assisted further processing or any other position relevant for the processing of the building element or piles can be assigned directly in the shape recognition process or later on.

[0023] The inventive method enables many and very different applications, such as e.g.: identification of building elements or piles thereof at production or construction sites, improved automatic production processes for individual building elements with integrated quality control, as well as improved production processes of building structures made from building elements with less manual processing steps.

[0024] In the present context a "building element" in particular is meant to be a solid and shaped element with a defined structure that is intended for building purposes, e.g. for construction of a building structure.

[0025] The inventive method can be implemented with essentially any kind of building elements or piles thereof. For example, the building elements are reinforcement elements, support elements, beams, bricks, plates, panels, tiles, boards, and / or precast elements. For example, the elements can be made of metal, synthetic material, timber, mineral material, ceramics, mortar, concrete, gypsum, stone and / or composite materials. Especially, the building element is a reinforcement element, especially a reinforcement element for mortar or concrete structures. In particular, the reinforcement element is made from metal and / or synthetic material, especially from steel.

[0026] In particular, the building element is made of metallic wire. Especially, the building element is a straight or a non-straight element, especially a 2D structure made of metallic wire with at least one curvature and / or kink. Such kind of elements can e.g. be used for producing reinforced mortar or concrete structures, especially as components of 3D grid formwork and / or 3D mesh structures that are used as reinforcement and / or formwork in mortar or concrete structures.

[0027] Also the building element, especially the individual reinforcing element, can be a multicomponent structure, especially made from several interconnected components, e.g. wires. For example, the multi-component structure is a mat-like structure.

[0028] The building structure can in principle be any building structure. For example the building structure is a reinforcement structure, a support structure, an element of a building, in particular a mortar or concrete structure.

[0029] In particular, the building structure is a mesh structure for use in architecture, engineering and / or construction, in particular for use in reinforcement and / or formwork systems.

[0030] Especially the building structure is a three-dimensional (3D) reinforcement and / or formwork structure, especially for a mortar or concrete structure. E.g. the 3D reinforcement and / or formwork structure is a 3D mesh structure, especially made of steel wires.

[0031] For example, the 3D dimensional reinforcement and / or formwork structure is a grid structure that has a central region between a first grid side and a second grid side, the second grid side being arranged opposite the first grid side. In particular, the two grid sides are designed as two interconnected grids. In particular, the first grid side and the second grid side are connected to each other at a distance from each other by a plurality of bridging elements and / or webs. In an exemplary embodiment, first grid side and / or the second grid side is / are planar, curved and / or substantially uneven.

[0032] In an exemplary embodiment, a distance between the two grids is between 5 and 100 cm, preferably between 10 and 80 cm, particularly preferably between 15 and 50 cm.

[0033] Especially, bars forming the grids are aligned substantially horizontally and vertically. In an alternative embodiment, the bars of the grids are oriented substantially diagonally to a horizontal and a vertical direction

[0034] The three-dimensional reinforcement and / or formwork structure can e.g. be produced by providing a variety of building elements in the form of shaped wires, especially bent wires, whereby shaped wires may comprise different shapes, curvatures, and / or extensions. The shaped wires can be assembled, especially joined, e.g. via further building elements, especially connecting wires.

[0035] A pile comprises at least two, especially three, five, ten or more, building elements. Especially, the pile comprises at least two different building elements, differing in particular with respect to their size and / or shape. However, in another embodiment, the pile comprises only one type of building elements, which are identical in design.

[0036] In step A1, an image is taken. This can be done by any kind of image-generating device.

[0037] Especially, the image is taken by a camera and / or a laser scanner. A camera in particular is a camera for capturing images in the visible light spectrum. However, cameras operating in other wavelength regions may be used as well. The image may also be taken by a 3D camera, in particular embodied as 1) a stereo camera, or as 2) a triangulation system having (i) an active light source projecting a predefined pattern onto an object of interest and (ii) at least one camera, or as 3) a time-of-flight camera. In case the image is taken by a stereo camera, in fact two images may be taken from which three-dimensional information may be deduced.

[0038] Especially, the laser scanner may be a 1 D, 2D and / or 3D laser scanner. In particular, for elements with simpler geometrical structures, e.g. essentially one-dimensional elements and / or essentially two-dimensional elements, it may be sufficient to use 1 D scanners.

[0039] Also, two or more laser scanners and / or cameras may be combined to a laser scanner and / or camera array. Furthermore, 1 D and / or 2D laser scanners may be mounted on a robotic system that preferably is movable in one, two or three spatial directions. In this manner, even elements with complex 3D structures can be precisely mapped with 1 D and / or 2D scanners.

[0040] For example, the image-generating device is an image-and-distance sensor, especially an RGB-D sensor. An RGB-D sensor is meant to be a depth camera that provides both depth (D) and color (RGB) data as the output. However, other sensors are possible as well, e.g. a camera in combination with another range sensor, such as e.g. an ultrasound distance sensor.

[0041] The image-generating device can be located at a fixed position above a picking area. According to another possible embodiment, the image-generating device is mounted on a robotic system that preferably is movable in one, two or three spatial directions.

[0042] The image obtained in step A1 in particular is a 2D image of the individual building element and / or a pile in the visible spectrum. The image obtained in step A1 may, however, also be a 3D image of the individual building element and / or a pile in the visible spectrum, which 3D image may, for example, correspond to a pair of images acquired by a stereo camera with a defined geometric relationship between the pair of images. In step A2, an image analysis of the image is performed to extract a shape of the building element and / or the pile in the image. "Image analysis" is a technique known per se. Thereby, information is extracted from images by means of digital image processing techniques. The image analysis can e.g. be implemented as 2D or 3D object recognition and / or image segmentation. Using images obtained through a stereo camera, for example, a three-dimensional shape of the building element and / or the pile may be extracted.

[0043] For example, in a first step, an algorithm determines which parts of the image are background and which parts belong to the individual building element and / or pile to be identified. The resulting contour can be used to locate the element or pile in the image and to derive the shape.

[0044] In the present context, the term "shape", as mentioned for example in step A2, in particular stands for a graphical representation of an object's form or its external boundary, outline, and / or external surface. The shape can be a two-dimensional (2D) shape, being represented in 2D space; or a 3D shape, i.e. being represented in a 3D space.

[0045] The shape of the building element and / or the pile in the image may also be determined using machine-learning methods which may in particular be useful in case of the image taken in step A1 is taken in 1) noisy imaging conditions, e.g. due to dim lighting, or in 2) uneven imaging conditions, with part of the image being brighter than other parts, or 3) in case only part of the building element and / or the pile is visible in the taken image, e.g. due to occlusions / obstacles and / or due to position and viewing direction of the camera resp. 3D camera resp. laser scanner with respect to the building element and / or the pile.

[0046] A machine-learning model may be trained based on images taken in one or more of such imperfect imaging conditions for which at the same time a corresponding ground- truth shape is available, with the machine-learning model being trained to map the taken image onto the corresponding ground-truth shape. The ground-truth shape used for training the machine-learning model may also be part of the one or more predefined shapes of optional step A3. In case the imaged building element and / or pile is produced according to a pre-set ground truth using a construction machine, in particular embodied as a bending and / or cutting device, due to machining imperfections, the produced building element and / or pile may deviate from the pre-set ground truth. To train the machine-learning model accurately, the pre-set ground truth may therefore first be adapted to match the actually produced building element and / or pile, and this adapted pre-set ground truth may be used as ground-truth shape for training the machine-learning model.

[0047] The machine-learning model may alternatively also be trained only to correct, i.e. pre- process the taken image, and from the pre-processed image, the shape of the building element and / or the pile may then be extracted using available algorithms. The preprocessing may in particular be used to improve signal-to-noise ratio and / or restore homogeneous brightness throughout the image and / or interpolate missing segments in the image.

[0048] The construction machine that may be used for producing the building element and / or the pile may abstractly be considered to possess N actuating variables, with N being a positive natural number, wherein by changing any one of these N actuating variables, the produced output of the construction machine potentially may be changed. When producing the building element and / or the pile based on the pre-set ground truth, e.g. defined as a predefined data model of the building element and / or the pile, the N actuating variables may be nominally set to a first set of values. Due to natural drift in the construction machine configuration over time and / or due to a faultily chosen first set of values, the produced building element and / or pile may differ from the building element and / or pile which was intended to be produced according to the pre-set ground truth. For producing the intended building element and / or pile, it may be assumed that instead of the first set of values, the N actuating variables should have been set to a second set of values.

[0049] The machine-learning model may therefore be additionally trained to provide as output - besides the extracted shape - N correction values which encode how the first set of values is to be changed to produce a building element and / or pile that is closer to the desired pre-set ground truth than the actually produced building element and / or pile. To achieve this, the machine-learning model may be provided during training additionally with the first set of values as input and with the difference between the first set of values and the second set of values as output. Later during inference, i.e. after the training of the machine-learning model, the machine-learning model may be provided with the following input: the taken image (2D or 3D) and the nominally set first set of values; as output, the machine-learning model may then provide the extracted shape and the N correction values.

[0050] The N correction values may either be applied automatically in the construction machine using dedicated machinery within and / or outside the construction machine, or the N correction values may be manually applied to the construction machine by a machine operator.

[0051] In case it is determined that the building element and / or the pile produced by the construction machine does not correspond to the pre-set ground truth, the produced building element and / or pile may still be used for a subsequent construction, however: for this to be possible, building instructions may need to be adapted, i.e. construction steps, in particular subsequent construction steps following the construction steps in which the produced building element and / or pile is used, and further construction components that interact with the produced building element and / or pile may be adapted to the actually produced building element and / or pile. This way, the faultily produced building element and / or pile may still be used in construction. As it turned out, in the field of building technology the shape characterizes a huge variety of different building elements in a highly precise manner. Therefore, building elements and / or piles thereof can be identified in a highly reliable manner.

[0052] According to a special embodiment, steps A1 and A2 are repeated, in particular for a predefined period of time, e.g. for 1 - 200 seconds, and / or until an extracted shape in step A4 corresponds to one of the predefined shapes. The repetition of steps A1 and A2 can take place for example with frequency of at least 0.1 Hz, especially at least 1 Hz, in particular at least 3 Hz, or at least 10 Hz. This allows for a faster recognition of the building element or pile to be identified since the identification process works even when the element or pile to be identified is moved into the picking area and / or of the image-generating device is moved.

[0053] Especially, in optional step A3 of the inventive method, the extracted shape is compared with one or more predefined shapes, which are defined by predefined data models of building elements and / or piles. This can e.g. be realized by providing a database comprising a plurality of different predefined data models of building elements and / or piles. Such data models can for example be generated by computer aided design.

[0054] Especially, the comparison of the shapes takes into account geometrical similarity. This allows for example to decide whether certain building elements or piles thereof are present at a construction site and / or to assign gripping positions for robot-assisted further processing.

[0055] However, in still a further preferred embodiment, geometrical similarity as well as absolute dimensions are taken into account. In this manner, it is for example possible to check whether the building elements have a desired shape as well as to check whether their absolute dimensions are within a given tolerance. Absolute dimensions in the images obtained in step A2 can e.g. be obtained via calibration, reference objects, reference markers, and / or combination of the image taking process with distance measurements, e.g. via a range sensor, especially a RGB-D sensor, or using a 3D camera.

[0056] For example, in a picking area, reference markers, such as e.g. rulers, symbols identification tags can be placed, especially at predefined positions and / or distances. These reference markers then can be recognized in step A2 for determining absolute dimensions.

[0057] Subsequently, in optional step A4, it is possible to evaluate whether the extracted shape corresponds to one of the predefined shapes. Optionally, a deviation between the extracted shape and the corresponding predefined shape can be obtained. Thereby, the deviation can be determined with respect to the geometrical similarity and / or absolute dimensions.

[0058] Techniques for comparing shapes are known per se and can be implemented in various manner.

[0059] The steps A2 to A4 may optionally also be combined in the previously discussed machine-learning model, i.e. a joint machine-learning model may also be trained end- to-end to directly classify the image taken in step A1. Alternatively, two separate machine-learning models may be used, with the previously discussed machine-learning model providing extracted shape(s) and a further machine-learning model taking the extracted shape(s) as input and providing a same type of output as the joint machinelearning model. In case M predefined shapes are available, M being a positive natural number, the output of the joint machine-learning model or of the further machinelearning model may be an M-dimensional vector, with each element of the M- dimensional vector being populated with a natural number (including '0') indicating how often the corresponding predefined shape is in the image. In addition, for each identified shape in the image, the joint machine-learning model or the further machinelearning model may be trained to output a gripping position that in turn may be used for informing a robot where to grip the building element and / or pile that was imaged and identified in the taken image. The gripping position may specifically take into account deformations of the imaged building element and / or pile, for example due to the action of gravity on the imaged building element and / or pile. Similarly to the machine-learning model, the joint machine-learning model may also output N correction values.

[0060] In case the imaged building element and / or pile comprises varying texture on its outside and in case symmetry of the imaged building element and / or pile does not allow resolving precisely how the imaged building element and / or pile is placed (in case of the building element for example being a regularly spaced two-dimensional grid, after a rotation by 180 degrees around a rotation axis passing through a center of the two-dimensional grid and lying in a same plane as the two-dimensional grid, the shape of the two-dimensional grid would remain unchanged if viewed from a fixed camera position), the further machine-learning model or the joint machine-learning model may be trained to take the varying texture into account: the further machinelearning model or the joint machine-learning model may therefore also output an information indicating which part of the varying texture is visible in the taken image. In case the building element is essentially two-dimensional, for example, and only one side comprises a specific texture, the further machine-learning model or the joint machine-learning model may provide as output an indication of which of the two sides of the building element is better visible in the taken image.

[0061] In case the building element is embodied as a reinforcing bar, for example, it may comprise ridges / ribs on one of its sides, which ridges / ribs provide texture and which texture may be used for inferring which side of the reinforcing bar faces the camera resp. 3D camera resp. laser scanner. According to a further preferred embodiment, the method further comprises a step of identifying all individual building elements and / or piles in a predefined area. The area can e.g. be a storage area, especially the storage area of a construction site. Thereby, preferably, for each type of the identified individual building elements and / or piles in the predefined area the number of the identified individual building elements and / or piles is determined.

[0062] In an optional further step, it is checked whether the determined number equals a predefined number of required individual building elements and / or piles. The predefined number can e.g. be a derived from a construction plan of a building structure to be produced with the individual building elements and / or piles, from a delivery note and / or stock information.

[0063] In this manner it can in particular be checked if all of the individual building elements and / or piles required for a certain construction work are present at a job site and / or if all of the items ordered were delivered.

[0064] Further preferred, the method further comprises the step of classifying the individual building element or the pile in the image, especially depending on the comparison of step A3 and / or the evaluation of step A4.

[0065] For example, the step of classifying the individual building element or the pile in the image is performed as follows: if the extracted shape in step A3 corresponds to a predefined shape and / or the deviation is below a predefined threshold, the building element or the pile in the image is assigned to a first class, especially a class of matching elements; if the extracted shape in step A3 does not correspond to a predefined shape and / or the deviation is equal to or above a predefined threshold, the building element or the pile in the image is assigned to a second class, especially a class of nonmatching elements.

[0066] Depending on the class attributed to the individual building element or the pile, it can be decided how to further proceed with respect to this element or pile. For example, if the individual building element or the pile is classified as non-matching, it can be moved and placed to a disposing area or simply be left at the initial position and be ignored in a later process. An individual building element or pile that is classified as matching can for example be subjected to further process steps.

[0067] However, different classifications might be possible as well, e.g. comprising more than two classes, e.g. three or more classes. Thereby, for example, more than one threshold can be defined allowing for example to classify the individual building elements or piles "highly precise", "sufficiently precise" and "unsuitable". This allow for example to use highly precise building elements for the construction of building structures with low tolerances whereas the sufficiently precise building elements can be used for the construction of building structures with higher tolerances.

[0068] According to a further preferred embodiment, the inventive method further comprises the step of assigning a planned gripping position in the image, whereby the gripping position defines a position at which the building element or pile is to be gripped with a gripping device for the purpose of movement of the building element or pile. The assigning of the planned gripping position in particular is based on corresponding information in the predefined data models of the building elements and / or piles and / or in a construction plan of a building structure to be produced.

[0069] For example, the predefined data models of the building elements and / or piles may comprise gripping positions for each type of building element or pile for a given gripping device. Thereby, the predefined data models may comprise several sets of gripping positions for different types of gripping devices. As previously discussed, gripping positions may also be determined by the joint machine-learning model or the further machine-learning model, which determined gripping positions may take into account deviations in form, e.g. due to gravity, of a produced building element and / or pile from an ideal shape; the gripping positions determined by the joint machine-learning model or the further machine-learning model may be determined for a specific gripping device.

[0070] Gripping positions may also be determined based on the extracted shape without the use of the joint machine-learning model or of the further machine-learning model. In particular, the extracted shape may correspond to a deformed building element and / or pile, i.e. the imaged building element and / or pile of which the shape is extracted may be deformed compared to the predefined data model. Based on the extracted shape, the gripping position for a building element or pile for a given gripping device may be adapted.

[0071] This allows for picking up the building element or pile in a next step with a gripping device at the assigned gripping position and placing it at a specific location, especially in a predefined storage area, in a disposal area or in a processing area, and / or changing its orientation in space. By changing its orientation in space, building elements or piles e.g. can be oriented in a specific manner that facilitates further processing or allows for a more compact and / or ordered storage.

[0072] In an exemplary embodiment, the step of picking up the building element or pile can be performed as follows: if the extracted shape in step A3 corresponds to a predefined shape and / or the deviation is below a predefined threshold, the building element or pile is moved to a predefined storage area or a processing area; if the extracted shape in step A3 does not corresponds to a predefined shape and / or the deviation is equal to or above a predefined threshold, the building element or pile is left untouched or moved to a disposal area:

[0073] Thus, undesired building elements and / or piles can be separated from others and be sorted out.

[0074] According to another exemplary embodiment, in step A1 a pile of individual building elements is provided and an image of the pile is taken; and in step A2 the shape of the topmost building element of the pile of the is extracted; and, optionally, after step A4, the topmost building element is removed from the pile and placed in a storage area, especially such that the building elements in the storage area do not lie on top of each other or that only building elements of the same type do lie on top of each other.

[0075] Also, it is for example possible that after step A4, a building element and / or pile corresponding to a predefined shape is placed at a predefined storage area that is assigned to the respective predefined shape, in particular such that building elements and / or piles of the same shape are placed at the same predefined storage area in each case, whereas building elements and / or piles of different shape are placed at different predefined storage areas.

[0076] These measures allows for sorting of the building elements and / or piles, for example making further processing easier.

[0077] A further aspect of the present invention is directed to a method for producing individual building elements, which in particular are used for the construction of building structures made from several of the individual building elements, the method comprising the steps of:

[0078] B1 In a manufacturing process, especially a bending and / or cutting process, producing an individual building element or pile that is expected to have a predefined shape; B2 Performing the method as described above with the individual building element or the pile of step B1 as the building element or pile to be identified and the predefined shape of step B1 as the only predefined shape in steps A3 and A4;

[0079] B3 Optionally, depending on the outcome of step B2, (i) providing the building element for further use, e.g. by placing it in a specific storage area or by feeding it into a processing area, or (ii) disposing the building element, e.g. by placing it in a disposal area;

[0080] B4 Optionally, repeating steps B1 to B3.

[0081] Thanks to the inventive method, the manufacturing process can directly be controlled in a very efficient manner. Especially, with this approach each individual building element or piles can be checked during the manufacturing process. Put differently, a quality control on the level of individual building element or piles can be implemented without need of modification or slowing down the manufacturing process. This makes it possible that only products that fulfill certain requirements are provided for further use, whereas other products are sorted out. In step B2, the previously discussed machinelearning model and / or further machine-learning model or the joint machine-learning model may be employed. In case the further machine-learning model or the joint machine-learning model is employed, it may be specifically verified whether the output of the further machine-learning model or of the joint machine-learning model indicates that the predefined shape of step B1 is successfully produced.

[0082] Also, in case of problems during the manufacturing, process parameters can be adjusted and / or the process can immediately be stopped in case maintenance or repair work is required.

[0083] In an exemplary embodiment, (i) the building element is provided for further use if the extracted shape in step A4 corresponds to the predefined shape and / or the deviation is below a predefined threshold and the building element is disposed if the extracted shape in step A4 does not corresponds to the predefined shape and / or the deviation is equal to or above a predefined threshold. In a further specific embodiment, the building element may be gripped at its determined gripping position, the gripping position in particular being determined by the further machine-learning algorithm or the joint machine-learning algorithm, by a first robot and moved to a moving palette from which it may be picked up by a second robot and moved to a storage place or to a transport palette, for example.

[0084] Further, the method may comprise the step of (i) adjusting one or more manufacturing parameters based on the deviation obtained in step A4 and (ii) in step B4, repeating steps B1 to B3 with the adjusted parameters. Th, especially if the extracted shape in step A4 corresponds to the predefined shape and / or the deviation is below a predefined threshold.

[0085] However, other implementations are possible as well.

[0086] In particular, in the manufacturing process, an individual reinforcing element or a pile of such elements is produced. The reinforcing element in particular is made from metal wires, especially in a bending and / or cutting process. Especially the building element is straight or a non-straight element, in particular a 2D structure made of metallic wire with at least one curvature.

[0087] Especially, it is possible that in the manufacturing process a pile of prefabricated building elements is produced. In this case, the individual building elements can e.g. be produced before and / or in another process. Thereby, the above-described manufacturing process in particular is a process of piling.

[0088] A still further aspect of the present invention is directed to a computer-implemented method for producing a building structure, especially a 3D reinforcement and / or formwork structure, from multiple individual building elements, preferably with a robotic system, the method comprising the steps of: C1 Selecting a desired shape of an individual building element which is to be installed next, especially based on a digital construction plan of the building structure, especially the 3D reinforcement and / or formwork structure, whereby the desired shape is defined by a data model of the individual building element;

[0089] C2 Producing or providing an individual building element that is expected to have a shape corresponding to the desired shape;

[0090] C3 Performing the method for identifying individual building elements and / or piles as described above with the individual building element of step C2 as the building element to be identified and the desired shape as the only predefined shape in steps A3 and A4;

[0091] C4 If the extracted shape in step A4 corresponds to the predefined shape and / or the deviation is below a predefined threshold, assigning a planned gripping position in the image, especially based on corresponding information in the digital construction plan of the building structure; otherwise, preferably, stop the production process or dispose the building element and start again with step C2;

[0092] C5 Picking up the individual building element with a gripping device, especially of the robotic system, at the planned gripping position,

[0093] C6 Placing the individual building element at a desired position and, preferably, assembling and / or joining it with a support element, a further element of the building structure and / or a further individual building element;

[0094] C7 Optionally, repeating steps C1 to C6 until the building structure, especially the 3D reinforcement and / or formwork structure, is fully built.

[0095] This method allows for a highly automated and efficient production of building structures, especially 3D reinforcement and / or formwork structures. Thereby, each individual building element can be can be checked to see whether it fulfils the desired requirements and / or tolerances. This will greatly enhance quality of the overall production process and the final building structure.

[0096] In particular, the building elements used and the building structure produced in this method are the same as describe above with the method for identifying individual building elements and / or piles and the method for producing individual building elements and / or a piles.

[0097] Especially, in the method for producing a building structure, a 3D reinforcement and / or formwork structure is produced, in particular a grid structure as described above. Thereby, preferably, a variety of straight and / or a non-straight elements, especially comprising 2D structures made of metallic wire with at least one curvature, are assembled and / or joined together.

[0098] For example, a variety of building elements in the form of shaped wires in the form of non-straight elements, especially bent wires, in particular shaped wires that comprise different shapes, curvatures, and / or extensions, are placed at defined positions, e.g. on a support, and then joined via further building elements, especially connecting wires.

[0099] In particular, in step C2, the individual building element is produced by the manufacturing method as described above.

[0100] Especially, assembling and / or joining in step C6 is effected through material, form-fit and / or force-fit connection.

[0101] For example, assembling and / or joining in step C6 is effected by welding. This is a highly efficient method in case of weldable building elements such as e.g. metallic wires or rebars. However, other assembling and / or joining techniques, such as e.g. riveting, interlocking and / or adhesively bonding, can be used alternatively or in addition.

[0102] In particular, during and / or after the production of the building structure, especially the 3D reinforcement and / or formwork structure, the shape of the partly built or fully built building structure is determined with a range sensor, especially a RGB-D sensor, which is mounted on a robotic system so that the entire building structure can be scanned with the sensor from different directions.

[0103] Thereby, the robotic system preferably is movable in one, two or three spatial directions. Instead or in addition to an RGB-D sensor, another range sensor can be used, e.g. an ultrasound distance sensor.

[0104] This allows for monitoring the production process in real time and / or to check the quality of the final building structure.

[0105] Specifically, the shape of the partly built or fully built building structure, especially the 3D reinforcement and / or formwork structure, is compared with a predefined data model of the building structure, especially the 3D reinforcement and / or formwork structure, especially to identify deviations between the structure and the model.

[0106] A further aspect of the present invention is directed to a system, especially an Identification system, comprising: an image-generating device, in particular mounted on a robotic system and / or fixed above a picking area; optionally a gripping device, especially a robotic gripping system that is moveable in one, two directions of space; a control unit that is configured to perform the method for identifying individual building elements and / or piles comprising a plurality of individual building elements method as described above.

[0107] Another aspect of the present invention is directed to a system, especially a manufacturing system, comprising: an image-generating device, in particular mounted on a robotic system and / or fixed above a picking area; a manufacturing device, especially a bending and / or cutting device; optionally a gripping device, especially a robotic gripping system that is moveable in one, two directions of space; a control unit that is configured to perform the method for producing individual building elements and / or piles comprising a plurality of individual building elements, as described above.

[0108] Furthermore, the invention in another aspect is directed to a system, especially a production system, comprising: a manufacturing device, especially a bending and / or cutting device: an image-generating device, in particular mounted on a robotic system and / or fixed adjacent the manufacturing device; a gripping device, especially mounted on a robotic gripping system that is moveable in one, two directions of space; optionally a placing and / or assembling device, especially a robotic placing device and / or a welding device; a control unit that is configured to perform the method producing a building structure, especially a 3D reinforcement and / or formwork structure as described above.

[0109] Especially, the robotic placing device is a robotic gripping system that is moveable in one, two directions of space. In particular, the welding device is mounted on the robotic gripping system. Further advantageous embodiments and combinations of features of the invention result from the following detailed description and the totality of the patent claims.

[0110] Brief description of drawings

[0111] The drawings used to explain the embodiment show:

[0112] Fig. 1 A schematic illustration of a method for identifying individual building elements, e.g. reinforcing elements made of shaped steel rebars;

[0113] Fig. 2 A schematic illustration of a method for producing shaped building elements;

[0114] Fig. 3 A photographic image of a bending machine with a rotatable tool head in the course of bending a steel wire to form a building element as e.g. shown in Fig. 2;

[0115] Fig. 4 A schematic illustration of a method for producing a 3D reinforcement structure from multiple individual building elements with a robotic system;

[0116] Fig. 5 A photographic image of an individual reinforcing element in the form of a multi-component structure made from several interconnected steel wires on a picking table;

[0117] Fig. 6 A photographic image of the picking up of the building element of Fig. 5 by a robotic systems having an electromagnetic gripping device;

[0118] Fig. 7 A photographic image of the placing and joining of the building elements of Fig. 5 by the robotic systems on a support element and with previously placed building elements; Fig. 8 A graphical representation of a 3D reinforcement structure that was built by a process as described in connection with Fig. 4 and scanned with a RGB-D sensor mounted on a robotic system.

[0119] In principle, the same parts are given the same reference signs in the figures.

[0120] Ways of carrying out the invention

[0121] Fig. 1 schematically shows a method 10 for identifying individual building elements 1, 1 ', e.g. reinforcing elements made of shaped steel rebars.

[0122] On the left side, the method is shown with a first building element 1. In a first step, 11 an image of the building element 1 placed on a picking table is taken with an optical camera installed above the picking table in a fixed position. In the second step 12, an image analysis is performed whereby an algorithm determines which parts of the image are background and which parts belong to the individual building element 1 to be identified. The resulting contour is used to locate the building element in the image and to derive the shape 1a of the building element 1. In the subsequent step 13, the extracted shape 1a is compared with several predefined shapes 15a, 15b, 15c stored in a database 15. The predefined shapes 15a, 15b, 15c are defined by predefined data models of building elements. In the next step 14, it is evaluated whether the extracted shape 1a corresponds to one of the predefined shapes 15a, 15b, 15c, and a deviation of the spatial dimensions between the extracted shape 1a and the corresponding predefined shape 15a is calculated. In the exemplary case shown on the left side of Fig. 1, the shape 1a corresponds to the predefined shape 15a (indicated by "©") and the deviation A is below a given threshold TH. Thus, building element 1 could be identified and it was confirmed that its size and shape are within the given tolerances.

[0123] On the right side of Fig. 1, the method is shown with a second building element 1 ' that is different in design when compared to building element 1. The method is performed in the same manner as with the building element 1. However, as a result of step 14, it is established that the shape 1a' of the second building element 1 ' does not corresponds to one of the predefined shapes 15a, 15b, or 15c (indicated by "®") and the deviation A between shape 1a' and the predefined shape 15b, which is the most similar predefined shape, is higher than a given threshold TH. Thus, building element 1 ' could not be identified and its size and shape are not within the given tolerances.

[0124] Fig. 2 schematically shows a method 20 for producing shaped building elements 2', e.g. reinforcing elements made of shaped steel rebars, in a manufacturing process.

[0125] In step 21, individual building elements 2' that are expected to have a predefined shape are produced in a bending and cutting process from straight metallic wires 2. Thereby, each of the building elements produced is subjected to an identification process 22, i.e. a process similar to the method shown in Fig. 1. In the identification process 22, the shape and size of the individual building elements 2' are compared with a predefined desired shape 15a and it is evaluated whether the extracted shape corresponds to the desired shape 25a, and a deviation of the spatial dimensions between the extracted shape and the desired shape 25a is calculated.

[0126] Depending on the outcome of step 22, building elements 2'a corresponding to the desired shape 25a and having dimensions within acceptable tolerances are provided for further use, e.g. by placing them in a specific storage area S and building elements 2'b not corresponding to the desired shape 25a or not having dimensions within acceptable tolerances are placed in a disposal area D.

[0127] Optionally, method 20 may further comprise a step 23 of adjusting one or more manufacturing parameters based on the deviation obtained in step 22. This is for example helpful, if the extracted shape in step 22 essentially corresponds to the predefined shape but there is a certain deviation with respect to absolute dimensions that is outside the given tolerances. This will help to improve the quality of the next building elements to be produced. Fig. 3 shows a photographic image of a bending machine 30 with a rotatable tool head 31 in the course of bending a steel wire 2 to form a building element 2' as e.g. shown in Fig- 2.

[0128] Fig. 4 schematically shows a method 40 for producing a 3D reinforcement structure 46 from multiple individual building elements 3, 4 with a robotic system.

[0129] In a first step 41, a desired shape 45b, 45c, of an individual building element which is to be installed next is selected based on a on a digital construction plan 45a of the 3D reinforcement structure to be produced. The digital construction plan 45a for example is stored in a database 45, together with the predefined shapes 45b, 45c of individual building elements required to produce the 3D reinforcement structure.

[0130] In the next step 42, an individual building element, e.g. element 3, that is expected to have a shape corresponding to the desired shape 45b and that is stored in a storage area S together with further building elements 3, 4, is provided in a picking area. Providing the building element 3 can e.g. be done manually and / or with an automatic feeding system, e.g. a robotic feeding system.

[0131] Subsequently, the building element 3 provided is subjected to an identification process 43, i.e. a process similar to the method shown in Fig. 1. In the identification process 43, the shape and size of the individual building element 3 is compared with the desired shape 45b by first taking an image of the building element with a camera and extracting its shape by image analysis. Subsequently, it is evaluated whether the extracted shape corresponds to the desired shape 45b, and, optionally, a deviation of the spatial dimensions between the extracted shape and the desired shape 45b can be calculated.

[0132] If the extracted shape in step 43 corresponds to the predefined shape 45b and the deviation is below a predefined threshold, a planned gripping position of the robotic device is assigned in the image of the individual building element 3, especially based on corresponding information in the digital construction plan of the building structure. Otherwise, if the extracted shape in step 43 does not corresponds to the predefined shape 45b or the deviation is above a predefined threshold, the building element 3 is disposed and the process is started again with step 41.

[0133] In the next step 44, the building element 3 is picked up with a gripping device of a robotic system at the planned gripping position. Then, in the last step 45, the building element 3 is placed at a desired position, which is defined in the construction plan 45a, and joined with further elements of the building structure. Steps 41 to 45 are repeated until the 3D reinforcement structure 46 is fully built.

[0134] Fig. 5 shows a photographic image of an individual reinforcing element 52 in the form of a multi-component structure made from several interconnected steel wires on a picking table 51. Furthermore, the gripping positions 53 for a gripping device are shown as an overlay in the image (diamond-like shapes at the end of the arrows). Fig. 5 is a specific example of a situation as described schematically in connection with steps 42 and 43 of Fig. 4.

[0135] Fig. 6 shows a photographic image of the picking up of the building element 52 of Fig. 5 by a robotic systems 60 having an electromagnetic gripping device 61. Fig. 6 is a specific example of a situation as described schematically in connection with step 44 of Fig- 4.

[0136] Fig. 7 shows a photographic image of the placing and joining of the building element 52 of Fig. 5 by the robotic systems 60 on a support element and with previously placed building elements 52'. Fig. 7 is a specific example of a situation as described schematically in connection with step 45 of Fig. 4. In this case, the building elements 52, 52' after the building process has been completed form a 3D dimensional reinforcement and formwork structure that can be used for producing concrete structure. Fig. 8 shows a representation of a 3D reinforcement structure that was built by a process as described in connection with Fig. 4 and subsequently the shape of the structure was determined with RGB-D sensor that was mounted on a robotic system so that the entire 3D reinforcement structure could be scanned with the sensor from different directions. The so obtained shape can be compared with a predefined data model of the 3D reinforcement structure. In the representation of the 3D reinforcement shown in Fig. 8, deviations from the ideal shape or the predefined data model of the 3D reinforcement structure, respectively, are indicated by different colors.

[0137] While particular embodiments of the invention have been described, it is apparent to one skilled in the art that various variations and alternatives to the details described could be developed in the light of the disclosure as a whole. Accordingly, the particular arrangements and implementations of methods disclosed herein are intended only as an explanation and not as a limitation of the scope of protection.

[0138] For example, in the method of Fig. 1, steps 13 and 14 may be omitted, e.g. if the only data required is the shape of the building element. The shape of the building element can be used as such for manual comparison and / or any other purpose.

[0139] Instead or in addition to individual building elements, piles of building elements can be processed in the method shown in Fig. 1.

[0140] Also, in the method shown in Fig. 4, a further step of identifying all individual building elements in the storage area could be performed before starting the building process. This can be done by an additional an identification process as described in connection with Fig. 1 before starting the production process. Thereby, in particular, it can be checked whether the required types and number of individual building elements are present to finish the building structure as a whole. This information can e.g. be a derived from the construction plan of the building structure to be produced. Also, instead or in addition to individual building elements, piles of building elements can be identified in this manner.

[0141] Furthermore, the step of identifying all individual building elements in the storage area can be implemented independently of a production process. Thereby, individual building elements and / or piles of building elements can be identified in this manner.

[0142] For example, such a method can be used for checking whether all individual building elements and / or piles thereof have been delivered to a construction site.

[0143] Likewise, the methods described in connection with Fig. 1 - 8 can be implemented with other building elements or piles thereof, e.g. elements made from synthetic materials, wood or composite materials.

Claims

Claims1. Computer-implemented method for identifying individual building elements and / or piles comprising a plurality of individual building elements, whereby the individual building elements in particular are intended for the construction of a building structure made from several of the individual building elements, the method comprising the steps of:A1 Taking an image of an individual building element and / or a pile to be identified;A2 Performing an image analysis of the image to extract a shape of the building element and / or the pile in the image;A3 Optionally, comparison of the extracted shape with one or more predefined shapes, which are defined by predefined data models of building elements and / or piles;A4 Optionally, evaluating whether the extracted shape corresponds to one of the predefined shapes and, optionally, obtaining a deviation between the extracted shape and the corresponding predefined shape;A5 Optionally, repeating the steps with a next building element or pile to be identified.

2. The method according to claim 1, whereby step A2 is performed using a machinelearning model, or whereby steps A2 to A4 are performed using a joint machinelearning model, or whereby step A2 is performed using the machine-learning model and steps A3 and A4 are performed using a further machine learning model.

3. The method according to claim 1 or 2, whereby the individual building element is a reinforcement element and / or the pile comprises a plurality of individual reinforcingelements, whereby the individual reinforcing element(s) is / are intended for the construction of a 3D reinforcement and / or formwork structure made from several of the individual reinforcing elements.

4. The method according to any of preceding claims, whereby the method further comprises the step of classifying the individual building element or the pile in the image whereby:- if the extracted shape in step A3 corresponds to a predefined shape and / or the deviation is below a predefined threshold, the building element or the pile in the image is assigned to a first class, especially a class of matching elements;- if the extracted shape in step A3 does not correspond to a predefined shape and / or the deviation is equal to or above a predefined threshold, the building element or the pile in the image is assigned to a second class, especially a class of non-matching elements.

5. The method according to any of preceding claims, whereby the method further comprises the step of assigning a planned gripping position in the image, whereby the gripping position defines a position at which the building element or pile is to be gripped with a gripping device for the purpose of movement of the building element or pile, whereby the assigning of the planned gripping position in particular is based on corresponding information in the predefined data models of the building elements and / or piles.

6. The method according to any of proceeding claims, whereby:- if the extracted shape in step A3 corresponds to a predefined shape and / or the deviation is below a predefined threshold, the building element or pile is moved to a predefined storage area or a processing area;- if the extracted shape in step A3 does not corresponds to a predefined shape and / or the deviation is equal to or above a predefined threshold, the building element or pile is left untouched or moved to a disposal area.

7. The method according to any of preceding claims, whereby after step A4, a building element and / or pile corresponding to a predefined shape is placed at a predefined storage area that is assigned to the respective predefined shape, in particular such that building elements and / or piles of the same shape are placed at the same predefined storage area in each case, whereas building elements and / or piles of different shape are placed at different predefined storage areas.

8. Method for producing individual building elements and / or piles comprising a plurality of individual building elements, which in particular are used for the construction of building structures made from several of the individual building elements, the method comprising the steps of:B1 In a manufacturing process, especially a bending and / or cutting process, producing an individual building element or a pile thereof that is expected to have a predefined shape;B2 Performing the method according to claim 1 with the individual building element or pile of step B1 as the building element or pile to be identified and the predefined shape of step B1 as the only predefined shape in steps A3 and A4;B3 Optionally, depending on the outcome of step B2, (i) providing the building element for further use, e.g. by placing it in a specific storage area or by feeding it into a processing area, or (ii) disposing the building element, e.g. by placing it in a disposal area;B4 Optionally, repeating steps B1 to B3.

9. Method according to claims 8 and 2, whereby the machine-learning model or the joint machine-learning model provides as output N correction values, and wherein before carrying out step B4, the N correction values are used to change N actuation variables which N actuation variables influence the manufacturing process.

10. Method according to claim 8 or 9, whereby (i) the building element is provided for further use if the extracted shape in step A4 corresponds to the predefined shape and / or the deviation is below a predefined threshold and the building element is disposed if the extracted shape in step A4 does not corresponds to the predefined shape and / or the deviation is equal to or above a predefined threshold.

11. Method according to any of claims 8 - 10, further comprising the step of adjusting one or more manufacturing parameters based on the deviation obtained in step A4 and in step B4 repeating steps B1 to B3 with the adjusted parameters.

12. Computer-implemented method for producing a building structure, especially a 3D reinforcement and / or formwork structure,, from multiple individual building elements with a robotic system, the method comprising the steps of:C1 Selecting a desired shape of an individual building element which is to be installed next, especially based on a digital construction plan of the building structure, especially the 3D reinforcement structure, whereby the desired shape is defined by a data model of the individual building element;C2 Producing or providing an individual building element that is expected to have a shape corresponding to the desired shape;C3 Performing the method according to any of claims 1 - 7 with the individual building element of step C2 as the building element to be identified and the desired shape as the only predefined shape in steps A3 and A4;C4 If the extracted shape in step A4 corresponds to the predefined shape and / or the deviation is below a predefined threshold, assigning a planned gripping position in the image, especially based on corresponding information in the digital construction plan of the building structure; otherwise, stop the production process or dispose the building element and start again with step C2;C5 Picking up the individual building element with a gripping device of the robotic system at the planned gripping position,C6 Placing the individual building element at a desired position and assembling and / or joining it with a support element, a further element of the building structure and / or a further individual building element;C7 Optionally, repeating steps C1 to C6 until the building structure, especially the 3D reinforcement and / or formwork structure, is fully built.

13. The method according to claim 12, whereby during and / or after the production of the building structure, especially the 3D reinforcement structure, the shape of the partly built or fully built building structure, especially the 3D reinforcement and / or formwork structure, is determined with a range sensor, especially a RGB-D sensor, which is mounted on a robotic system so that the entire building structure can be scanned with the sensor from different directions.

14. The method according to claim 13, whereby the shape of the partly built or fully built building structure, especially the 3D reinforcement and / or formwork structure, is compared with a predefined data model of the building structure, especially the 3D reinforcement and / or formwork structure, especially to identify deviations between the structure and the model.

15. The method according to any of preceding claims, whereby the building element, especially the individual reinforcing element, is a non-straight element, especially a 2D structure made of metallic wire with at least one curvature and / or kink.

16. The method according to any of preceding claims, whereby the individual building element, especially the individual reinforcing element, is a multi-component structure, especially made from several interconnected wires, such as e.g. a mat-like structure.

17. A system, comprising:- an image-generating device, in particular mounted on a robotic system and / or fixed above a picking area;- a control unit that is configured to perform a method according to any of preceding claims.- optionally a gripping device, especially a robotic gripping system that is moveable in one, two directions of space; - optionally, a manufacturing device, especially a bending and / or cutting device; optionally, a placing and / or assembling device, especially a robotic placing device and / or a welding device.

Images