Method and device for detecting construction products and / or processes on a construction site

Abstract

The present invention relates to a method and a device for detecting construction products and / or construction processes on a construction site, wherein a construction product and / or process is detected on the construction site by means of sensors, and a product- and / or process-specific sensor data set is provided, which is evaluated by means of an artificial neural network for the identification and / or characterization of the construction product and / or process. It is proposed that the detection of the construction products or processes is no longer approached by a central artificial neural network, but rather that several separate artificial neural networks are used for this purpose, which differ from one another and have each been trained only for different sub-areas of the construction products and / or construction processes present on the construction site.

Description

[0001] The present invention relates to a method and a device for detecting construction products and / or construction processes on a construction site. A sensor system detects a construction product and / or process on the site and provides a product- and / or process-specific sensor data set, which is evaluated by an artificial neural network to identify and / or characterize the construction product and / or process. The invention further relates to a construction machine with such a device.

[0002] On construction sites, it is helpful for automated workflows to automatically record construction products such as brick pallets, reinforcing iron, roof truss beams, or building materials such as sand, gravel, or cement, as well as certain construction processes such as excavation, concreting a floor slab, or erecting formwork walls. The identification of the respective construction product and / or process, or at least its characterization or typification, can be used, for example, to control automated work processes. For example, the identification of the construction objects within a crane's working area can be used to "tell" a crane for an automated crane lift whether or where, for example, a specific formwork element or any formwork elements at all are present within the crane's working area.

[0003] Alternatively or additionally, the automated identification of construction objects or processes can also be used to monitor the progress of the construction site, for example to check whether certain construction objects have already been installed or assembled in the correct place by comparing them with the data from a BIM, i.e. a so-called Building Information Model.

[0004] A well-known approach to object recognition uses artificial neural networks, sometimes abbreviated to ANN, which are used for machine learning and artificial intelligence. They interpret various data sources such as images, sounds, text, tables, time series, or even graphs, and extract information or patterns to apply them to unknown data and make data-driven predictions or determinations. To successfully use such artificial neural networks to detect patterns in data, the artificial neural networks must be regularly trained beforehand with known data, the so-called training data, which the neural networks use to learn and improve their accuracy over time.

[0005] This "training" of the neural networks is inherently very complex and requires a large amount of sample data and a large amount of varied training data in order to recognize more complex patterns. This problem is exacerbated on construction sites, as a large number of different building products are present and many different construction processes are carried out. This diversity and quantity of building products and construction processes on a construction site would require an almost unmanageable amount of training data to enable the neural network to actually identify a specific building product from the sensor data set provided by sensors such as cameras or laser detection sensors in the context of a crane lift. In any case, the diversity and quantity of training data would overwhelm a system provider and developer of a construction site recording and / or documentation system.

[0006] For example, WO 2019 / 137815 A1 proposes using sensors to record building materials on a construction site for the purpose of data comparison with a BIM. In addition to the size and properties of the building material, which are recorded, for example, by spectral sensors, the shape of the building product is also recorded by sensors, which is then analyzed using a machine vision algorithm to recognize the building product. For this purpose, the algorithm was trained on the basis of a training data set composed of image data in order to generate characteristic features from the training images and create a collection of feature vectors. In order to be able to unambiguously assign a feature vector to an object, a so-called classifier is trained.

[0007] For the reasons mentioned above, however, such training regularly overwhelms a system provider of a system that records or documents the entire construction site.

[0008] The present invention is therefore based on the object of creating improved methods and devices of the type mentioned above that avoid the disadvantages of the prior art and advantageously develop the latter. In particular, a recognition system is to be provided that can reliably identify the construction products and processes used on a construction site without overburdening a system provider.

[0009] According to the invention, the stated object is achieved by a method according to claim 1, a device according to claim 10 and a construction machine according to claim 15. Preferred embodiments of the invention are the subject of the dependent claims.

[0010] It is therefore proposed that the recognition of construction products or processes should no longer be approached using a central artificial neural network, but rather that several separate artificial neural networks be used, each of which has been trained only for sub-areas, but compared to one another for different sub-areas of the construction products and / or construction processes that may be present at the construction site. According to the invention, the sensor data set, which has been provided by the sensors on the construction site when detecting a construction product and / or a construction process and has been pre-processed if necessary, is transmitted by the recognition system on the construction site to several separate artificial neural networks and evaluated by the several artificial neural networks to identify the construction product and / or process, wherein the results of the evaluations by the artificial neural networks are transmitted back to the recognition system on the construction site.By connecting several separate neural networks, not only can an overloading of the recognition system on the construction site due to excessive data volumes or a provider of the construction site recognition system due to excessive training effort for training a central neural network be avoided, but a relatively high spread of different identification probabilities and thus a more reliable recognition of the construction product or the respective construction process can be achieved.

[0011] Advantageously, the plurality of neural networks can be trained in different ways and each only with a subset or a subset of the construction products and / or construction processes that may be present on the construction site as a whole, and / or trained with training data sets that each describe or represent only a section of the total possible construction products and / or construction processes on a construction site, wherein different training data sets are used for different artificial neural networks to train the algorithm, so that each neural network is only trained with "its" construction products and / or construction processes.

[0012] In particular, the artificial neural networks can be trained in a manufacturer- or supplier-specific manner only for the product range and / or process range of the respective manufacturer or supplier of construction products, whereby such product- and / or process range-specific training can be carried out, for example, in the sense of a deep learning process in which a large number of training data with a high degree of diversity can be fed to the respective neural network, whereby the said training data only relate to the construction products and / or processes that the respective manufacturer or supplier has in its range.

[0013] The sensor data transmitted by the recognition system at the construction site to the various artificial neural networks, which were generated and possibly pre-processed during the sensory detection of a construction product and / or a construction process on the construction site, can therefore be analyzed more finely and complexly by the artificial neural network that has been intensively trained with the training data for the specific construction product or the specific construction process than by an artificial neural network that has been trained with training data for a completely different construction product.

[0014] For example, if an image dataset depicting a formwork element is sent as a query to an artificial neural network trained with images of formwork elements, a high probability of recognition can be expected. However, if the same image dataset is sent as a query to an artificial neural network trained with images of reinforcing bars and mesh, not with images of formwork elements, a high probability of recognition cannot be expected; instead, a result such as "unknown" will likely be returned.

[0015] In this respect, the reported results of the evaluations, viewed as a whole, can be used by the differently designed or differently trained artificial neural networks to narrow down or identify the respective building product with increased reliability, since the different evaluations can be assigned significantly different recognition probabilities.

[0016] The recognition system on the construction site can select the most suitable or most reliable result from the multiple results of the evaluations transmitted by the various artificial neural networks, for example based on a probability of match for a specific construction product and / or a specific construction process.

[0017] For this purpose, the recognition system on the construction site can, for example, take into account, evaluate or compare evaluation characteristics transmitted by the artificial neural networks, such as probabilities of match.

[0018] Alternatively or additionally, the recognition system on the construction site can also utilize other supplementary information or compare the evaluation results with additional information beyond the responses from the neural networks. For example, the recognition system can compare the returned results with data from a construction site information model, i.e., a BIM, to determine, for example, whether a construction product identified by an artificial neural network is actually present on the construction site or has already been delivered. This can be used to exclude or classify as unlikely results that, according to the BIM information, are not present on the construction site at all.

[0019] By using separate neural networks, the evaluation time required to evaluate a sensor data set can also be significantly reduced, since the amount of data is smaller per neural network, for example in terms of the number of existing feature vectors with which the sensor data is compared.

[0020] Alternatively, or in addition to such a comparison with BIM data, the returned results provided by the various artificial neural networks can also be evaluated with or against each other. For example, the recognition system on the construction site can make majority decisions, such as considering a particular building product identified if three out of five responses from the five artificial neural networks in this case identify the building product in question as such, and two other responses represent deviations.

[0021] In an advantageous development of the invention, the transmission of a sensor data set or a query data set can also be controlled depending on the availability of a specific range or selection of building products and their construction processes on the construction site. The range of building products and / or construction processes currently available on the construction site can be determined, for example, by querying the BIM server or provided by the BIM. Based on these data, the recognition system then preselects the artificial neural networks to which the query data set is then actually sent.

[0022] For example, if the BIM provides the information that only construction products from suppliers A, B, and C, and construction processes from service provider D, are currently available on the construction site, the recognition system can transmit a sensor data set or query data set provided by the sensors and possibly pre-processed only to the neural networks of manufacturers A, B, C, and service provider D, because their neural networks have been specifically trained for the possible construction products and construction processes. Other neural networks that might otherwise be available for queries, for example, from other manufacturers E, F, and G, can be excluded based on the aforementioned preselection.

[0023] Alternatively, however, presumably less qualified neural networks of the aforementioned manufacturers E, F and G could also be requested, whereby, for example, a pre-weighting can be generated and possibly stored, in particular to the effect that the returned results of the networks E, F and G will presumably have a lower hit probability, which can then be taken into account when selecting the appropriate result by the recognition system.

[0024] The invention is explained in more detail below using a preferred embodiment and the accompanying drawings. In the drawings: Fig. 1: a schematic representation of a device for detecting construction products and / or construction processes on a construction site according to an advantageous embodiment of the invention, which illustrates the sensors used on the construction site and the sensor data provided thereby as well as the transmission of said sensor data to various artificial neural networks.

[0025] As the Fig. As shown in Figure 1, the device 10 for determining construction products and / or processes comprises a recognition system 1, which can be installed on a construction site 5 and, for example, can comprise a construction site server or be at least partially implemented therein. Alternatively or additionally, the recognition system 1 can also be installed on a construction machine, such as a crane, by means of which construction products can be processed or transported on the construction site 5.

[0026] The said detection system 1 can, for example, have as electronic components a microprocessor, a program memory, a working memory and input / output units, for example comprising a data transmission module, in order to process sensor data and to be able to communicate with neural networks 3.

[0027] In particular, sensor data can be supplied to the detection system 1 by a sensor system 2, which the said sensor system 2 can provide when detecting construction products and / or processes on the construction site 5.

[0028] Said sensor system 2 can comprise various sensors or detectors, wherein, in particular, an image-capturing sensor system or imaging sensors can be provided, for example in the form of an image and / or video camera 6 and / or a LIDAR sensor system and / or a radar sensor system and / or a laser sensor system for scanning building products using a laser beam and / or an infrared sensor system for generating an infrared image and / or an ultrasound sensor system for generating an ultrasound image of the building products. Alternatively or additionally, physical sensors 7 can also be used, the signals of which represent profile curves, cf. Fig. 1, wherein such sensors may comprise, for example, a spectral sensor and / or a color temperature sensor and / or a reflectance sensor.

[0029] Preferably, the sensors of the sensor system 2 can operate without contact, although buttons such as temperature sensors or pH value sensors can also be provided.

[0030] The sensor data provided to the recognition system 1 is compiled by the recognition system 1 into a sensor data set or query data set, possibly with preprocessing of the received sensor data, such as signal filtering and / or signal amplification or other signal-directing measures. The recognition system 1 transmits the sensor data set via one or more data transmission interfaces 8 to various artificial neural networks 3A, 3B, and 3N, wherein the transmission of the sensor data set to the various artificial neural networks can occur in parallel or simultaneously. For example, the transmission interface 8 can include a connection to a data network such as the Internet or a wireless network such as a mobile radio network, wherein the transmission interface 8 can be designed, for example, in the form of a WLAN interface or a wireless module.

[0031] How Fig. 1 shows, the multiple different artificial neural networks 3 can each be trained for a manufacturer-specific range of building products. For example, the neural network 3A of a building product manufacturer can have been trained with training data that only reflects or describes the building products that the manufacturer has in its product portfolio. The training data can be sensor-captured image data of the building products sold by the manufacturer, or sensor data relating to the building products captured by other sensors. The artificial neural network has recognized patterns in the sensor data, for example in the image data, that represent the building products, for example using a deep learning method based on the diverse and extensive training data.The data patterns or the patterns in the profiles of physical sensors used to record the building products in the training phase can be used by the artificial neural network, for example, to generate algorithm rules that can then be used to analyze future sensor data sets or to identify a specific building product from these sensor data sets.

[0032] The Fig. 1, the artificial neural network 3B of another manufacturer of building products, which is also shown, has been trained in a similar manner, but with different training data, namely those that characterize or depict the specific building product range of this further manufacturer, whereby this other building product range may also have been detected by sensors, preferably by different sensors and in repeated detection loops, in order to obtain a large and diverse amount of training data for training the neural network.

[0033] The same applies to the further artificial neural network 3N, which may have been trained in a similar way, but again with different training data that depicts or reflects the spectrum of building products and / or building processes offered by manufacturer N.

[0034] The sensor data or the corresponding query data set transmitted simultaneously by the recognition system 1 to the plurality of artificial neural networks 3 are / is then analyzed by the plurality of artificial neural networks 3 in order to assign the data or profile patterns to a construction product and / or to a construction process and to identify or at least categorize or type a corresponding construction product and / or a corresponding construction process.

[0035] The results of the several requested artificial neural networks 3 are then transmitted back to the recognition system 1 by the said artificial neural networks 3, cf. Fig. 1.

[0036] The recognition system 1 determines a specific construction product from the various responses of the various artificial neural networks 3, for example based on a probability of match as previously described, and makes the identification available to a downstream application 4, for example a construction machinery control device, for example in the form of a crane control, which can then, for example, carry out an automated lift in order to transport the identified construction product to the correct location on the construction site.

[0037] The recognition system 1 provided on construction site 5 therefore no longer needs to maintain all construction product types and manufacturers thereof, or all construction process types and service providers for them, but is designed to integrate and / or query various, in particular differently trained, artificial neural networks 3, in particular with a sensor data set or a query data set relating to construction products and / or processes that are also used on construction site 5. This allows for load distribution and, in particular, a reduction in the complexity of object and process recognition on construction site 5. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] WO 2019 / 137815 A1

[0006]

Claims

[1] Method for detecting construction products and / or construction processes on a construction site (5), wherein at least one construction product and / or process on the construction site (5) is detected by means of a sensor system (2) and a sensor data set resulting from the detection is provided, which is evaluated by means of an artificial neural network (3) to determine the construction product and / or process, characterized by that the said sensor data set is transmitted from a recognition system (1) from the construction site (5) to a plurality of separate artificial neural networks (3A, 3B, 3N) and is evaluated by the plurality of artificial neural networks (3) to identify a construction product and / or construction process, wherein the results of the evaluations of the plurality of artificial neural networks (3) are transmitted back to the recognition system (1) to recognize the construction product and / or process. [2] Method according to the preceding claim, wherein the plurality of separate artificial neural networks (3) are different from one another and are individually trained with differing, network-specific training data. [3] Method according to the preceding claim, wherein the plurality of artificial neural networks (3) are each trained with training data which reflect only a subset of the total construction products and / or construction processes used on the construction site (5), wherein the training data for the various artificial neural networks (3) are generated on the basis of different construction product ranges and / or different construction process ranges, so that each artificial neural network (3) is trained with individual, network-specific training data. [4] Method according to one of the preceding claims, wherein each of the plurality of artificial neural networks only maps the construction product range and / or construction process range of a construction product and / or construction process provider. [5] Method according to one of the preceding claims, wherein the recognition system (1) identifies a respective building product and / or a respective building process from the returned results of the evaluations of the plurality of artificial neural networks (3) using at least one piece of additional information. [6] Method according to the preceding claim, wherein the recognition system (1) uses a probability of match with a respective construction product and / or construction process as additional information for the identification. [7] Method according to one of the two preceding claims, wherein the recognition system (1) for said identification compares the results obtained from the evaluations of the various artificial neural networks (3) with additional information from a building information model (BIM) to determine whether a building product identified in a respective result and / or a building process identified in the result is present on the construction site (5) on the basis of the information from the BIM. [8] Method according to one of the preceding claims, wherein the recognition system (1) for identifying the building product and / or building process compares the returned results of the evaluations of the various artificial neural networks (3) with each other and determines a majority probability for a building product and / or a building process. [9] Method according to one of the preceding claims, wherein the recognition system (1) for the transmission of said sensor data set makes a pre-selection of the artificial neural networks (3) to be queried with the sensor data set on the basis of information from a / the construction site information model (BIM), in such a way that the sensor data set is only transmitted to the artificial neural networks (3) which are trained with training data on construction products which are also present on the construction site (5) on the basis of the information from the BIM. [10] Device for determining construction products and / or construction processes on a construction site (5), with a sensor system (2) for detecting at least one construction product and / or process on the construction site (5) and providing a sensor data set resulting from the detection, characterized bya recognition system (1) configured to transmit said sensor data set from the construction site (5) to a plurality of separate artificial neural networks for evaluating the sensor data set and to receive results of the individual evaluations of the artificial neural networks from said artificial neural networks (3) and to identify a respective construction product and / or a respective construction process from the results. [11] Device according to the preceding claim, wherein said recognition system (1) is in communication with said plurality of separate artificial neural networks (3) via one or more data transmission interfaces (8), wherein said artificial neural networks (3) are trained differently from one another and are individually trained with network-specific training data, each of which represents only a section of the construction product and / or construction process range that is present on the construction site (5) as a whole. [12] Device according to one of the two preceding claims, wherein the sensor system (2) comprises at least one optical and / or imaging sensor module for providing image data of the detected building product and / or the detected building process. [13] Device according to the preceding claim, wherein said optical and / or imaging sensor comprises a camera for providing a camera image of the building product and / or the building process, wherein the artificial neural networks (3) are trained with training data describing the external appearance, in particular an outline contour and contour size of the respective building products. [14] Device according to one of the preceding claims, wherein the sensor system (2) comprises at least one physical sensor for providing a gait pattern that describes a property of a detected building product and / or building process, wherein the artificial neural networks (3) are trained with training data that describe gait lines of a physical detection and / or physical property of at least one building product and / or building process. [15] Construction machine with a device designed according to one of claims 10 to 14. [16] Construction machine according to the preceding claim, which is designed as a crane.

Images