Method and apparatus for recognising construction products and / or processes at a construction site
Patent Information
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- LIEBHERR WERK BIBERACH GMBH
- Filing Date
- 2024-10-21
- Publication Date
- 2026-07-08
AI Technical Summary
Existing systems face challenges in reliably identifying and characterizing construction products and processes on construction sites due to the vast variety and quantity of products and processes, which requires a substantial amount of training data for artificial neural networks.
The use of multiple separate artificial neural networks, each trained for specific sub-areas of construction products and processes, allows for the evaluation of sensor data sets by comparing results from different networks to achieve more reliable identification.
This approach reduces the overwhelming burden of training data on system providers, enhances the reliability of detection by distributing identification probabilities, and allows for efficient evaluation and data transmission.
Smart Images
Figure EP2024079665_01052025_PF_FP_ABST
Abstract
Description
[0001] Method and device for detecting construction products and / or processes on a construction site
[0002] The present invention relates to a method and a device for detecting construction products and / or construction processes on a construction site. A sensor system detects a construction product and / or process on the construction site and provides a product- and / or process-specific sensor data set, which is evaluated by an artificial neural network to identify and / or characterize the construction product and / or process. The invention further relates to a construction machine with such a device.
[0003] On construction sites, it is helpful for automated workflows to automatically record construction products such as brick pallets, reinforcing iron, roof truss beams, or building materials such as sand, gravel, or cement, as well as certain construction processes such as excavation, concreting a floor slab, or erecting formwork walls. The identification of the respective construction product and / or process, or at least its characterization or typification, can be used, for example, to control automated work processes. For example, the identification of the construction objects within a crane's working area can be used to "tell" a crane for an automated crane lift whether or where, for example, a specific formwork element or any formwork elements at all are present within the crane's working area.
[0004] Alternatively or additionally, the automated identification of construction objects or processes can also be used to monitor the progress of the construction site, for example to check whether certain construction objects have already been installed or assembled in the correct place by comparing them with the data from a BIM, i.e. a so-called Building Information Model.
[0005] A well-known approach to object recognition uses artificial neural networks, sometimes abbreviated to ANN, which are used for machine learning and artificial intelligence. They interpret various data sources such as images, sounds, text, tables, time series, or even graphs, and extract information or patterns to apply them to unknown data and make data-driven predictions or determinations. To successfully use such artificial neural networks to detect patterns in data, the artificial neural networks must be regularly trained beforehand with known data, the so-called training data, which the neural networks use to learn and improve their accuracy over time.
[0006] This "training" of the neural networks is inherently very complex and requires a large amount of sample data and a large amount of varied training data in order to recognize more complex patterns. This problem is exacerbated on construction sites, as there is a large number of different building products and many different construction processes are carried out. This diversity and quantity of building products and construction processes on a construction site would require an almost unmanageable amount of training data to enable the neural network to actually identify a specific building product from the sensor data set provided in the context of a crane lift by sensors such as cameras or laser detection sensors. In any case, the diversity and quantity of training data would overwhelm a system provider and developer of a construction site recording and / or documentation system.
[0007] For example, WO 2019 / 137815 A1 proposes using sensors to record building materials on a construction site for the purpose of data comparison with a BIM. In addition to the size and properties of the building material, which are recorded, for example, by spectral sensors, the shape of the building product is also recorded by sensors, which is then analyzed using a machine vision algorithm to recognize the building product. For this purpose, the algorithm was trained on the basis of a training data set composed of image data in order to generate characteristic features from the training images and create a collection of feature vectors. In order to be able to unambiguously assign a feature vector to an object, a so-called classifier is trained.
[0008] For the reasons mentioned above, however, such training regularly overwhelms a system provider of a system that records or documents the entire construction site.
[0009] The present invention is therefore based on the object of creating improved methods and devices of the type mentioned above that avoid the disadvantages of the prior art and advantageously develop the latter. In particular, a recognition system is to be provided that can reliably identify the construction products and processes used on a construction site without overburdening a system provider.
[0010] According to the invention, the stated object is achieved by a method according to claim 1, a device according to claim 10 and a construction machine according to claim 15. Preferred embodiments of the invention are the subject of the dependent claims. It is therefore proposed that the recognition of the construction products or processes is no longer carried out using a central artificial neural network, but rather that several separate artificial neural networks be used for this purpose, which have each been trained only for sub-areas, but in comparison to one another for different sub-areas of the construction products and / or construction processes that may be present at the construction site. According to the invention, the sensor data set that is provided by the sensors on the construction site when detecting a construction product and / or a construction process and, if necessary,has been pre-processed, is transmitted by the recognition system on the construction site to several separate artificial neural networks and evaluated by the several artificial neural networks to identify the construction product and / or process, whereby the results of the evaluations by the artificial neural networks are transmitted back to the recognition system on the construction site. By connecting several separate neural networks, not only can the recognition system on the construction site be prevented from being overwhelmed by excessive amounts of data or a provider of the construction site recognition system from being overwhelmed by excessive training effort to train a central neural network, but a relatively high spread of different identification probabilities and thus more reliable recognition of the construction product or the respective construction process can be achieved.
[0011] Advantageously, the plurality of neural networks can be trained in different ways and each only with a subset or a subset of the construction products and / or construction processes that may be present on the construction site as a whole, and / or trained with training data sets that each describe or represent only a section of the total possible construction products and / or construction processes on a construction site, wherein different training data sets are used for different artificial neural networks to train the algorithm, so that each neural network is only trained with "its" construction products and / or construction processes.
[0012] In particular, the artificial neural networks can be trained in a manufacturer- or supplier-specific manner only for the product range and / or process range of the respective manufacturer or supplier of construction products, whereby such product- and / or process range-specific training can be carried out, for example, in the sense of a deep learning process in which a large number of training data with a high degree of diversity can be fed to the respective neural network, whereby the said training data only relate to the construction products and / or processes that the respective manufacturer or supplier has in its range.
[0013] The sensor data transmitted by the recognition system at the construction site to the various artificial neural networks, which were generated and possibly pre-processed during the sensory detection of a construction product and / or a construction process on the construction site, can therefore be analyzed more finely and complexly by the artificial neural network that has been intensively trained with the training data for the specific construction product or the specific construction process than by an artificial neural network that has been trained with training data for a completely different construction product.
[0014] For example, if an image dataset depicting a formwork element is sent as a query to an artificial neural network trained with images of formwork elements, a high probability of recognition can be expected. However, if the same image dataset is sent as a query to an artificial neural network trained with images of reinforcing bars and mesh, not with images of formwork elements, a high probability of recognition cannot be expected; instead, a result such as "unknown" will likely be returned.
[0015] In this respect, the reported results of the evaluations, viewed as a whole, can narrow down or identify the respective building product with increased reliability using the differently designed or differently trained artificial neural networks, as significantly different recognition probabilities can be assigned to the various evaluations. Advantageously, the various artificial neural networks are not only trained individually and independently of the other neural networks, but are also autonomous and operator- or manufacturer-controlled, independent of the other networks. In particular, the manufacturers or suppliers of the respective building products and construction processes retain full sovereignty over the data relating to their own building products or construction processes, and in particular also over the training data used to train the respective artificial neural network.The operator of any artificial neural network can deny any other manufacturer and / or operator of a respective other artificial neural network access to “its” artificial neural network and can also control the access that the construction site detection system has and, if necessary, block or disable it.
[0016] For example, the communication between the construction site detection system and the multiple neural networks can be controlled and monitored via an application programming interface (API), whereby the respective manufacturer or operator of the artificial neural network can use the said application programming interface (API) to control access to its artificial neural network, in particular to block or disable the transmission of sensor data and / or the return transmission of analysis results or to make it dependent on the successful completion of an access authorization.
[0017] The recognition system on the construction site can select the most suitable or most reliable result from the multiple results of the evaluations transmitted by the various artificial neural networks, for example based on a probability of match for a specific construction product and / or a specific construction process.
[0018] For this purpose, the recognition system on the construction site can, for example, consider or evaluate evaluation characteristics transmitted by the artificial neural networks, such as probabilities of match, or compare them with each other. Alternatively or additionally, the recognition system on the construction site can also use other supplementary information or compare the evaluation results with additional information beyond the responses of the neural networks. For example, the recognition system can compare the returned results with data from a construction site information model, i.e., a BIM, for example, to determine whether a construction product identified by an artificial neural network is actually present on the construction site.has already been delivered, for example to exclude or classify as unlikely those results which, according to the BIM information, are not even present on the construction site.
[0019] By using separate neural networks, the evaluation time required to evaluate a sensor data set can also be significantly reduced, since the amount of data is smaller per neural network, for example in terms of the number of existing feature vectors with which the sensor data is compared.
[0020] Alternatively, or in addition to such a comparison with BIM data, the returned results provided by the various artificial neural networks can also be evaluated with or against each other. For example, the recognition system on the construction site can make majority decisions, such as considering a particular building product identified if three out of five responses from the five artificial neural networks in this case identify the building product in question as such, and two other responses represent deviations.
[0021] In an advantageous development of the invention, the transmission of a sensor data set or a query data set can also be controlled depending on the availability of a specific range or selection of building products and their construction processes on the construction site. The range of building products and / or construction processes currently available on the construction site can be determined, for example, by querying the BIM server or provided by the BIM. Based on these data, the recognition system then preselects the artificial neural networks to which the query data set is then actually sent.
[0022] Preferably, the selection of artificial neural networks to which a query is directed or to which a respective sensor data set is transmitted for identification is dynamically updated continuously or cyclically based on information stored or kept available in the BIM regarding the construction products and / or construction processes on the construction site. Such dynamic updating of the potentially queried networks can increase the hit rate and reliability while simultaneously reducing the required analysis performance and the data transmission volume and performance.
[0023] Advantageously, the aforementioned BIM can dynamically update the list of construction products and / or construction processes present on the construction site, preferably at relatively short intervals of at least once a week or once a day, so that the list of construction products or construction processes present on the construction site is preferably up-to-date on a daily basis. In particular, "old" construction products or construction processes that are no longer present on the construction site, or that have already been installed or completed, are removed from the dynamically updated list, so that this list actually only contains the construction products and construction processes that are current on the day or according to the review cycle. This allows the selection of artificial neural networks to be queried to be further narrowed down to achieve even better results.
[0024] For example, if the BIM provides the information that only construction products from suppliers A, B, and C, and construction processes from service provider D, are currently available on the construction site, the recognition system can transmit a sensor data set or query data set provided by the sensors and possibly pre-processed only to the neural networks of manufacturers A, B, C, and service provider D, because their neural networks have been specifically trained for the possible construction products and construction processes. Other neural networks that might otherwise be available for queries, for example, from other manufacturers E, F, and G, can be excluded based on the aforementioned preselection.
[0025] Alternatively, however, presumably less qualified neural networks of the aforementioned manufacturers E, F and G could also be requested, whereby, for example, a pre-weighting can be generated and possibly stored, in particular to the effect that the returned results of the networks E, F and G will presumably have a lower hit probability, which can then be taken into account when selecting the appropriate result by the recognition system.
[0026] The invention is explained in more detail below using a preferred embodiment and the accompanying drawings. In the drawings:
[0027] Fig. 1: a schematic representation of a device for detecting construction products and / or construction processes on a construction site according to an advantageous embodiment of the invention, which illustrates the sensors used on the construction site and the sensor data provided thereby as well as the transmission of said sensor data to various artificial neural networks.
[0028] As shown in Fig. 1, the device 10 for determining construction products and / or processes comprises a recognition system 1, which can be installed on a construction site 5 and, for example, can comprise a construction site server or be at least partially implemented therein. Alternatively or additionally, the recognition system 1 can also be installed on a construction machine, such as a crane, by means of which construction products can be processed or transported on the construction site 5.
[0029] The said detection system 1 can, for example, have as electronic components a microprocessor, a program memory, a working memory and input / output units, for example comprising a data transmission module, in order to process sensor data and to be able to communicate with neural networks 3.
[0030] In particular, sensor data can be supplied to the detection system 1 by a sensor system 2, which the said sensor system 2 can provide when detecting construction products and / or processes on the construction site 5.
[0031] Said sensor system 2 can comprise various sensors or detectors, wherein in particular an image-capturing sensor system or imaging sensors can be provided, for example in the form of an image and / or video camera 6 and / or a LIDAR sensor system and / or a radar sensor system and / or a laser sensor system for scanning building products using a laser beam and / or an infrared sensor system for generating an infrared image and / or an ultrasound sensor system for generating an ultrasound image of the building products. Alternatively or additionally, however, physical sensors 7 can also be used, the signals of which represent profile curves, cf. Fig. 1 , wherein such sensors can comprise, for example, a spectral sensor system and / or a color temperature sensor and / or a reflectance sensor system.
[0032] Preferably, the sensors of the sensor system 2 can operate without contact, although buttons such as temperature sensors or pH value sensors can also be provided.
[0033] The sensor data provided to the recognition system 1 are compiled by the recognition system 1 into a sensor data set or query data set, possibly with preprocessing of the received sensor data, such as signal filtering and / or signal amplification or other signal-directing measures. The recognition system 1 transmits the sensor data set via one or more data transmission interfaces 8 to various artificial neural networks 3A, 3B, and 3N, wherein the transmission of the sensor data set to the various artificial neural networks can occur in parallel or simultaneously. For example, the transmission interface 8 can include a connection to a data network such as the Internet or a wireless network such as a mobile radio network, wherein the transmission interface 8 can be designed, for example, in the form of a WLAN interface or a wireless module.
[0034] As Fig. 1 shows, the multiple different artificial neural networks 3 can each be trained for a manufacturer-specific range of building products. For example, the neural network 3A of a building product manufacturer can have been trained with training data that only reflects or describes the building products that the manufacturer has in its product portfolio. The training data can be sensor-captured image data of the building products sold by the manufacturer, or sensor data relating to the building products captured by other sensors. The artificial neural network has recognized patterns in the sensor data, for example in the image data, that represent the building products, using a deep learning method based on the diverse and extensive training data, for example.The data patterns or the patterns in the profiles of physical sensors used to record the building products during the training phase can be used by the artificial neural network, for example, to generate algorithm rules that can then be used to analyze future sensor data sets or to identify a specific building product from these sensor data sets.
[0035] The artificial neural network 3B of another manufacturer of building products, also shown in Fig. 1, has been trained in a similar manner, but with different training data, namely data that characterizes or reflects the specific range of building products offered by this other manufacturer. This other range of building products may also have been sensor-captured, preferably using different sensors and in repeated capture loops, in order to obtain a large and diverse set of training data for training the neural network. The same applies to the further artificial neural network 3N, which may in turn have been trained in a similar manner, but again with different training data that depicts or reflects the spectrum of building products and / or construction processes offered by manufacturer N.
[0036] The sensor data or the corresponding query data set transmitted simultaneously by the recognition system 1 to the plurality of artificial neural networks 3 are / is then analyzed by the plurality of artificial neural networks 3 in order to assign the data or profile patterns to a construction product and / or to a construction process and to identify or at least categorize or type a corresponding construction product and / or a corresponding construction process.
[0037] The results of the several requested artificial neural networks 3 are then transmitted back to the recognition system 1 by the said artificial neural networks 3, see Fig. 1.
[0038] The recognition system 1 determines a specific construction product from the various responses of the various artificial neural networks 3, for example based on a probability of match as previously described, and makes the identification available to a downstream application 4, for example a construction machinery control device, for example in the form of a crane control, which can then, for example, carry out an automated lift in order to transport the identified construction product to the correct location on the construction site.
[0039] The recognition system 1 provided on construction site 5 therefore no longer needs to maintain all construction product types and manufacturers thereof, or all construction process types and service providers for them, but is designed to integrate and / or query various, in particular differently trained, artificial neural networks 3, in particular with a sensor data set or a query data set relating to construction products and / or processes that are also used on construction site 5. This allows for load distribution and, in particular, a reduction in the complexity of object and process recognition on construction site 5.
[0040] With the proposed detection system 1, manufacturers of construction products or providers of construction processes can be included in the detection process. Advantageously, these manufacturers or providers can each train their own, independently operable, artificial neural network using only their products, which leads to more refined results not only through the specific product selection but also through the respective manufacturer's extensive product knowledge for the respective product range.
[0041] In particular, the artificial neural networks remain with the respective manufacturer or provider, whereby the corresponding data is stored, for example, in a manufacturer- or provider-specific cloud and can only be used by the recognition system 1 of the construction site 5 via an API, i.e. via a described interface or, at best, even in the future via a standardized interface.
[0042] In principle, it is also conceivable that this network could also be transferred to the construction site 5, for example, via a data storage device or a computer unit on site. Irrespective of this, the respective manufacturer of the construction products or provider of the construction processes retains data sovereignty over "its" artificial neural network 3 and the product information contained therein. In particular, the aforementioned manufacturer or provider can update "its" artificial neural network at any time, especially upon the addition of new products or processes or the discontinuation of previously offered products or processes.
[0043] The recognition system 1 can provide the technological basis for addressing and querying the distributed and separate artificial neural networks, for example, via sensor and / or image information. This can be done using the described detection sensor system 2 on the construction site 5, which enables observation of construction products and construction processes. The aforementioned detection sensor system 2 or the recognition system 1 can, but does not have to, have its own knowledge base, as this can be distributed across the various artificial neural networks 3.
[0044] Which artificial neural networks 3 are queried in each case can be defined by the framework conditions of the construction site 5. This can be done, for example, by manual input, but in particular can also be automated using the described digital building or construction site model (BIM). Information from the BIM can be used, e.g., information regarding the manufacturers of the materials and construction products used and / or regarding the providers of the construction processes, to dynamically integrate the relevant artificial neural networks 3. This can advantageously be done in combination with a construction schedule that is updated daily or at least at short intervals.
[0045] The evaluation, which can be carried out, for example, via weighting and probabilities from the results of the queried artificial neural networks 3 analogous to the respective referenced source, can provide significantly better recognition results.
[0046] A major advantage of the preferably daily integration of only relevant sources or artificial neural networks 3 is a significant increase in the overall probability of a true-positive result, since potential sources of error, such as manufacturers who are not represented on site, can be excluded from the outset.
[0047] Through the decentralized interaction or the use of external knowledge bases, in particular in the form of the aforementioned artificial neural networks via an application programming interface, linked to a preferably cloud-based artificial neural network, a high-quality recognition system for building products and construction processes on construction sites can be achieved.
Claims
Claims 1 . Method for recognizing building products and / or construction processes on a construction site (5), wherein at least one building product and / or process on the construction site (5) is detected by means of a sensor system (2) and a sensor data set resulting from the detection is provided, which is evaluated by means of an artificial neural network (3) to determine the building product and / or process, characterized in that the said sensor data set is transmitted by a recognition system (1) from the construction site (5) to a plurality of separate artificial neural networks (3A, 3B, 3N) and is evaluated by the plurality of artificial neural networks (3) to identify a building product and / or construction process, wherein the results of the evaluations of the plurality of artificial neural networks (3) are transmitted back to the recognition system (1) to recognize the building product and / or process.
2. Method according to the preceding claim, wherein the plurality of separate artificial neural networks (3) are different from one another and are individually trained with differing, network-specific training data.
3. Method according to the preceding claim, wherein the plurality of artificial neural networks (3) are each trained with training data which reflect only a subset of the total construction products and / or construction processes used on the construction site (5), wherein the training data for the various artificial neural networks (3) are generated on the basis of different construction product ranges and / or different construction process ranges, so that each artificial neural network (3) is trained with individual, network-specific training data.
4. Method according to one of the preceding claims, wherein each of the plurality of artificial neural networks only maps the construction product range and / or construction process range of one construction product and / or construction process provider.
5. Method according to one of the preceding claims, wherein each of the plurality of separate artificial neural networks (3) is trained and managed autonomously and independently of the other artificial neural networks (3), wherein the training data and the information data sets of each neural network (3) are blocked against access by the other artificial neural networks (3).
6. Method according to one of the preceding claims, wherein the transmission of the sensor data set to a respective one of the plurality of artificial neural networks (3) and / or the return transmission of the result of the evaluation of a respective one of the plurality of artificial neural networks (3) to the recognition system (1) is made dependent on an approval by said respective one of the plurality of artificial neural networks (3) or an access control device cooperating therewith and is blocked in the absence of such approval.
7. Method according to one of the preceding claims, wherein the transmission of the sensor data set from the recognition system (1) to the plurality of separate artificial neural networks (3A, 3B, 3N) is carried out via an application programming interface (API) and / or the return transmission of the results of the evaluations from the plurality of artificial neural networks (3) to the recognition system (1) is carried out via said application programming interface (API).
8. Method according to the preceding claim, wherein said application programming interface (API) checks an access authorization of the recognition system to each of said plurality of artificial neural networks (3), wherein the access authorization to be checked by said application programming interface API is individually specified by each of the plurality of neural artificial networks (3).
9. Method according to one of the preceding claims, wherein the recognition system (1) identifies a respective building product and / or a respective building process from the returned results of the evaluations of the plurality of artificial neural networks (3) using at least one piece of additional information.
10. Method according to the preceding claim, wherein the recognition system (1) uses a probability of match with a respective construction product and / or construction process as additional information for the identification.
11. Method according to one of the two preceding claims, wherein the recognition system (1) for said identification compares the results obtained from the evaluations of the various artificial neural networks (3) with additional information from a building information model (BIM) to determine whether a building product identified in a respective result and / or a building process identified in the result is present on the construction site (5) on the basis of the information from the BIM.
12. Method according to one of the preceding claims, wherein the recognition system (1) for identifying the building product and / or building process compares the returned results of the evaluations of the various artificial neural networks (3) with each other and determines a majority probability for a building product and / or a building process.
13. Method according to one of the preceding claims, wherein the recognition system (1) for the transmission of said sensor data set makes a pre-selection of the artificial neural networks (3) to be queried with the sensor data set on the basis of information from a / the construction site information model (BIM), in such a way that the sensor data set is only transmitted to the artificial neural networks (3) which are trained with training data on construction products which are also present on the construction site (5) on the basis of the information from the BIM.
14. Method according to the preceding claim, wherein said sensor data set is transmitted only to the artificial neural networks (3) which are trained with training data on building products which are available on the basis of the information from the BIM on the construction site (5) at the time at which the sensor data were recorded by the sensor system (2) on the construction site (5).
15. Method according to one of the two preceding claims, wherein the selection of the artificial neural networks (3) to which a sensor data set is to be transmitted is dynamically updated using information from the construction site information model (BIM).
16. Method according to the preceding claim, wherein the construction site information model (BIM) is dynamically updated with regard to the information as to which construction products are present on the construction site (5) and / or which construction processes are carried out on the construction site (5), wherein the dynamic update is carried out at least weekly, preferably at least once every working day, and such construction products that are present on the construction site (5) were present but are no longer present, and / or such construction processes that were carried out on the construction site but have been completed are removed from the data set of information on the basis of which the artificial neural networks (3) are selected for transmitting the sensor data set.
17. Device for determining construction products and / or construction processes on a construction site (5), with a sensor system (2) for detecting at least one construction product and / or process on the construction site (5) and providing a sensor data set resulting from the detection, characterized by a recognition system (1) which is configured to transmit said sensor data set from the construction site (5) to a plurality of separate artificial neural networks for evaluating the sensor data set and to receive results of the individual evaluations of the artificial neural networks from said artificial neural networks (3) and to identify a respective construction product and / or a respective construction process from the results.
18. Device according to the preceding claim, wherein said recognition system (1) is in communication with said plurality of separate artificial neural networks (3) via one or more data transmission interfaces (8), wherein said artificial neural networks (3) are trained differently from one another and are individually trained with network-specific training data, each of which represents only a section of the construction product and / or construction process range that is present on the construction site (5) as a whole.
19. Device according to the preceding claim, wherein said data transmission interface (8) is an application programming interface (API) for controlling access of the recognition system (1) to the artificial neural networks (3).
20. Device according to one of claims 17 to 19, wherein the sensor system (2) comprises at least one optical and / or imaging sensor module for providing image data of the detected building product and / or the detected building process.
21. Device according to the preceding claim, wherein said optical and / or imaging sensor comprises a camera for providing a camera image of the building product and / or the building process, wherein the artificial neural networks (3) are trained with training data describing the external appearance, in particular an outline contour and contour size of the respective building products.
22. Device according to one of the preceding claims, wherein the sensor system (2) has at least one physical sensor for providing a gait pattern that describes a property of a detected building product and / or building process, wherein the artificial neural networks (3) are trained with training data that describe gait lines of a physical detection and / or physical property of at least one building product and / or building process.
23. Construction machine with a device designed according to one of claims 17 to 22.
24. Construction machine according to the preceding claim, which is designed as a crane.