FaÇade inspection system and method

Abstract

A method of integrating a façade condition data capture process with the generic use of a BMU such that the façade condition data is recorded while mounted on a BMU. An inspection mapping apparatus which is configured to mount on a BMU to integrate façade maintenance tasks and façade condition data capture, captures the façade condition data on a pathway travelled by the BMU. A method for passively or actively recording façade condition data onto a façade site map whose relative geometry is known. An apparatus for passive or active capture of façade condition data, the apparatus including: a mounting frame for mounting one or more sensors and a computer processing cabinet to a portion of a building maintenance unit (BMU) having a cradle portion, the frame including: a base portion for mounting to the cradle portion; a sensor mounting portion for mounting one or more sensors; and one or more legs extending from the base portion to elevate the sensor mounting portion at a predetermined distance above or below the cradle portion, the apparatus further including: one or more sensors mounted on the sensor monitoring portion for sensing façade condition; a computer processing cabinet in electrical and data communication with the one or more sensors, the computer processing cabinet being mounted on the mounting frame. A method of refining the geolocation of the apparatus working at heights by correlating visually features with architectural features whose geolocations are known.

Description

[0001] FACADE INSPECTION SYSTEM AND METHOD 1. The present technology relates generally to building facade inspection apparatus, systems and methods. 2. Multi-storey building facades are exposed to constantly-changing stresses, including but not limited to: heat loads, solar stresses, structural movements and wind stresses. These stresses and others can cause, or speed up, the development of structural problems in the facade, either directly, or in the facade connection to the building structure. 3. One consequence of this is that facade elements or the whole facade itself can then break off and fall to the ground, far below. Facades and individual facade elements are extremely heavy; given sufficient height, they can crash to the ground with explosive force. 4. Other consequences are increased water leakage, thermal breaks, damage to the building, or reduction in its running efficiency. 5. Visual inspections facilitate the monitoring of the facade condition. These can be done by a site inspection, with a human inspector gaining access to the facade via rope access or using a Building Maintenance Unit (BMU). 6. Using a human operator for facade inspections presents genuine safety concerns relating to working at heights; falls, and other suspension injuries. 7. Human inspectors make errors and miss details, and the process is very labour intensive, not least of all because the task requires very intensive training. It is difficult to trace accurate details of issues identified during this kind of inspection. Once a defect is identified, inspectors take a picture of it, and then try to manually geolocate the issue by rough methods such as counting windows or other facade features - which is subject to human error due to the often repeating facade, particularly for skyscrapers. 8. There are safer alternatives currently available. A drone can fly across the facade surface and take video or stills, or a wall-crawling robot can do the same job. Drones become expensive, can be overly regulated since they require special clearances, or are again, like other methods, inaccurate. Other robots pose risks because they can be blown away by high winds since they are inadequately restrained. All of them require a separate investment in time and infrastructure because a human, or a separate machine, a separate path management software program, and separate routing process must be implemented and / or scheduled each time an inspection is required. 9. All the abovementioned inspection processes involve capturing, recording and mapping facade condition data by actively controlling the path of a drone or a robot with a control system (either manually or via computing processor). This presents a drawback in that there is map software and path planning and / or operator time involved, which is expensive and subject to onsite conditions (weather, equipment maintenance, setup time, permit approval) that may cause delays. 0. Drones cannot operate in high winds, and are susceptible to motor failure, or bird strike, or control failure due to control signal failure from building interference. These can cause the drone to crash to the street below, which is undesirable and can cause injury. They are also strictly regulated, which can reduce their flexibility and usefulness. 1. There is also a problem in that known geolocation data collection / localisation methods operate poorly in the context of tall buildings because of a phenomenon known as ‘urban canyon’, as well as the nature of the vision targets. Many imaging or sensing systems (e.g., LiDARs) behave poorly on heavily glazed surfaces such as skyscrapers. This means that it cannot be accurately known exactly where visible facade issues are located, after they have been identified. 2. Moreover, existing facade inspection methods lack traceability - only ‘interesting’ parts of the facade are documented via manual photo-taking, and there is uncertainty with where the photos are taken. Again, location data associated with the photos is not accurate and users have to try to localise themselves through crude methods such as manually counting floors or facade features, which is unreliable. It is difficult to confirm whether or not some facade problem was missed without a second inspection. 3. The present inventors have sought to ameliorate one or more of the above disadvantages, =providing a useful and novel alternative to known solutions. 4. In this specification, when a sensor or sensor suite or kit element is described or claimed herein as being “BMU mounted” it is to be understood that the mounting location can be on a Building Maintenance Unit (BMU) cradle or basket, or it could be on an associated wire rope on which the BMU is suspended, or on a suspension wire connector, or on a frame mounted on the BMU. The sensor could be located at the front or the rear of the BMU, most usefully away from cleaning or maintenance operations. The sensor may also be above or below the BMU cradle. BMUs can also include swing stage BMUs, self-powered platforms and or platforms suspended from monorails. 5. Furthermore, when the term ‘map’ is used, it is intended to include the expression “model” and to encompass a 2D representation of a surface, or a 3D model. 6. Furthermore, in this specification the terms ‘basket’, ‘cradle’, ‘platform’ and ‘gondola’ are used interchangeably to refer to the structure suspended by the main BMU in which workers occupy when conducting works on the building facade. 7. Broadly, the present technology provides a method of integrating a facade condition data capture process with a building management process undertaken by a Building Maintenance Unit (BMU) such that the facade condition data is recorded while mounted on a BMU which is undertaking, say, a cleaning process. 8. Broadly, once facade condition data is recorded, embodiments produce a browsable map of recorded visible features which are substantially accurately geolocated. 9. Broadly the present technology provides a method for harvesting geolocated facade condition data to populate a geopositionally-known facade site map.

[0002] SU M ARY

[0003] 0020. In accordance with one aspect of the present technology there is provided a system for fitment to a vehicle configured to traverse a facade, the system being configured to capture visual facade condition data, the system including: one or more mounts suitable for mounting to a building maintenance unit (BMU); a plurality of sensors mounted on the mounts, the sensors including: vision sensors; and geolocation sensors; a computer processing cabinet in electrical and data communication with the plurality of sensors, the computer processing cabinet being mounted on the BMU and including a processor, wherein the processor is configured to fuse matched architectural feature data, with geolocation sensor data, wherein matched architectural feature data is processed data that matches facade feature data identified by the vision sensors with known facade feature data from architectural documentation data; and geolocation sensor data is raw or processed data from the geolocation sensors.

[0004] 0021. In an embodiment the matched architectural feature data is obtained by a feature counting step in the processor, in which extracted features from the architectural documentation data are identified, and correlated with a timeseries data set of detected visible features from a computer vision data set.

[0005] 0022. In an embodiment the detected visible features are detected by steps in the processor taken from the group of steps consisting of: computer vision, artificial intelligence, and optical flow.

[0006] 0023. In an embodiment the vision sensors are 2D or 3D sensors including one or more from the group consisting of: RGB camera; LIDAR; thermal camera; depth camera; CCTV camera; ultrasonic wave generator.

[0007] 0024. In an embodiment the geolocation sensors include an encoder, altimeter, barometer, an IMU and / or a GPS locator.

[0008] 0025. In an embodiment the raw or processed geolocation data is fused by a method including the step of sensor fusion, and / or regression. 6. In an embodiment the method includes the step of receiving and / or preparing a known facade feature database from architectural documentation which includes extracted facade feature data coupled with a geolocalisation / geolocation of the extracted facade feature.7. In an embodiment the technology further includes the step of preparing a discretised map of extracted facade features from the architectural documentation data. 8. In an embodiment the method further includes the step of preparing a browsable map of facade condition data recorded by the vision sensors, the browsable map being populated by the vision sensor data in substantially accurate geolocations. 9. In an embodiment the method further includes the step of evaluating with the processor, the observed facade condition, and assigning risk ratings to various areas of the facade, for further inspection. 0. In an embodiment the method further includes the step of receiving the architectural documentation data into the processor from a database. 1. In an embodiment the method further includes the step of preparing a database of geolocated architectural features extracted from this process by the computer processor for use in producing matched geolocation timeseries data. 2. In accordance with one aspect of the present invention there is provided an apparatus for passive or active capture of facade condition data, the apparatus including: a. one or more mounts for mounting one or more sensors and a computer processing cabinet to a maintenance unit (BMU) having a cradle portion, the mounts including: i. one or more sensors mounted on the mounts for sensing facade condition: ii. a computer processing cabinet in electrical and data communication with the one or more sensors, the computer processing cabinet being mounted on the BMU. 3. In one embodiment there is an additional wired data communication stream configured to obtain BMU system information via a cable loom with the BMU suspension cables.4. In one embodiment the wired connection to the BMU allows for commands to be sent to the BMU computer to control BMU operations such as movement. 5. The mounts may be all on one or more sensor bars. 6. The mounts may be disparately mounted on different parts of the BMU including on suspension cabling, on a BMU cradle, in a BMU cradle, on a BMU crossbar, on a BMU spreader beam, on a BMU cradle crossbar. 7. The sensor bar may be elevated above the BMU cradle to keep the sensors away from maintenance activities being conducted from the BMU. 8. The one or more sensor bars may be mounted on one or more frame legs to elevate the bar above the activities conducted from the BMU cradle. 9. In accordance with an aspect of the present technology there is provided a system for sensing the condition of a facade, the system including: a. an automatic sensing start system including: i. A facade imaging sensor: ii. a geolocation sensor; iii. a computer processor that is configured to commence facade condition data capture, b. wherein when the system senses that the geolocation of the unit has exceeded a preset threshold, the computer processor commences recording data from the imaging sensor. 0. This can also be triggered actively through onboard actuators or computer processor controls on the digital user interface (e.g., mobile app). 1. The sensor in question may be one or a combination of several geolocation sensors, including IMU (Inertial measurement unit), GPS / GNSS, barometers, encoders, etc. 2. Another instance of this is using visual cameras. Through onboard computer processes, when the camera identifies visual cues (e.g., the roof ledge), the data capture system is triggered. 3. In accordance with one aspect of the present invention there is provided an apparatus for passive or active capture of facade condition data, the apparatus including: 4. one or more sensors and a computer processing cabinet mounted on a portion of a building maintenance unit (BMU) in a manner unobtrusive to nominal BMU operations, the one or more sensors for sensing facade condition and / or geolocation; and a. a backup power supply in case of sudden power failure. 5. In one embodiment the backup power supply is battery or capacitive. 6. In one embodiment there is a supplementary device connected to the BMU to assist with determining the vertical displacement of the primary unit / BMU cradle. This device includes: a. In one alternative, a rope guide with encoder connected to the BMU suspension wire ropes. b. In one alternative, an encoder connected to the BMU jib head / crossbar rope guide. c. In one alternative, an independent power supply. d. In one alternative, a power connection to the BMU power supply. 7. In one embodiment, the sensor mounting portion includes a weather shield to protect the sensors from the elements. 8. In one embodiment there is auxiliary lighting integrated into the sensor portion to illuminate the target facade. 9. In one embodiment there is provided a video stream of facade data coming from the sensors. 0. In one embodiment there is provided an automated computer process for identifying dark portions of a video stream and engaging auxiliary lighting when dark areas are identified. 1. In one embodiment the machine vision sensors will have automated software processes that allow it to self-calibrate parameters (exposure, ISO, focus) to ensure high quality data is always captured. Including when the target is heavily shadowed or subject to glare from the sun. 2. In accordance with one aspect of the present technology, there is provided a method for refining the geolocation data captured from a plurality of sensors using the visual sensors together with known geolocated features from prior knowledge of building details. The arrangement is such that, by using the prior knowledge we have of the building, from the architectural documentation (2D or 3D), we can correlate visual cues from the vision sensors to steadily ‘recentre’ our geolocation. This means that the geolocation need only be accurate in between these visual cues. 3. In accordance with one aspect of the present technology, there is provided a method for registering captured data from a plurality of facade inspection sensors with a digital facade model of a building, the method including the steps of, in a computer processing system: a. recording geolocation data with the plurality of sensors; b. localising and transforming the geolocation data into a relative coordinate system of the digital building facade model of the building to provide aligned data; and c. using the aligned geolocation data to directly assign data points onto the digital building representation as points in space (2D or 3D). 4. In one embodiment the method includes refining the geolocation data to provide refined geolocation data. 5. In one embodiment the method includes an additional step where the data points are autonomously assessed / evaluated by a digital process using Artificial intelligence or machine learning, and then verified by a discipline expert. 6. In an embodiment the processor assesses the vision data and adds metadata about where the vision data was recorded at a specific location and what is of interest at those points.7. In an embodiment the processor makes observations relating to defects identified and assigns a risk / urgency rating and a location description (for example, floor 10, drop 3.8. In an embodiment the data points are assessed / evaluated by a robot system and then verified by an operator. 9. In one embodiment the refining step includes one or more of: sensor fusion; object detection; computer vision feature extraction; optical flow tracking; extracting key points from LiDAR (Light Detection and Ranging) information; and SLAM (Simultaneous localisation and mapping). 0. In one embodiment the localising and transforming step includes one or more of: aligning Relative Levels of the building; aligning horizontal coordinates with the building facade. 1. In one embodiment the assigning step is arranged such that the data points may include a plurality of types of sensor data (for example, temperature, geometry, image) and be in any suitable data format. 2. In one embodiment the assigning step is arranged such that the data points may include facade condition meta data (for example, observations / defects identified, risk levels, urgency, cost estimate) and be in any suitable data format. 3. In an embodiment the method includes the additional step of using optical flow to track system movement and pose / angle / view change. 4. In accordance with one aspect of the present technology, there is provided a method for registering captured vision data from a plurality of facade inspection sensors with a digital facade model of a target building, the method including the steps of, in a computer processing system: 5. recording geolocation data with the captured machine vision data; a. localising and transforming the geolocation data into a relative coordinate system of the digital building facade model of the building to provide aligned data; and b. performing image stitching operations on the collated image data to create a larger, stitched image c. projecting the stitched image onto the digital building facade model of the target building to create a pictorial Digital Twin. 6. In one embodiment the vision data includes colour images, depth images, thermal images and the like. 7. In accordance with another aspect of the present invention there is provided a method for passively or actively recording multi-storey building facade condition data, the method including the steps of, in a computer processing system: a. receiving into the computer processing system, map data comprising a plurality of map elements relating to a multi-storey building facade; b. receiving into the computer processing system, geolocated facade condition sensor data from one or more sensors; c. associating, in the computer processing system, the geolocated facade condition sensor data with one or more relevant map elements to populate the map with facade sensor data. 8. In one embodiment the map elements are discretised. 0069. In accordance with one aspect of the present invention there is provided a method for registering captured data from a plurality of sensors with a digital facade model of a target building, the method including the steps of, in a computer processing system:

[0009] 0070. recording geolocation data with the data from the vision sensors; a. extracting building facade pointcloud data collected by the plurality of sensors to provide pointclouds b. registering the pointclouds to combine them into one large point cloud of the building facade

[0010] 0071. In one embodiment the combined point cloud is compared with a master 3D model of the target building to remove erroneous noise points and reflections.

[0011] 0072. In accordance with still another aspect of the present invention there is provided a computer-implemented method for passively or actively recording multi-storey building facade condition data with a BMU-mounted condition monitoring apparatus, the method including the steps of: a. sensing facade condition data with one or more BMU-mounted sensors; b. receiving into the computer processing system, map data comprising a plurality of map elements relating to a multi-storey building facade;

[0012] 0073. receiving into the computer processing system, geolocated facade condition sensor data from the one or more sensors as the BMU moves across the surface of the facade;

[0013] 0074. associating, in the computer processing system, the geolocated facade condition sensor data with one or more relevant map elements to populate the map with facade sensor data.

[0014] 0075. In accordance with still another aspect of the present invention there is provided a computer-implemented method for passively or actively recording multi-storey building facade condition data with a BMU-mounted condition monitoring apparatus, the method including the steps of: a. sensing geolocated facade condition data with one or more BMU-mounted sensors as the BMU moves across a surface of the facade; b. transmitting across a computer network, the geolocated facade condition data to a computer processing system for associating with one or more relevant map elements to populate a map with facade condition data.

[0015] 0076. In one embodiment there is a frame provided to mount the sensors to the BMU.

[0016] 0077. In one embodiment there are frame legs to elevate the sensor bar above the BMU cradle.

[0017] 0078. In one embodiment the sensor bar is mounted to suspension cables.

[0018] 0079. In one embodiment there are included fasteners to fasten the frame to the cradle.

[0019] 0080. In one embodiment the fasteners fasten the frame to a rear face of the cradle.

[0020] 0081. In one embodiment there are included one or more legs to support the sensor mounting portion at the predetermined height above the cradle. The predetermined height is a height sufficient to be clear above the heads of cleaning operatives or other workers in the BMU cradle.

[0021] 0082. In one embodiment the sensor mounting portion is a sensor bar for mounting a plurality of sensors.

[0022] 0083. In one embodiment the sensor bar is configured to adjust the mounting position of a sensor on the bar. Say, along the bar, or on a gimbal.

[0023] 0084. In one embodiment the sensors include one or more from the group consisting of: RGB camera; LIDAR; thermal camera; depth camera; CCTV camera; ultrasonic.

[0024] 0085. In one embodiment the computer processing cabinet includes an altimeter. The altimeter may be mounted on the sensor bar.

[0025] 0086. In one embodiment the computer processing cabinet includes an IMU. The IMU may be mounted on the sensor bar.

[0026] 0087. In one embodiment the computer processing cabinet includes a GPS. The GPS may be mounted on the sensor bar or otherwise connected to the BMU, for example, on the cross bar of the BMU.

[0027] 0088. In one embodiment the computer processing cabinet is disposed in a BMU cradle cabinet, internal to the cradle walls.

[0028] 0089. In one embodiment the computer processing cabinet includes a computer processor. 0090. In one embodiment the computer processing cabinet includes a network module.

[0029] 0091. In one embodiment the network module includes an antenna and a mobile device signal router.

[0030] 0092. In one embodiment the computer processing cabinet includes a data storage device.

[0031] 0093. In one embodiment the computer processing cabinet includes a power bus for connecting to BMU power.

[0032] 0094. In one embodiment the computer processing cabinet includes a power supply in case of failure of BMU power.

[0033] 0095. In one embodiment there are provided signal and power cables from the computer cabinet to mains power associated with the building on which the facade and BMU are mounted.

[0034] 0096. In one embodiment the signal and power cables are integrated with hoist cables on the BMU.

[0035] 0097. In one embodiment the sensor bar connects directly to the BMU cradle wire rope connection brackets.

[0036] 0098. In one embodiment the method includes the step of receiving CAD model data into the computer processing system.

[0037] 0099. In one embodiment the method further includes the step of flattening the CAD model data to convert that CAD data to 2D map data so that it can more readily be associated with geolocated facade sensor data in the associating step.

[0038] 0100. In one embodiment the method further includes the global alignment of the digital 3D building model coordinate system so that it can more directly be associated with geolocated facade sensor data in the associating step.

[0039] 0101. In accordance with a further aspect of the present invention there is provided a computer-implemented method of conducting facade inspections from a mobile device or desktop, or immersive computing device or other computing device, the method including the steps of: a. receiving a populated map of geolocated facade condition sensor data associated with one or more relevant map elements; i. interrogating the map using the screen or other computing processor system inputs to investigate images or other facade condition data to check for facade faults.

[0040] 0102. In accordance with another aspect of the present technology there is provided an electronic computing device comprising a processor and a memory having instructions that when performed cause the electronic device to perform one or more of the methods described above.

[0041] ADVANTAGES

[0042] 0103. Embodiments of the present technology provide a hardware and software system that leverages existing BMU processes to record facade condition data and enable virtual desktop inspections that can replace in-person inspections.

[0043] 0104. Embodiments of the present technology augments window cleaning operations, since those cleaning operations should be, and are generally, conducted over an entire building facade. Hence, during a cleaning cycle the entire building facade is imaged once by embodiments of the present technology.

[0044] 0105. Embodiments of the present technology provide facade vision data which would ordinarily, by known technologies, require dedicated operations and / or path planning to control the places on the facade to which the system travels for inspection.

[0045] 0106. Embodiments of the present technology provide a combination of sensors and digital methods (software) to fuse different streams of data into one accurate geolocation estimate.

[0046] 0107. Embodiments of the present technology provide a plurality of sensors which include GPS, GPS+RTK, rope encoders, barometers, altimeters, LiDAR to facilitate improved geolocation.

[0047] 0108. Embodiments of the present technology provide improved geolocation by digital methods which include sensor fusion, computer vision, level detection, optical flow tracking, feature tracking. And comparing it to prior knowledge known of the building - taken from architectural drawings

[0048] 0109. Embodiments of the present technology provide a modular system, which in one embodiment allows cameras to be angled to view around facade features and capture details. 0110. Embodiments of the present technology provide accurate geolocation / localisation methods which allow facade condition data from vision sensors to be accurately geolocated in space, onto a 2D / 3D model, for ready browsing.

[0049] 0111. Embodiments of the present technology provide an independent system that only requires a power connection, either mains or onboard battery or capacitor, to function. This allows embodiments to be retrofitted onto virtually any existing BMU

[0050] 0112. Embodiments of the present technology provide a modular system which facilitates it being configured to suit a specific BMU and building facade.

[0051] 0113. Embodiments of the present technology provide an apparatus that can be split into two main components, separately mounted on the BMU, which facilitates in providing a balanced load on the BMU. This allows it to be adaptable to lighter BMUs which are more likely to be thrown off balance by a heavy mass on one side or another.

[0052] 0114. Embodiments of the present technology provide an apparatus mountable by nondestructive means - e.g., via clamps, through-bolts or clamping plates. Other embodiments or the same embodiments can also utilise destructive means such as penetrating fasteners or welding or other means to connect to the BMU.

[0053] 0115. To enable a clearer understanding, an embodiment of the invention will hereinafter be described with reference to the figures, in which:

[0054] 0116. Figure 1 is an isometric view of a BMU cradle with an apparatus for capture of facade condition data mounted on a rear wall thereof;

[0055] 0117. Figure 2 is a similar view of a different BMU with a similar augmenting system for capture of facade condition data, with a computer cabinet inside the BMU cradle;

[0056] 0118. Figure 3 is a front elevation view of the BMU with piggybacked facade condition data capture apparatus;

[0057] 0119. Figure 4 is a front elevation view of the BMU of Figure 2 with piggybacked facade condition data capture apparatus

[0058] 0120. Figure 5 is a side elevation view of the computer unit and data capture apparatus;

[0059] 0121. Figure 6 is a side elevation view of the computer unit and data capture apparatus of Figure 2;

[0060] 0122. Figure 7 is a front elevation view of the facade condition capture apparatus;

[0061] 0123. Figure 8 shows an isometric view of the Figure 2 embodiment without the cradle;

[0062] 0124. Figure 9 is a high level schematic view of the retrofit data capture system shown in Figures 1 to 3, showing the componentry and connections between each;

[0063] 0125. Figure 10 is a schematic flowchart of an embodiment of process to populate a 2D or 3D map using geolocated facade condition data;

[0064] 0126. Figure 11 is a detailed schematic view, more detailed, than that shown in Figure 5, of the data capture system shown in Figures 1 to 5

[0065] 0127. Figure 12 is a flow diagram showing steps undertaken by the computer processing system of an embodiment of the invention; and

[0066] 0128. Figure 13 is a graphical user interface of the browsable final product of an embodiment, showing browsable accurately gelocalised computer images of a facade. 0129. Referring to the drawings there is shown a system for capture of visual facade condition data, the apparatus for fitment to a vehicle such as for example a BMU, which is configured to traverse a facade. The facade may be of a building, but it may be of a bridge, or an underpass, or a tunnel or like facade.

[0067] 0130. The system includes an apparatus generally indicated at 10 which includes: a mounting frame 20 for mounting one or more sensors and a computer processing cabinet 50 to an outside portion of a building maintenance unit (BMU) 30. The BMU 30 includes a cradle portion 32. The computer processing cabinet 50 is mounted to the outside of the BMU 30 so that it is clear of cleaning and other maintenance operations conducted by operators on the BMU 30.

[0068] 0131. The frame 20 includes a base portion 22 for mounting to the cradle portion 32, and a sensor mounting portion 24 for mounting one or more sensors 40 - 45. The base portion 22 includes one or more legs 25 which extend upwardly to elevate the sensor mounting portion 24 to a predetermined distance above the cradle portion 32. The reason for the elevated sensor mounting portion 24 is to position the one or more sensors 40 distal the cleaning and maintenance operations conducted by operators on the BMU 30.

[0069] 0132. The one or more sensors 40 mounted on the sensor monitoring portion 24 are for sensing facade (not shown) condition. The computer processing cabinet 50 is in electrical and data communication with the one or more sensors 40, the computer processing cabinet 50 being mounted on the mounting frame 20 at the base portion 22 of the legs 25.

[0070] 0133. The frame 20 is fastened to the cradle 32 by fasteners 33 in the form of pipe clamps 34.

[0071] 0134. The frame 20 includes three legs 25 to support the sensor mounting portion 24 at the predetermined height above the cradle 32. The predetermined height is a height sufficient to be clear above the heads of cleaners or other occupants in the BMU.

[0072] 0135. The sensor mounting portion 24 is a sensor bar 26 for mounting a plurality of sensors. There are two sensor mounting bars 26 provided in the embodiment shown at Figure 1. The sensors 40 include: RGB camera 41; LIDAR 42; thermal camera 43; depth camera 44; CCTV camera 45. These sensors are directed toward the facade to receive visual and other facade condition data from the facade in accordance with their design and function. 0136. The computer processing cabinet 50 is connected to the sensor bar 26 to receive sensor data from the sensors 40 but also, to supplement the sensor data, and includes an altimeter 52, IMU 54, GPS 56.

[0073] 0137. The computer processing cabinet 50 also includes a computer processor 57, data storage device 58, and a network module 59. The network module includes an antenna 55 and a mobile device signal router 53 to facilitate processed data transfer to a cloud infrastructure 60. The computer processor 57 could be any kind of processor (such as that found in a desktop or laptop, micro or mini computer or tablet), and the processing or control could be shared with one or more cloud-connected devices via the network module 59, including via mobile application 70 running on mobile device 72.

[0074] 0138. The computer processing cabinet 50 further includes a power bus 62 for connecting to BMU power. Also, the computer processing cabinet 50 includes an onboard power supply 63 in case of failure of BMU power. There is provided an emergency switch 64 for emergency power cutoff. There are also provided signal and power cables 65 from the computer cabinet 50 to mains power associated with the building on which the facade and BMU 20 are mounted. The signal and power cables 65 are integrated with hoist cables on the BMU 20 for increased safety of BMU workers, to reduce tangling and catching during raising and lowering of the BMU 20. The sensor bar 26 connects directly to a cradle wire rope connection brackets (not shown) on the BMU 20.

[0075] 'i’!

[0076] 0139. The BMU 20 is deployed to move over a building facade to conduct regular maintenance and cleaning. Piggybacking the apparatus to the BMU 20 reduces expenditure on separate drive and control systems to optimise the facade inspection process.

[0077] 0140. One of the advantages of the embodiments of the present technology is that, although the sensor data may be received in no particular order and without any particular plan that may suit the complete collection of facade condition sensor data, the data may be mapped to a building facade drawing. Generally speaking, cleaning processes do cover an entire facade surface so we receive a comprehensive scan of the entire facade from the cleaning process. The imaging and geolocalisation process is described below.

[0078] 0141. Furthermore, a BMU 20 will operate closer to the facade than other known sensor-laden machines like drones, producing higher quality data that is able to capture and diagnose more facade observations. High resolution image data can be efficiently captured and rapidly geolocalised.

[0079] 0142. In addition, the proposed data collection system harvests facade condition data passively or actively during other BMU operations such as routine window cleaning. Since cleaning occurs roughly quarterly, this augmentation of the the BMU produces more data at shorter intervals in time, which enables greater traceability of the building facade. These cleaning operations should be, and are usually, comprehensive and cover the entire facade, so a complete building snapshot can be obtained.

[0080] 0143. Additionally, the system is proportionally cheaper than other systems, since the system runs passively, which is to say, in the background of a BMU cleaning operation. This facilitates reduced labour costs because there is no specialist needed on site, and fewer costs due to delays. The geolocated vision data is readily available for people or digital processes to investigate and make observations about the facade condition, such as to perform a virtual facade inspection. As well, it can also be used to assist traditional inspections as a sanity check or to validate a hypothesis before / after a site visit - making the visit more efficient.

[0081] 0144. A further advantage is that a separate data harvesting run need not be scheduled. The facade condition is merely checked at the same time as another operation which requires the use of similar real estate.

[0082] 0145. A still further advantage is that the data produced by the system is interoperable not only across disciplines but backend digital architectures also. The data captured may be fed into other defect detection algorithms such as those for drones. Similarly, the process, backend or logistical, are also interoperable, and can be applied to comparable data not collected by the system.

[0083] 0146. The following additional advantages are also provided by embodiments of the present technology: a. Modular, adaptive payload to retrofit to existing infrastructure b. Option to inspect facade every time the BMU is in operation c. Sensor suite to create “digital twin” of the building facade d. Improved safety by limiting the time that building facade inspectors are required to be at risk of fall e. Automated warning of facade defects for prompt reactive maintenance f. Predictive warning of facade defects for proactive maintenance g. Reduced facade defect risk h. Lifetime facade history logged, and improved defect traceability i. Modular, payload can be deeply integrated into BMU, including BMU cradle structure and BMU computer. j . Applicable to different industries and payload delivery methods.

[0084] 0147. Further, there are advantages relating to improvements to the BMU: a. Traceability of BMU use; for example, its movements across the facade are all logged, as well as the times that it has been operating, by recording that data and adding it to the discretised map; b. Traceability of works conducted from the BMU by CCTV recording. 0149. Referring to the drawings and the description, now there is set out a description of a method for which the system may be deployed. The method is for the active and / or passive recording of multi-storey building facade condition data. Generally, passive collection refers to augmenting a BMU maintenance or monitoring operation by piggybacking on the BMU. The reason one embodiment is described as passive is because there is no active control over the route taken over the facade when harvesting data relating to the condition of the facade. The harvesting route taken is dependent on the route taken during a maintenance or cleaning operation conducted by maintenance operators from the BMU. A dedicated data harvesting route can be run to capture a specific area of the facade.

[0085] 0150. The method includes the steps set out below.

[0086] 0151. The processor 57 is caused by the software to commence recording vision data. This could be when the BMU is detected to be moving across a building facade (not shown). This is step 500 in Figure 10.

[0087] 0152. The processor 57 is caused to record geolocated facade condition vision data with one or more BMU-mounted sensors 40. This is step 510 on Figure 10. The sensors 40 are disposed on a frame at a height above, and behind, the heads of cleaning or maintenance workers standing in the BMU cradle 22, so that the cleaning operation is not interrupted by the sensing operation, and vice versa.

[0088] 0153. The processor 57 is caused to receive into the computer processing system 57, geolocated facade condition sensor data from one or more vision sensors 40. This is step 520 in Figure 10.

[0089] 0154. After that, the method involves in the processor 57, associating the sensor data onto a stored map of a facade which may be discretised. This is step 530 in Figure 10.

[0090] 0155. The method is finally complete when the map is populated in this way, sensor data element by sensor data element.

[0091] 0156. It is possible that not the whole map is populated. There may be used interpolation methods to infer the condition of those missing portions of the map. There may be machine learning methods deployed to make those inferences. The location of the unpopulated map elements may be recorded and an alert list generated. If the list is large enough, or a contiguous area of elements large enough, the computer processing system 57 causes an alarm to be sounded, and those areas marked for future or immediate data capture, either by manual inspection or by more actively controlled BMU or by drone.

[0092] 0157. If the discretised fagade map is not stored on the computer processing system 57, it may be generated by it. The first step in that process is to receive into the computer processing system 57, CAD model data of the building (not shown) into the computer processing system 57 at step 600 in Figure 10.

[0093] 0158. Then, the processor causes the CAD data to be flattened to a 2D map to denote the surface of the fagade. This is step 610 in Figure 10. In some embodiments 3D maps are also used, and in that instance there is no flattening step executed.

[0094] 0159. After this, the computer processing system causes the 2D (or 3D) map to be broken into discrete elements. This is step 620 in Figure 10.

[0095] 0160. At this stage, the discretisation is complete and the map is stored on the storage device 58 or the cloud. Alternatively, the map is transmitted live to a server. The association step follows at 530 as described above.

[0096] 0161. In the computer processor itself, there is a method of passively or actively recording the fagade condition data onto a discretised fagade map, discussed as set out below. There is a computer-implemented method for passively recording and geolocation tagging large scale asset condition data for assets such as multi-storey building facades, rail corridors, tunnels, the method including the steps of, in a computer processing system: a. receiving into the computer processing system, map data comprising a plurality of discretised map elements relating to a multi-storey building fagade (step 620); b. receiving into the computer processing system, geolocated fagade condition sensor data from one or more sensors (step 510) taken from an unrelated maintenance process; c. associating the geolocated fagade condition sensor data with one or more relevant discretised map elements to populate the map with fagade condition sensor data (step 530).

[0097] 0162. The system provides the ability for a computer-implemented method of conducting fagade inspections from a mobile device or desktop, or immersive computing device or other computing device. That method includes the steps of: a. receiving a populated model 550 of geolocated facade condition sensor data associated with one or more relevant discretised map elements and then interrogating the model 550 using a touch screen or other computing processor system input to investigate machine vision or other facade condition data to check for facade condition or to review the surface.

[0098] 0163. The captured data is stored in cloud storage as discussed below. a. There is storage of image, video, pointcloud and mapping data in a networked server store;

[0099] The configuration of database management system that exposes an interface to access and organise this data; b. In one embodiment there is a data management strategy deployed, wherein each data point is assigned a level to facilitate movement of captured data between hot, cold and archival storage, as well as retention / deletion of data. c. Provision of API services to perform CRUD (create, read, update, delete) operations on data which may be organised by type / purpose i. BMU tracking ii. Video streaming iii. Defect analysis

[0100] 0164. The computer processing system processes harvested and processed data in the cloud. There is provided an orchestration strategy of cloud compute resources to support geolocation methods, and / or other services required by it: a. Stitching of machine vision data (images, pointclouds) together to provide panoramic views of facade b. Identification / rej ection of reflections to distinguish facade elements. c. automated defect detection

[0101] 0165. The system further includes a digital system including several digital features / processes that utilise the data collected by the system to provide value for Building Owners / Clients. a. BMU Travel History: Utilising the digital model of the building and the refined geolocation data to create a visual representation of where the BMU has been. b. One method of visual representation is using a heatmap to highlight the building facade at different gradients based on how long it has been since the last time the BMU has been used at this location. c. The CCTV camera of the system may be used to identify the purpose of works in the BMU cradle. E.g., through identification of window cleaning equipment through object detection, it may be inferred that window cleaning is being undertaken. d. An alert may be sent out to owners when an area of the facade is 'overdue' for cleaning, inspection or maintenance works.

[0102] 0166. There is also provided a BMU Operator Monitor: With data collected from the CCTV camera and the geolocation of the system, this is a process to create traceability of the works conducted from the BMU. The method for this includes: a. Associating the CCTV data with the refined geolocation of the system. b. Using Al computer processes to analyse the imagery of the BMU, the works conducted may be identified either autonomously or manually. i. One method of identifying the works from the BMU may be through object detection of the equipment onboard. E.g., through identification of window cleaning equipment through object detection, it may be inferred that window cleaning is being undertaken. ii. Another method may be through manual assessment of the imagery and assigning an appropriate label.

[0103] 0167. Another, similar method of another embodiment is shown at Figure 12f. The computer processor 57 is caused to generate fused geolocation data (step 800) which is geolocation data of increased accuracy.

[0104] 0168. The fusing step 800 includes a digital method of combining, in computer processor 57 (which may be cloud connected to conduct some, most, or all the processing offsite): a. geolocation sensor data (810) from onboard geolocation sensors (852, 854, 856) with matched feature geolocation timeseries data (826) which is: b. timeseries onboard vision data of visible features (832) combined with feature data (831) discretised or extracted from architectural drawings (850)

[0105] 0169. The matched feature data 826 can be obtained by suitable digital methods, including scheduled traversal, feature counting and the like, which occur within the processor.

[0106] 0170. Feature extraction includes a digital step (which is to say it occurs in a computing processor) in which features 830 from the architectural drawings 850 are identified, by a computing processor.

[0107] 0171. A database 831 of geolocated architectural features extracted from this process is created by the computer processor. Later, this database is used in producing matched / correlated geolocation timeseries data 826.

[0108] 0172. In parallel, a timeseries of visible features 832 is created by the processor, which is facade condition data from computer vision sensors 841, 842, 843, 844 which has been processed by digital detection methods including Artificial intelligence, machine learning, optical flow and other numerical methods.

[0109] 0173. The vision data 845 is computer vision sensor data from 2D or 3D sensors (841, 842, 843) and is used to provide the timeseries of visible features 832.

[0110] 0174. The processor 57 includes an additional step where the data points are autonomously assessed / evaluated by a digital process using Artificial intelligence or machine learning, and then verified by a discipline expert. So, the processor 57 is an electronic processing device and / or computer processor, or it may be part of a system of networked processors, which is or are configured to deploy statistical tools, using one or more numerical regression analyses, to: identify features from architectural documentation including drawings; to detect features from computer vision data; to improve geolocation data from a range of localisation sensors, and to fuse the detected feature data with the improved geolocation data.

[0111] 0175. In any event, at least for the assessment and identification of architectural features, the computer processor 57 or networked system of processors conducts this analysis by powering at least a machine learning feature extraction engine 835 and feature detection engine 845, a matching engine 827, a sensor fusion engine 815 feature quality assessment engine 890, a fusion engine 805, with the optional support of a data engine (not shown), a mapping engine (not shown), a data quality engine (not shown) and an optical flow engine (837). As discussed below, while a single processing device 57 is shown in the Figures, it will be appreciated that steps may be performed by multiple processing devices. Moreover, reference to an “engine” includes conceptual reference to a set of functional tasks / instructions, and thus the functionality provided by an “engine” may also be distributed among multiple processing devices (real and / or virtual).

[0112] 0176. In operation, the feature extraction engine 835 is fed data from the architectural documentation engine 855 to identify architectural features. While feature extraction is described in this example, in other examples generating features in the processor 57 may not be required, for instance, in an example using unsupervised machine learning. In this regard, features may be input into the extraction engine 835 which uses an unsupervised machine learning algorithm to extract features from the drawings.

[0113] 0177. In some embodiments, a reference model includes an at least partially trained machine learning model, and thus one of the steps undertaken by the processor 57 may include training the reference model. As will be appreciated, training a machine learning model may be performed in any suitable manner including online or offline. Thus, feature extraction and detection, matching and sensor fusion and quality assessment may be performed in any suitable manner, including online - where it may be performed during run-time in any suitable order. In this regard, the reference model could be updated during run-time as additional data feeds and metrics are determined.

[0114] 0178. In the disclosed embodiment, the feature extraction, feature identification, sensor fusion matching and feature quality assessment steps includes training the reference model offline using the architectural drawings 850 and unstructured image sensor data 840. Accordingly, the matching engine, and other engines may be operable outside of run-time and / or on a remote processing device. In this regard, while training the reference model may consume considerable computational power, this can be done prior to (or in parallel with) run-time assessments. Thus, run-time assessments eg step 827 or 890 could be performed utilising significantly less processing power, and in some instances, in real-time. In one example, the machine learning reference model may include one or more regressors. Thus, they are compact when stored in memory, and require less computing power when performing predictions using the regressors.

[0115] 0179. As discussed above, the processor 57 may have access to learned data, or has spent some time itself learning to identify facade features from various ones in a networked database. The processor 57 may havealso spent time learning to refine and match geolocation data with raw and other computer vision data, to identify facade quality issues and facade problems from training exercises relating to computer vision observations. The vision observations may be from LIDAR, ultrasound, cameras, and other sensors.

[0116] 0180. The processor 57 is caused then to assess the computer vision data and adds data and information about the vision data that was recorded at a specific location and what is of interest at those points.

[0117] 0181. The processor 57 is caused to make observations relating to facade features generally and any defects identified and assigns a risk / urgency rating and a geolocation description (for example, floor 10, drop 3.

[0118] 0182. The processor 57 in a display engine causes the database of recorded visual features to be accurately placed on an architectural drawing or map and then allows browsing of features on the map.

[0119] The facade risks are assessed / evaluated by the processor 57 and then may be verified by an operator using the browser.

[0120] 0184. Figure 13 shows a graphical user interface of the browsable final product of an embodiment, showing browsable accurately gelocalised computer images of a facade. The computer processor 57 is caused to display. There is shown a browsable map at 999, an interface 998 to record observations on facade, and a display of digitally detected and located facade elements at 997. In the display 997 there can be checked the quality of the facade since the vision is actual vision data recorded by the vision systems 840. There is facade condition metadata shown at 996. 'i’!

[0121] 0183. It can be seen that the method of the preferred embodiment includes the steps of receiving data into a processor relating to architectural facade features 850, extracting onto a digital discretised map those visual features in step 830, assigning map location data to those features 830, detecting and matching / correlating those features with computer vision data 845 processed from sensors 841, 842, 843, 844, and then assigning an accurate geolocation for each packet of correlated computer vision data 845, so that each facade feature captured by the visual sensors can be accurately geolocated.

[0122] 0184. At that point, the feature data captured from a sweep of the facade can be readily searched and / or browsed by scrolling across a face of the architectural drawing to which has been added the captured visual data from the sensors 841, 842, 843, 844.

[0123] 0185. It will be understood to persons skilled in the art of the invention that many modifications may be made without departing from the spirit and scope of the invention.

[0124] 0186. It is to be understood that any prior art publication referred to herein does not constitute an admission that the publication forms part of the common general knowledge in the art.

[0125] 0187. In the claims which follow and in the preceding description of the invention, except where the context requires otherwise due to express language or necessary implication, the word “comprise” or variations such as “comprises” or “comprising” is used in an inclusive sense, i.e. to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments of the invention.

Claims

1. A system for fitment to a vehicle configured to traverse a facade, the system being configured to capture visual facade condition data, the system including: one or more mounts suitable for mounting to a building maintenance unit (BMU); a plurality of sensors mounted on the mounts, the sensors including: vision sensors; and geolocation sensors; a computer processing cabinet in electrical and data communication with the plurality of sensors, the computer processing cabinet being mounted on the BMU and including a processor, wherein the processor is configured to fuse matched architectural feature data, with geolocation sensor data, wherein matched architectural feature data is processed data that matches facade feature data identified by the vision sensors with known facade feature data from architectural documentation data; and geolocation sensor data is raw or processed data from the geolocation sensors.

2. The system in accordance with claim 1 wherein the matched architectural feature data is obtained by a feature counting step in the processor, in which extracted features from the architectural documentation data are identified, and correlated with a timeseries data set of detected visible features from a computer vision data set.

3. The system in accordance with claim 2 wherein the detected visible features are detected by steps in the processor taken from the group of steps consisting of: computer vision, artificial intelligence, and optical flow.

4. The system in accordance with any one of claims 1 to 3 wherein the vision sensors are 2D or 3D sensors including one or more from the group consisting of: RGB camera; LIDAR; thermal camera; depth camera; CCTV camera; ultrasonic wave generator.

5. The system in accordance with any one of claims 1 to 4 wherein the geolocation sensors include an encoder, altimeter, barometer, an IMU and / or a GPS locator.

6. The system in accordance with any one of claims 1 to 5 wherein the raw or processed geolocation data is fused by a method including the step of sensor fusion, and / or regression.

7. The system in accordance with any one of claims 1 to 6 wherein the method includes the step of receiving and / or preparing a known facade feature database from architectural documentation by extracting facade features coupled with a geolocation of the extracted facade feature.

8. The system in accordance with claim 7 further including the step of preparing a discretised map of extracted facade features from the architectural documentation data.

9. The system in accordance with any one of claims 1 to 8 further including the step of preparing a browsable map of facade condition data recorded by the vision sensors, the browsable map being populated by the vision sensor data in substantially accurate geolocations.

10. The system in accordance with any one of claims 1 to 9 further including the step of evaluating with the processor, the observed facade condition, and assigning risk ratings to various areas of the facade, for further inspection.

11. The system in accordance with any one of claims 1 to 10 further including the step of receiving the architectural documentation data into the processor from a database.

12. The system in accordance with any one of claims 1 to 11 further including the step of preparing a timeseries of visible architectural features for use in producing matched geolocation timeseries data.

13. The system in accordance with any one of claims 1 to 12 wherein a frame is provided to mount the sensors to the BMU.

14. The system in accordance with any one of claims 1 to 13 wherein there are frame legs to elevate the sensor bar above the BMU cradle.

15. The system in accordance with any one of claims 1 to 14 wherein the sensor bar is mounted to suspension cables.

16. The system in accordance with any one of claims 1 to 15 wherein there are included fasteners to fasten the frame to the cradle.

17. The system in accordance with any one of claims 1 to 16 wherein the fasteners fasten the frame to a rear face of the cradle.

18. The system in accordance with any one of claims 1 to 17 wherein there are included one or more legs to support the sensor mounting portion at the predetermined height above the cradle.

19. The system in accordance with any one of claims 1 to 18 wherein predetermined height is a height sufficient to be clear above the heads of cleaners or other workers in the BMU cradle.

20. The system in accordance with any one of claims 1 to 19 wherein the sensor mounting portion is a sensor bar for mounting a plurality of sensors.

21. The system in accordance with any one of claims 1 to 20 wherein the sensor bar is adjustable in height, and / or includes mounts that are adjustable to adjust the mounting position of a sensor on the bar.

22. The system in accordance with any one of claims 1 to 21 wherein the computer processing cabinet includes a power supply in case of failure of BMU power.

23. The apparatus in accordance with any one of claims 1 to 22 wherein the signal and power cables are integrated with hoist cables on the BMU.

Images