Device and method for the hybrid-dynamic representation of an infrastructure element

Abstract

The invention relates to a device and a method for the hybrid-dynamic representation of an infrastructure element at a location (2). The method comprises the steps of: A) using a camera to generate a first data set representing a panoramic view image (4) of the infrastructure element (1) at a location (2) within the infrastructure element (1); B) generating a second data set representing an image of a CAD panoramic view which comprises CAD data of the infrastructure element (1) representing surfaces of the infrastructure element (1) visible from the location (2); C) aligning the first and second data sets in an identical orientation; and D) displaying the aligned data sets on a display device.

Description

[0001] Description

[0002] Device and method for hybrid-dynamic representation of an infrastructure facility

[0003] The present invention relates to a device and a method for the hybrid-dynamic representation of an infrastructure facility. In particular, the present invention relates to a data-efficient, manageable hybrid representation of a photorealistic image of an infrastructure facility and a corresponding CAD representation.

[0004] A personal inspection of a property, a factory hall, or similar facility can sometimes involve significant travel costs, time expenditure, and potential disruption to operations. With the increasing availability of optical sensors (cameras, 3D cameras, laser scanners, etc.), it is generally possible to conduct a purely virtual inspection.

[0005] This allows a wide range of impressions of the property to be obtained, which can be taken into account when subsequently initiating measures. In particular, data from a CAD representation, which can, for example, define a planning status or existing data, can be compared with a construction actually created on site in order to conduct the building acceptance or at least identify errors / deviations. Furthermore, the CAD data can be used to determine important information for system expansions, modernization measures, the inspection of hazardous areas, and any optimization needs. The handling of CAD data is becoming increasingly challenging, especially with larger models, and requires powerful software and hardware. As a result, the CAD data or hybrid representations generated based on it are only available on a few computers within a company.

[0006] Software packages such as web browsers, however, are not suitable for handling the data volumes and their representation without resorting to streaming methods. However, the latter require resource-intensive calculation of the CAD data representation on the server side.

[0007] Based on the aforementioned prior art, it is an object of the present invention to provide a virtual tour of an infrastructure facility with a comparison of the associated CAD data while requiring minimal hardware and software requirements. The aforementioned object is achieved according to the invention by a method having the features of claim 1 and by a device having the features of claim 12. The subclaims describe preferred developments.

[0008] According to the first aspect, a method for the hybrid-dynamic representation of an infrastructure facility (e.g., a property, a building, a factory hall, a parking lot, a multi-storey car park, a production facility, or similar) is provided. The representation is created from a location, which can also be understood as the "viewer location." The location is therefore the position that determines the perspective on the hybrid-dynamic representation to be generated. The representation can be described as "dynamic" because the user / viewer can adjust both their location and their perspective / orientation as desired. In other words, the user can "dynamically" select one of many virtual locations or viewing positions and also dynamically adjust or "pan" the "viewing direction" within the representation. The representation can be described as "hybrid" because both CAD-based content and content generated using sensors (e.g.,Content captured by optical or radar sensors can be displayed spatially corresponding to each other for the selected location and orientation.

[0009] In other words, a "hybrid" representation is created as needed from a planning data set and an image of reality. Even though the subsequent steps can then be carried out for multiple locations that differ from the location currently being considered, only one location is referenced in the example. In a first step, a mobile camera is used. The camera can, for example, be carried by a user in the hand, on the body, or on a cart. Of course, the camera can also be mounted on a self-propelled base / carriage or an aircraft such as a drone. The camera generates an initial data set that creates a 360° panoramic image of the infrastructure facility from the aforementioned location. The 360° panoramic image thus encompasses at least a 360° panning range closed in the plane and determines the optical data required for this.Any imaging process that can reproduce reality in image form is suitable as a recording device. This includes, for example, thermal imaging cameras, laser images, radar images, ultrasound images or optical recordings such as 2D / 3D cameras and ToF cameras. This recording device or this recording technology can be high-resolution, in particular ultra-high-resolution. The panoramic view can in particular also include the ground and / or the upward solid angle range, so that up to 360° solid angles in all directions can be recorded by the camera and represented by the first data set. In a further step, a second data set is generated which represents a CAD (Computer Aided Design) panoramic view resulting from a computer-aided, spatial, digital (hereinafter "CAD") model of the infrastructure facility at the site.Viewed from the location, the image represents the visible areas of the infrastructure facility or the elements contained in the CAD data model. In particular, only the amount of data relating to the location is created which can be captured by free optical axes between the location and the structure of the infrastructure facility. This significantly reduces the amount of data in the CAD model to those areas closest to the viewer or the location. Thus, for non-(partially) transparent areas, exactly one data point is obtained for each spatial direction, which is depicted in the CAD panoramic view. Positions behind these points are not required for the depiction of the CAD panoramic view, and in particular are not included in the second data set.In other words, the second data set represents a visual representation of the CAD data from the location's perspective and, in particular, lacks further information from the CAD model. Such a view can also be determined as a spherical projection from the CAD model or as a cube projection from the CAD model. In a third step, the first and second data sets are compared for identical alignment. In other words, assignments are created between the first data set and the second data set, resulting in a multi-part data set containing, on the one hand, the visual representation of the infrastructure facility and, on the other hand, the CAD data-based representation of the infrastructure facility.The result is a multi-layered (optical as well as design data) image of the infrastructure facility similar to a panoramic photograph. When the synchronized data sets are subsequently displayed on a display device, the user can freely select their perspective from the location or pan through the panorama, obtaining a visual representation, a CAD-based representation, or, in particular, a hybrid representation of the infrastructure facility. In particular, the user can display or hide individual CAD elements (including additional information (including sensor-based or computer-aided information)), or overlay them onto the visual representation of the infrastructure facility, filtered by category (e.g., information categories, material categories, component categories, categories of different trades (electrical, metal construction, hydraulics, drywall, etc.)).For example, the elements can be predefined data components within a shared data file, or they can belong to respective, optionally overlaid data files. This allows for selective review of the differences that arose during planning and implementation without the user ever having to physically enter the infrastructure facility. In the next step, the reconciled data sets are displayed on the display device without requiring the user to use a particularly powerful hardware / software combination. Instead, a browser-based display (e.g., the web browser Safari, Google Chrome, Microsoft Internet Explorer, or similar) can be used.This has the further advantage that a low bandwidth is required to provide the hybrid-dynamic representation according to the invention, and special knowledge of using the CAD program, as well as its installation and data acquisition, is not required. In this way, many parallel accesses to the hybrid-dynamic representation provided by the invention can be made using simple IT devices and untrained personnel. The level of information within a company regarding the conditions within the infrastructure facility can thus be significantly improved without requiring unreasonable investments.

[0010] Preferably, the representation of the CAD panoramic view can comprise a 360° view, which relates to all solid angle ranges in the plane or in elevation. The projection of the 3D data onto a spherical surface and / or a cube surface can be carried out according to the prior art. However, in the manner according to the invention, a comparison of the two data sets is required to ensure identical alignment. The representation of the CAD panoramic view is preferably carried out in an image format optimized for web browsers, in particular WebP (pronounced "Weppy"), in order to support resource-efficient processing. When the CAD panoramic view is created, additional information on the represented data sets, such as their file names, creation or modification dates, or data sources, is optionally stored so that the user can display the contained information on the date, data status, and version as needed, optionally in the browser itself.

[0011] The panoramic view can comprise a 360° image of the location created using a mobile camera. The mobile camera can be mounted on a vehicle or worn on the body, for example. The mobile camera can also comprise multiple lenses or multiple individual cameras, the images of which are combined to create a 360° view. The first data set can be supplemented, in particular, by distance data, which can be determined, for example, using a laser. In other words, the camera can be enhanced with a rangefinder, which combines the spatial points visible from the location at any given time into an optical image of the infrastructure facility that contains the distance data. The distance information can be determined using laser Doppler scanning. It can be used to compare the data sets with regard to orientation.In this way, precise measurements and other information regarding the infrastructure facility can also be determined from the combined data set.

[0012] Optionally, a user request can also be determined, which can be expressed, for example, through user input via a user interface. The user request can refer to the display or hiding of either the entire panoramic view image and / or the CAD panoramic view or its components / portions. In response, the desired data set or the portions it contains are hidden or hidden. For this purpose, the user can, for example, specify the transparency levels of individual portions or enable or disable the display of individual portions.

[0013] If position data or coordinates of individual (measurement) points of a distance scan are available within the display, for example due to a previously performed laser scan, the user can also express a wish to display or hide the coordinates of a measurement point and / or the distance of the measurement point to a second measurement point or to the location within the panoramic view image. For example, the user request can result in a connection between the two measurement points being displayed and labeled with the current distance between the two measurement points. On-site surveying is therefore no longer necessary. This information can be determined based on laser scan data acquired during the optical image capture of the infrastructure facility.

[0014] The alignment of the first and second data sets for identical alignment can be performed automatically, providing corresponding views of the data sets. For example, reference points contained in the CAD panoramic view data and the panoramic view image can be compared and aligned in the display. For example, the user can define a point in the CAD data that is marked with a label or other reference object before the infrastructure facility is optically imaged using the camera. Automatic identification of the imaged label / reference point can then be used by a computer to automatically match the point with the predefined data point within the CAD panoramic view.However, matching can also be performed using optical pattern recognition or using information about the respective component contained in the CAD data and automatically identifying the corresponding property of the photographed real model. For example, a CAD component that has the attributes "bare metal," "concrete," or "lamp" can be automatically identified in the optical data by automatically determining the color, surface structure, or light intensity of the respective component. Thus, manual input for matching is not required.

[0015] Preferably, a user input can be determined that identifies a location in the infrastructure facility. For example, the user can click on an aisle or a location contained in the aisle in an overview display. To do so, they can, for example, move a cursor to the location or a display unit representing the location and confirm (click). In response, it is determined that the user wishes to view the infrastructure facility with respect to this location, and a data set comprising the first data set and the second data set is loaded from a server via a WLAN connection and / or an internet connection.At this point, the data regarding additional locations does not yet need to be loaded using the user's device, resulting in significantly lower bandwidth required for handling the data packets and significantly lower cost and energy consumption for the hardware / software combination used by the user. In particular, mobile devices can be used to support the display in the manner described in the invention. Examples include tablets, laptops, smartphones, smart wearables (AR glasses, etc.).

[0016] The information contained in the second data set from the CAD data model can refer to different categories of infrastructure components. For example, concrete structure / building structure, electrical components (pipes, units, lights, etc.), fresh water / wastewater / fresh air / exhaust air, heating / air conditioning systems, ladders, and walkways can be defined as categories that can be displayed or hidden at the touch of a button. Optionally, the transparency levels of the categories and / or both data sets can be adjusted (e.g., using a slider) in relative or absolute steps or continuously.

[0017] As stated at the beginning, the basic steps explained in connection with a first location within the infrastructure facility can be repeated with regard to a further location or many further locations until the desired parts or the entire infrastructure facility have been recorded in the manner according to the invention. The user or users can now optionally use a representation of the locations to load and display respective views of the infrastructure facility. The locations can have a distance from one another which is selected taking into account resolution and effort. For example, the distance between respective immediately adjacent locations or locations closest to one another can be between 50 cm and 5 m, in particular between 1 m and 3 m, and very preferably between 1.2 m and 2 m.The aforementioned dimensions have proven to be advantageous in order to provide a comparatively small amount of data and a comparatively high utility value for the user.

[0018] According to a second aspect of the present invention, a device is proposed which is capable of carrying out the method according to the invention. In particular, the device is configured to process the CAD data in the form of the CAD model or a preprocessed stage of the data as well as the image data of the image captured by the camera. For this purpose, the device has a data input, an evaluation unit, and a data output. The evaluation unit can, for example, have a programmable processor which is capable of sending the data sets, which have been compared with one another with regard to their alignment, to a user who has submitted a corresponding request via the data output. The device can, for example, be configured as a server which is configured via an intranet or internet to handle data and process user requests.In this way, the device is designed to implement the features, feature combinations, and the resulting advantages of the method according to the invention in a manner that is clearly evident, so that reference is made to the above explanations to avoid repetition. The figures show:

[0019] Fig. 1 is a perspective view of a recording device carrier which photographs a marked infrastructure facility by means of a camera;

[0020] Fig. 2 is a schematic representation of components for data processing which, in conjunction with an embodiment of a device according to the invention, enable the execution of a method according to the invention;

[0021] Fig. 3 shows a representation created according to the invention with dimension lines; Fig. 4 shows an alternative representation to Fig. 3 with a transparency control and increased CAD components;

[0022] Fig. 5 shows an alternative representation to Fig. 4 comprising further control elements for influencing the representation; and

[0023] Fig. 6 is a flowchart illustrating steps of an embodiment of a method according to the invention.

[0024] Fig. 1 shows a user 20 of a recording device, who could also be referred to as a "recording device wearer", who records an infrastructure 1 in the form of a column with an optical marker 11 using a camera 7 and a scanner 8. The camera 7 is mounted on a frame which the user 20 carries on his shoulders. The user 20 measures the infrastructure 1 from different positions, which are referred to as "locations" in the context of the present disclosure, since the user 20 remains at these positions to allow the camera 7 and the scanner 8 to scan the surroundings. The data collected in this way using the camera 7 and the scanner 8 are immediately buffered and sent to a data storage device available in the network the next time the network is connected.

[0025] Fig. 2 shows a memory 14, which temporarily stores the data from the scanner 8 shown in Fig. 1 and the camera 7. Via data lines and a data input 16, the data is fed to a stationary server 13 with an evaluation unit 17. Via another data memory 15, the data input 16 receives CAD data from the data stored in the memory 14. The evaluation unit 17 is able to process the CAD models received from the memory 15 in a location-specific manner. In other words, for each location stored in the data of the memory 14, a respective CAD panoramic view is created and a simplified image of the same is created. In other words, the CAD model is evaluated in such a way that the optical appearance at the respective location is produced as a panoramic image. The remaining data concerning

[0026] Model components that are not visible from the location are removed from the image. The evaluation unit 17 is then configured to combine the two data sets (image of the CAD panoramic view and panoramic view image) in such a way that an identical alignment of the two data sets is achieved. In other words, corresponding representations of infrastructure components are moved to the same spatial position, resulting in a multi-layer panoramic photo or panoramic model.If a user (not shown) requests a hybrid-dynamic representation of the infrastructure facility according to the invention at a predefined location on the computing unit provided via the Internet 10, comprising the display device 16 in the form of a screen, the evaluation unit 17 transmits the compared data sets via the data output 18 and the Internet 10 for displaying the same on the display device 6. The data sets are so small that the processing, viewing, and interaction of the user with the display can take place in a browser 9. In this way, only low demands are placed on the software and hardware of the user (viewer), and many employees / participants can use the results created according to the invention.

[0027] Fig. 3 shows an exemplary representation of a factory hall as infrastructure 1, which contains components of a panoramic view photograph 4 and an image of a CAD panoramic view 5. Using an overview display (top view) 22, the user can call up any location within the factory hall. This call automatically leads to a download of a data set compared in the manner according to the invention. The marking 11 on a concrete column was used during photography of the factory hall to compare the photographically recorded data with the CAD model, which the marking 11 also represents. At different positions, a user marked two points, which are connected by a design line 19. The design line has respective text fields 21, which supplement the design lines with numerical values ​​for distances between two points (in meters).The viewer can thus measure the factory hall without having to be there himself.

[0028] Fig. 4 shows an alternative representation to Fig. 3, which also includes components of a panoramic view 4 and a CAD panoramic view 5. Using a slider 23, the user can adjust the transparency of the CAD panoramic view 5. Using various buttons 26, the user can determine their location, share their location, remove an element belonging to the CAD panoramic view, and zoom into the display.

[0029] Fig. 5 shows an alternative representation to Fig. 4, in which different locations 2 are displayed, and a location 2 is selected by the user with a cursor 24 as the next location 2 to be used. A text field 25 is located at the top left of the image, which allows a free search for other positions within the displayed factory hall. A mixer bar 12 is shown on the right edge of the screen, via which the information contained in the CAD panoramic view can be selected, deactivated, and adjusted with regard to its share in the display (transparency).

[0030] Fig. 6 shows steps of an embodiment of a method according to the invention for the hybrid-dynamic representation of an infrastructure facility at a location. In step 100, a camera is used to generate a first data set representing a panoramic view of the infrastructure facility at a location within the infrastructure facility. The camera pivots motor-driven and fully automatically within a predefined, in particular 360°, spatial area. In parallel, in step 200, the position of the device used for the recording or the recording device carrier within the infrastructure facility is determined, or information for subsequent position determination is acquired.Optionally, the surrounding area is also scanned using a scanner during this step, with position data being determined while using the camera to create a point cloud representing a respective set of coordinates of points within the infrastructure facility. This data provides a more precise localization of the respective surface points contained in the panoramic view image and facilitates the subsequent comparison process. In step 300, a second data set is generated representing an image of a CAD panoramic view containing surfaces of the infrastructure facility visible from the location. A panoramic photo is automatically created within the CAD data model at the location, which is automatically compared with the first data set for identical alignment in step 400.In other words, the views are rotated around the location so that they each have identical orientations and each element of the infrastructure facility is represented twice: visually and in CAD data.

[0031] Steps 100 to 400 are then repeated in step 500 to generate corresponding data at additional locations. The additional locations are only one step or a few steps away from the first location used. In step 600, user input representing a location on a PC is determined within a browser window. The user clicks with a cursor on a location which is displayed, for example, in a floor plan view. In response, the compared data set is downloaded from a server in step 700. This can be done, for example, via the Internet or an intranet. In step 800, the compared data set is then shown on the display device within the browser. In step 900, user input is received again, this time representing a request to show or hide the panoramic view image and / or the CAD panoramic view.To do this, the user can select a corresponding slider, a corresponding checkbox or similar relating to a category for the CAD model. In response to this, in step 1000 the panoramic view image and / or the CAD panoramic view is removed from the display or displayed again. In step 1100, another user input is received, which this time represents a request to display or hide the coordinates of a point or the distance between two points within the panoramic view image and / or the CAD panoramic view. In response to this, in step 1200 the coordinates of the point are displayed or hidden and / or the distance between two points is displayed or hidden again in the form of a text field. In step 1300, a user request is once again determined, which this time represents the display or hiding of a category in the second data set.In response, in step 1400, the category (a factory structure, electrical components, waste water / fresh water, exhaust air / fresh air, machines, ladders, walkways, etc.) is hidden or shown in the representation on the display device.

[0032] List of reference symbols Infrastructure facility Location Panoramic view image Illustration of a CAD panoramic view Display device Mobile camera Scanner Browser Internet Marking Slider Server , 15 Memory Data input Evaluation unit Data output Dimensioning line User (recording device carrier) 1 Text field Floor plan representation 3 Transparency control 4 Cursor 5 Search field 6 Buttons 00 to 1400 Process steps

Claims

Patent claims 1. Method for the hybrid dynamic representation of an infrastructure facility (1) at a location (2) comprising the steps • A) Using (100) a sensor for generating an optical image, hereinafter referred to as “camera (7)”, to generate a first data set representing a panoramic view (4) of the infrastructure facility (1) at a location (2) within the infrastructure facility (1), • B) generating (300) a second data set representing an image of a CAD panoramic view (5) which comprises CAD data of the infrastructure facility (1) representing areas of the infrastructure facility (1) visible from the location (2), • C) Aligning (400) the first and second data sets for an identical alignment and • D) Displaying (800) the compared data records on a display device (6).

2. Method according to claim 1, wherein the CAD all-round view (5) is a 360° view, in particular a spherical surface projection or a cube surface projection.

3. The method according to claim 1 or 2, wherein the panoramic view image (4) is an image of a 360° view at the location, which has been created by means of a mobile camera (7) within the infrastructure facility (1), wherein the first data set is enriched in particular with laser-based environmental information.

4. Method according to one of the preceding claims further comprising Receiving (900) a user input representing a request to display or hide the surround view image (4) and / or the CAD surround view (5) and in response thereto Show or hide (1000) the panoramic view image (4) and / or CAD panoramic view (5) from the display.

5. Method according to one of the preceding claims, wherein the display is carried out by means of a web browser (9) displayed on the display device (6).

6. Method according to one of the preceding claims further comprising Determining (200) position data representing a respective set of coordinates of points within the infrastructure facility (1) while using the camera (7), Receiving (1100) a user input representing a request to display or hide a coordinate of a measurement point and / or a distance between two coordinates of measurement points and in response thereto Show (1200) or hide the coordinate of the measuring point and / or the distance between the two points.

7. Method according to one of the preceding claims, further comprising bringing the panoramic view image (4) and the CAD panoramic view (5) for a respective location (2) into alignment by means of computer-aided optical pattern recognition.

8. Method according to one of the preceding claims further comprising Determining (600) a user input representing the location (2) and in response thereto Loading (700) the data set from a server (13) by means of a WLAN connection and / or an Internet (9) connection.

9. Method according to one of the preceding claims, wherein the infrastructure facility (1) comprises a property, in particular a building, preferably a workshop and / or comprises predefined markers or survey points (11) which are recorded by means of the camera (7) and which have a spatial reference to the CAD data.

10. Method according to one of the preceding claims further comprising Determining (1300) a user request representing hiding or showing a category (12) of information contained within the second data set and in response thereto Hiding or showing (1400) the category (12) in the representation on the display device (6).

11. Method according to one of the preceding claims further comprising Repeating (500) steps A), B) and C) according to claim 1 for a further location (2) within the infrastructure facility (1), which in particular has a distance from the location (2) of 50 cm to 5 m, in particular 1 m to 3 m, preferably 1.2 m to 2 m.

12. Device comprising a data input (16), an evaluation unit (17) and a data output (18), wherein the device is configured to carry out a method according to one of the preceding claims.

Images