Colored marking of relevant regions in an augmented or virtual reality environment

Abstract

Proposed is a technical system (1) comprising: - computing means (2), which are arranged in particular separately from imaging means (3) of the technical system (1) and which are designed to generate visualization information for implementing in an augmented reality environment and to transmit said information to the imaging means (3); and - the imaging means (3), which are designed to generate a visual reproduction on the basis of the visualization information, received from the computing means (2), for an augmented reality environment. The technical system (1) is characterized in that the computing means (2) are designed to: - receive, from a user of the technical system (1), information relating to a technical apparatus (4, 7) that is designed for operation in a technical plant, in particular a process plant; - generate at least one first hologram at a first position within the technical plant on the basis of the received information; - generate at least one second hologram at a second position within the technical plant on the basis of the received information; - generate a three-dimensional object around the first and the second position; and - determine regions of overlap of the two three-dimensional objects, the imaging means (3) being designed to generate a colored depiction of the regions of overlap in the augmented reality environment for the user of the technical system (1).

Description

[0001] Description

[0002] Color coding of relevant areas in an augmented or virtual reality

[0003] The invention relates to a method. The invention also relates to a technical system having the features of the preamble of claim 9. The invention also relates to the use of a technical system.

[0004] During the acceptance of components, buildings, trains, process plants, machines, or other industrial goods or systems (hereinafter referred to as technical systems), anomalies are regularly documented. To achieve sustainable quality improvement in production and its manufacture, it is desirable to identify local clusters of anomalies in order to address possible causes.

[0005] If documentation is still on paper, a corresponding evaluation is hardly possible or only with enormous effort. The same applies to corresponding visualization. If documentation is already digital, automated queries are generally possible. However, it is very difficult to always describe the location of an anomaly in text in the same way and with sufficient precision, which in many cases makes automated evaluation very difficult.

[0006] The invention is based on the object of specifying a system for locating technical errors in a technical system, which can be operated and used efficiently and easily.

[0007] The above-stated problem is solved by a method having the features of claim 1. Furthermore, the above-stated problem is solved by a technical system according to claim 9. Furthermore, the problem is solved by using a technical system according to claim 11. Advantageous further developments are the subject of the dependent claims.

[0008] A method according to the invention comprises the following method steps: a) generating a first hologram at a first position within a technical installation by means of a technical system for augmented reality at the instigation of a user of the technical system or automatically by the technical system; b) generating a second hologram at a second position within a technical installation by means of a technical system for augmented reality at the instigation of a user of the technical system or automatically by the technical system; c) generating a three-dimensional object around the first position of the first hologram and around the second position of the second hologram by the technical system, d) determining overlapping areas of the two three-dimensional objects by the technical system, e) displaying the overlapping areas in color for the user.

[0009] A (human) user first uses a technical system. This technical system is designed to display an augmented reality for the user. The user can therefore use, for example, data glasses or a smartphone designed for this purpose. Within the augmented reality, the user creates a first hologram at a first position and a second hologram at a second position, for example using a designated gesture with a finger. The same user can create each hologram. Equivalently, a different user can create the holograms in the augmented reality. The technical system can also create the holograms automatically. To do this, it can access various data that it uses to automatically create the holograms.This data can, for example, be older inputs from users that were subjected to image recognition and automated positioning of the holograms.

[0010] The holograms created by the user or users each represent a purely virtual object in augmented reality (AR for short). In the simplest case, the holograms each represent a point within a coordinate space of the augmented reality. The user or users can create the holograms, for example, in the area of ​​a technical device where they have noticed an irregularity. This can occur, for example, during inspection measures in the technical system.

[0011] Following the creation of the holograms by the user(s), the technical system automatically creates a three-dimensional object around the first and second position. In other words, the technical system creates the object in a spatial region of the augmented reality in which the holograms are located. These three-dimensional objects can, in principle, have any desired shape. Their extent should only extend beyond the spatial extent of the holograms in order to obtain more meaningful results when determining the overlap areas. The objects can, but do not have to, have an identical extent.

[0012] It is particularly advantageous for the three-dimensional objects to be spheres with a predefined radius. The hologram is preferably located at a center point of the sphere. The radii of the two spheres are advantageously identical. The spherical shape of the three-dimensional objects allows the spatial sharpness to be specifically adjusted when determining the overlapping areas. If the radius for the spheres is chosen too small, this can result in no overlapping areas occurring. If the radius is chosen too large, the overlapping areas become too large and the significance decreases.

[0013] Preferably, the user, for whom the overlapping areas are displayed in color, can freely adjust the radius of the spheres or the spatial extent of the three-dimensional objects in order to achieve optimal values ​​for his needs.

[0014] The advantage of the method according to the invention can be clearly explained using the example of an inspection of a technical device: A first and a second user each note technical irregularities on the technical device during the inspection. They record this using a technical system for augmented reality that they use by generating a hologram at the location of the irregularity (corresponding to the first or second position). However, a technical device often has many components in a small space that could be affected by the irregularity. In addition, the manual placement of the holograms is subject to a certain spatial blur.By calculating and color-coding the overlapping areas of the three-dimensional objects around the first and the second position, the spatial blur of the manual marking is reduced on the one hand and the targeted guidance of the user's attention, who is shown the overlapping areas in color, is improved on the other hand.

[0015] Depending on the expected blurring during the generation of the holograms, the radius of the spheres around the positions can be between 5 cm and 50 cm, preferably between 15 cm and 35 cm.

[0016] Within the scope of a preferred development of the invention, at least a third hologram is generated at a third position, wherein a three-dimensional object is generated around the third position. Weighted overlap regions of the three-dimensional objects are determined. The weighting is determined based on the number of three-dimensional objects overlapping in these overlap regions, wherein the weighting of the overlap regions is expressed in the color representation for the user in the augmented reality by a different coloring of the overlap regions. In other words, the coloring of the overlap regions depends on how many three-dimensional objects (for example spheres, but also cubes, cuboids or similar) overlap in these regions. The more holograms that are generated, the less the spatial blur tends to be.

[0017] The color representation of the overlapping areas for the user in augmented reality or virtual reality can be achieved using a smartphone, data glasses, a data contact lens, or a tablet. The same technical system used to generate the holograms can be used for the augmented reality representation. The color representation of the overlapping areas can also be achieved, alternatively or additionally, in virtual reality.

[0018] Particularly preferably, the previously explained process steps are performed in a first time period and repeated in a second, later time period, with differences in the overlapping areas being highlighted in color. This allows the conditions before and after quality improvement measures to be displayed sequentially or using different colors to visually examine the effectiveness of the measures. Different production shifts can also be compared with each other in this way, for example.

[0019] As previously explained, the first and second holograms (and further holograms) can be generated in relation to a technical device within the technical system, for example to document anomalies on the technical device. The color representation of the overlapping areas can depend on a type of technical system or on a type of technical device within the technical system. One example is a train door control. By applying a previously explained method, the user of door control no. 123 can be shown the clusters of errors that have been detected by the previous inspections of door controls no. 1, 2, 3, 4, ... 122. In this case, the user can be offered various filter and aggregation functions, such as

[0020] • all errors of this door control in this car model

[0021] • all errors of this door control from this supplier, in all car models

[0022] • all errors of all door controls in this car model

[0023] • all errors of all variants of this door control in this car model

[0024] • all errors of all variants of this door control in all car models.

[0025] The previously formulated task is also solved by a technical system comprising:

[0026] Computing means, which are in particular arranged separately from image means of the technical system, and which are designed to generate visualization information for an application in an augmented reality and to transmit this to the image means, the image means, which are designed to generate a pictorial representation on the basis of the visualization information for an augmented reality received from the computing means.

[0027] The computing means can be designed separately from the image means (and possibly from image capture means), preferably in a cloud-based environment. For example, if data glasses are used, the computing capacity of the data glasses may not be sufficient, which is why the computing-intensive identification and graphics operations are carried out on a computing device that is (physically) separate from the data glasses (a computer, server, computer network, etc.). The data glasses receive the necessary graphic information from the computing device in order to visually display the hologram(s) in augmented reality. The data glasses can have a (small) part of the computing means in order to process the information received from the external computing device.

[0028] The system is characterized in that the computing means ( 2 ) are designed to :

[0029] - to receive information relating to a technical device ( 4 ) designed for operation in a technical installation, in particular a process installation, from a user of the technical system ( 1 ),

[0030] - to generate at least one first hologram at a first position within the technical system on the basis of the received information,

[0031] - to generate at least one second hologram at a second position within the technical installation on the basis of the information received,

[0032] - to create a three-dimensional object around the first and second positions, and

[0033] - to determine overlapping areas of the two three-dimensionally extended objects, wherein the image means (3) are designed to generate a color representation of the overlapping areas for the user of the technical system in the augmented reality.

[0034] The computing means can additionally be designed to generate visualization information for an application in a virtual reality, and wherein the technical system comprises further image means which are designed to generate a visual representation on the basis of the information generated by the computing means

[0035] (2) receive visualization information for a virtual reality, and wherein the further image means are designed to generate a color representation of the overlap areas for the user of the technical system in the virtual reality. The system can be able to capture its surroundings using image capture means, for example a photo camera, a video camera, a laser scanner, a LiDAR device or the like, and to recognize objects such as machines, tanks, boilers, pipelines, etc. that are present in the captured environment. The computing means comprised by the system can use a position and / or a capture direction of the image capture to be able to (unambiguously) identify captured objects. For the identification, the computing means can, for example, use system documentation such as a piping and instrumentation diagram.This makes it easier to place holograms (manually or automatically), which will be discussed later.

[0036] The user's actual view can be a direct visual impression of the surroundings. However, the user can also be shown an image by the image means which results from the image capture by the image capture means. For example, if the image capture means is designed as a night vision device, the image captured by the night vision device together with the objects identified by the computing means can be presented to the user of the system by the image means. In this case, the image means not only display the objects identified by the computing means, but also the capture by the image capture means, for example as a two-dimensional image on a smartphone, a tablet or data glasses.

[0037] The previously formulated problem is additionally solved by the use of a technical system which is designed as explained above, for the representation of an augmented reality or a virtual reality in a technical system, in particular a process system or production system, for the detection of abnormalities in a technical device within the system. These technical systems each have a control system or at least a computer-aided module for controlling and regulating the ongoing process or production. In the present context, a control system is understood to be a computer-aided technical system which comprises functionalities for displaying, operating and managing a technical system such as a manufacturing or production system.

[0038] In addition, the object is achieved by the use of a technical system which is designed as explained above, by a user for the representation of an augmented reality or a virtual reality in a technical plant, in particular a process plant or production plant, for the training of an operator of the technical plant.

[0039] The technical system may comprise detection means configured to detect a unique identification of components of the technical module by the operator in order to assist the operator in manually arranging the at least one hologram within the augmented reality environment.

[0040] The detection means can be configured to detect the unique identification of components of the technical module using camera means, wherein the camera means and the imaging means are arranged on a (common) optical device. For example, the camera means can comprise a camera that can identify components of the technical system via object recognition, which can facilitate the positioning of the hologram. The optical device can be configured as a smartphone, a tablet, a notebook, data glasses, or contact lenses for augmented reality.

[0041] The detection means can also be designed for a laser-based detection of components of the technical module, wherein the computing means in this case are designed to take into account the laser-based detection of the components, and preferably additionally a digital model of the technical module, for the arrangement of the holograms.

[0042] The above-described properties, features and advantages of this invention, as well as the manner in which they are achieved, will become clearer and more clearly understood in connection with the following description of the embodiment, which is explained in more detail in connection with the drawings.

[0043] FIG. 1 shows a technical system 1 that serves to generate augmented reality (AR) for a user of the technical system. The technical system 1 comprises computing means 2 and imaging means. These are designed to present the augmented reality to the user in the environment of a technical device 4 within a technical system.

[0044] The computing means 2 are implemented in a cloud-based environment 5 and may, for example, comprise a personal computer (PC). The computing means 2 are connected to the image means 3 via a bidirectional, wireless connection 6 (NEC, WiFi, Bluetooth, etc.). The image means 3 are designed as data glasses. The computing means 2 are designed to generate visualization information for an application in an augmented reality and to transmit this to the image means 3, which then generate a corresponding image representation (cf. FIG. 2).

[0045] The technical device 4 has a large number of individual components (cf. FIG. 2) which are designed to carry out process-related functions. The user can use the computing means 2 to generate holograms at various positions within the technical system, which are then displayed to him by the image means 3 within the augmented reality. The computing means 2 each generate a three-dimensional object, for example a sphere, around a first position of a first hologram generated by the user and around a second position of a second hologram generated by the user. The computing means 2 then determine overlapping areas of the two three-dimensional objects and generate a color representation of the overlapping areas, which is illustrated by way of example in FIG. 2.

[0046] FIG 2 shows a further technical device 7 which includes, among other things, pipes and valves. The user or various users have generated a large number of holograms which are intended to indicate abnormalities on the technical device 7. The computing means 2 have generated overlapping regions 8a, 8b, 8c, 8d by an intersection operation of the overlapping three-dimensional objects around the hologram positions and display these for the user in the augmented reality. The actual holograms are not shown in FIG 2 because this would not provide any additional benefit for the user and would reduce the clarity of the augmented reality display.

[0047] The overlapping areas are color-coded based on the weighting of the overlaps. For example, areas in which many three-dimensional objects spatially overlap can be marked with a dark red color, while areas in which only a few three-dimensional areas spatially overlap can be marked with a light yellow. To improve the significance, a minimum number of overlapping three-dimensional objects can be defined (e.g. 3), below which no coloring occurs. This type of color coding is generally referred to as a "heat map".

[0048] Using a method according to the invention, the user receives a compact, transparent, and immediately understandable overview of where abnormalities occur in the technical system and where they occur most frequently. By modifying the parameters introduced, the display can be sharpened and adapted interactively. This allows hotspots to be identified quickly and easily, and appropriate measures to reduce abnormalities in the future to be implemented. Furthermore, the display can also be used for the induction and training of new employees, who should quickly receive an overview of the most important components. This ensures that all planned, necessary acceptance tests are carried out, even for new employees, and that areas with abnormalities are given particular focus.In contrast to lists of components with frequent anomalies, for example, the assignment is easier and the spatial relationship can be recognized immediately.

[0049] Although the invention has been illustrated and described in detail by the preferred embodiment and the figures, the invention is not limited by the disclosed examples and other variations can be derived therefrom by those skilled in the art without departing from the scope of the invention.

Claims

Patent claims 1. Method, comprising: a) generating a first hologram at a first position within a technical installation by means of a technical system (1) for augmented reality at the instigation of a user of the technical system (1) or automatically by the technical system (1); b) generating a second hologram at a second position within a technical installation by means of a technical system (1) for augmented reality at the instigation of a user of the technical system (1) or automatically by the technical system (1); c) generating a three-dimensionally extended object around the first position of the first hologram and around the second position of the second hologram by the technical system (1), d) determining overlapping areas of the two three-dimensionally extended objects by the technical system (1) , e) representation of the overlapping areas (8a, 8b, 8c, 8d) for the user, in particular color representation.

2. Method according to claim 1, wherein the three-dimensionally extended objects are spheres with a predeterminable radius, the hologram being located at a center of the sphere.

3. Method according to claim 2, wherein the radius of the balls each has an amount between 5 cm and 50 cm, preferably between 15 cm and 35 cm.

4. Method according to one of the preceding claims, in which at least one third hologram is generated at a third position, wherein a three-dimensionally extended object is generated around the third position, and in which weighted overlapping areas (8a, 8b, 8c, 8d) of the three-dimensionally extended objects are determined, wherein the weighting is determined based on the number of three-dimensional objects overlapping in these overlapping areas (8a, 8b, 8c, 8d), wherein the weighting of the overlapping areas (8a, 8b, 8c, 8d) is expressed in the color representation for the user in the augmented reality by a different coloring of the overlapping areas (8a, 8b, 8c, 8d).

5. Method according to one of the preceding claims, in which the color representation of the overlapping areas (8a, 8b, 8c, 8d) for the user in the augmented reality and / or in a virtual reality is carried out by a smartphone, data glasses, a data contact lens or a tablet.

6. Method according to one of the preceding claims, in which the method steps are carried out in a first time range and carried out again in a second, later time range, differences in the overlapping areas (8a, 8b, 8c, 8d) being marked in color.

7. Method according to one of the preceding claims, in which the first and the second hologram are generated with respect to a technical device (4, 7) within the technical installation.

8. The method according to claim 7, wherein the color representation of the overlapping areas (8a, 8b, 8c, 8d) depends on a type of technical installation or on a type of technical device (4, 7) within the technical installation.

9. Technical system (1) comprising: Computing means (2) which are arranged in particular separately from image means (3) of the technical system (1) and which are designed to generate visualization information for an application in an augmented reality and to transmit this to the image means (3), the image means (3) which are designed to generate an image representation based on the visualization information for an augmented reality received from the computing means (2), characterized in that the computing means (2) are designed to: - to receive information relating to a technical device (4, 7) designed for operation in a technical installation, in particular a process installation, from a user of the technical system (1), - to generate at least one first hologram at a first position within the technical system on the basis of the received information, - to generate at least one second hologram at a second position within the technical installation on the basis of the information received, - to create a three-dimensional object around the first and second positions, and - to determine overlapping areas of the two three-dimensionally extended objects, wherein the image means (3) are designed to generate a representation, in particular in color, of the overlapping areas for the user of the technical system (1) in the augmented reality.

10. Technical system (1) according to claim 9, wherein the computing means (2) are additionally designed to generate visualization information for an application in a virtual reality, and wherein the technical system (1) comprises further image means which are designed to generate an image representation on the basis of the information generated by the computing means (2) received visualization information for a virtual reality, and wherein the further image means are designed to generate a color representation of the overlapping areas (8a, 8b, 8c, 8d) for the user of the technical system (1) in the virtual reality.

11. Use of a technical system (1) according to one of claims 9 or 10 by a user for the representation of an augmented reality or a virtual reality in a technical plant, in particular a process plant or a manufacturing plant, for the detection of anomalies of a technical device (4, 7) within the plant.

12. Use of a technical system (1) according to one of claims 9 or 10 by a user for the representation of an augmented reality or a virtual reality in a technical plant, in particular a process plant or a manufacturing plant, for training an operator of the technical plant.

Images