Method for displaying user support information in an augmented-reality application, and augmented-reality apparatus

Abstract

The present invention relates to a method for displaying at least one item of user support information in relation to at least one device or one of its components in an augmented-reality application. The present invention also relates to an augmented-reality apparatus for providing at least one item of user support information in relation to at least one device or one of its components to a user.

Description

[0001] Method for displaying user support information in an augmented reality application and augmented reality device

[0002] Description

[0003] field of technology

[0004] The present invention relates to a method for displaying at least one item of user support information relating to at least one device or one of its components in an augmented reality application. The present invention also relates to an augmented reality device for providing at least one item of user support information relating to at least one device or one of its components to a user.

[0005] State of the art

[0006] In the industrial environment, as machinery becomes increasingly digitized, more and more information about the equipment and its components is being collected, processed, and stored. Based on this data, error reports or maintenance information, for example, can be generated. This information can be used to support on-site service personnel during their service work.

[0007] However, as the amount of data increases, it becomes more difficult to maintain an overview of the individual information.

[0008] Summary of the invention

[0009] It is therefore an object of the present invention to overcome the described disadvantages of the prior art and in particular to provide means by which a reliable provision of information to service employees is possible even with a larger amount of data.

[0010] The object is achieved by the invention according to a first aspect in that a method for displaying user support information relating to at least one device or one of its components in an augmented reality application, wherein at least one image of a section of the environment in which the device is at least partially visible is recorded, wherein the user support information to be displayed is obtained and displayed in the image or in a field of view of a user of the augmented reality application that at least partially contains the section of the environment, wherein a marker assigned to the device is identified in the image and the identified marker is evaluated, and wherein at least partially based on at least one result of the evaluation (i) a position at which the user support information is displayed in the image or in the field of view of the user,as the display position and / or (ii) a perspective adjustment of the user support information or parts thereof is proposed.

[0011] The invention is therefore based on the surprising finding that, based on structures defined in the image and associated with the device, further information can be derived from the structures, which make it possible to display user support information at a desired location and with a spatial impression in an image of the device or in a user's field of vision.

[0012] By appropriately determining the display position of the user support information in the image according to the invention, the user support information can be displayed in the image or the field of view with reference to a specific component of the device, for example, in its proximity and / or via an assignment line to this component. This allows the user to very reliably assign the user support information to be provided to a physical component of the device. Furthermore, by adjusting the perspective of the user support information, it is made even easier for the user to assign the information to be displayed to the correct component, especially when viewing the device at an angle.

[0013] As a result, the proposed method advantageously allows maintenance information to be reliably displayed on the component to be maintained as user support information. This facilitates maintenance of the device and its components, and in particular, maintenance can be carried out more reliably and quickly.

[0014] Even if several identical or identical devices or components are in use, the proposed method can display information about one of these components in the image or in the user's field of vision with reference to the correct component. This enables reliable assignment of the information and eliminates or at least greatly reduces the risk of accidentally swapping components. This allows information to be provided reliably even with increasing data volumes.

[0015] The user assistance information can therefore be provided to the user of the augmented reality application very reliably by superimposing the user assistance information in the user's field of view and / or by superimposing the user assistance information in the image that the user can view.

[0016] Preferably, in the context of this application, a perspective adaptation of the user support information or parts thereof is understood to mean an adaptation of the presentation of the user support information or parts thereof such that a spatial effect is created at least partially when the user support information is displayed in the image or in the field of view (in particular for the user of the augmented reality application, who may be a viewer of the image, or in whose field of view the user support information is displayed).

[0017] By using the marker at least partially for both determining the overlay position and for adjusting the perspective of the user support information, the marker can be used for multiple purposes. This makes it possible to implement the method with less effort. In the real world, only a corresponding marker assigned to the device needs to be provided in such a way that it can be captured together with the device in the image. Further adjustments to the device are therefore not necessary. Information for determining the overlay position is advantageously determined using the marker contained in the image. The overlay position can be determined directly using the marker, or intermediate information can be determined using the marker, which in turn is used to determine the overlay position, particularly with the inclusion of other sources.

[0018] Using the marker contained in the image, information for perspective adjustment is advantageously determined. The information required for perspective adjustment can be determined directly from the marker (as explained in more detail later), or intermediate information can be determined from the marker, which in turn, particularly with the inclusion of additional sources, can be used to determine the information required for perspective adjustment.

[0019] The captured image with the visible marker serves as a starting point for obtaining further information for displaying the user support information. To do this, the marker is identified in the image and then evaluated. The user support information can then be displayed in the user's field of vision. Alternatively, the image can also be used to display the user support information.

[0020] This also makes the method very easy to retrofit to existing devices. This allows the advantages of the proposed method to be exploited even in existing environments.

[0021] Preferably, the marker is at least partially, preferably completely, visible in the surrounding area. Thus, the marker is also reliably fully or at least partially visible in the recorded image.

[0022] For example, the marker can be assigned to the device by attaching the marker to the device and / or by positioning the marker such that it is visible in the surrounding area of ​​the image.

[0023] A physical object can advantageously be provided as a marker, which has a surface area provided with a pattern, in particular one that can be read in the image, and / or which is arranged on the device. The marker can also be formed integrally with the device. For example, a surface area of ​​the device can be formed as a marker, which surface area has a surface structure (which is formed or can be formed, for example, by engraving, for example by means of a laser or a mechanical process) or physical properties (such as a refractive index of a material), whereby a pattern is formed, in particular one that can be read in the image. Examples of marker patterns are one- or multi-dimensional codes, such as one-dimensional barcodes or two-dimensional data codes, in particular QR codes. Another example of advantageous marker patterns is AprilTags.

[0024] Preferably, the identification and evaluation of the marker in the image is carried out at least partially using digital image recognition and / or processing. For example, the marker or an image region that at least partially contains the marker can be first identified in the image, and then the identified marker and / or image region can be evaluated accordingly.

[0025] The at least one result of the evaluation may have multiple partial results, such as a first, second, third, etc. result of the evaluation. The individual partial results may at least partially have different processing steps and / or at least partially build on one another (so that a partial result is at least partially derived from another partial result).

[0026] Preferably, the position at which the user support information is displayed in the image or in the user's field of vision is determined as the display position, at least partially based on at least a first result of the evaluation. Optionally, the perspective adjustment of the user support information or parts thereof can also be performed, at least partially based on the first result of the evaluation.

[0027] Preferably, the perspective adjustment of the user support information or parts thereof is performed at least partially based on at least one second result of the evaluation. Optionally, the display position can also be determined at least partially based on the second result of the evaluation.

[0028] Preferably, the user support information to be displayed is displayed at the specific display position in the image or in the user's field of vision.

[0029] Advantageously, the user support information may contain status data, performance data, parameters and / or a designation of a component of the device or of the device itself.

[0030] An advantageous procedure could, for example, be as follows: For an image of a device captured with a camera, the orientation (e.g., the position and / or orientation) of a marker attached to the device is estimated. For this purpose, a rotation matrix and a translation vector are mathematically defined relative to the camera. The corner points of the user support information to be displayed are defined in a coordinate system of the marker. The corner points of the user support information are displayed in the image using the rotation matrix and the translation vector and, optionally, a camera intrinsic (for example, world coordinates can be converted into image coordinates). The content of the user support information (bar charts, line charts, text, values) is available as 2D image material and is displayed in perspective in the image.To do this, a linear geometric transformation is calculated based on the corner points of the user support information. This allows a representation of the user support information with a spatial impression to be obtained.

[0031] Alternatively or additionally, it can also be provided that the display position is also determined depending on the user support information to be displayed, in particular a categorization of the user support information to be displayed, and / or the component to which the user support information to be displayed is assigned.

[0032] For example, the user support information to be displayed can be categorized according to its display position. Possible categories include "top," "bottom," "left," "right," and "center," and combinations thereof.

[0033] The assignment of the user support information to the respective component can be obtained separately from and / or in parallel with the user support information to be displayed.

[0034] Alternatively or additionally, it can also be provided that at least some, preferably all, components of the device are stored together with the information assigned to them in a data-based collection and that the data-based collection is included in the determination of the display position and / or in the perspective adjustment, wherein the information assigned to the components preferably represents or makes it possible to determine at least one display position for a user support information assigned to the respective component.

[0035] This makes it advantageously possible to determine the display position for the user support information for the component to which the user support information is assigned in the data-based collection (for example, the output position described later can be stored in the data-based collection).

[0036] Preferably, the fade-in position is specified relative to a position of the marker or it is determined as a position relative to the marker position.

[0037] The display positions assigned to (or determinable for) the components can be user support information-dependent display positions. In this case, a single component can be assigned (or determinable for) multiple display positions, which can be selected depending on the user support information.

[0038] The collection can, for example, be organized in a tree structure. This also makes it possible to visualize the dependencies between components.

[0039] Alternatively or additionally, it can also be provided that the marker contains readable and / or decodable information, such as a unique identifier of the device, as marker information and the marker information is read and / or decoded from the marker and the user support information is requested, in particular from a memory and / or from a remote computer, at least partially based on the marker information.

[0040] The marker information can advantageously be used to clearly identify the device.

[0041] The marker information can advantageously be used to request user support information.

[0042] Using the marker information, an overview of available user support information can also be requested. Then, in a subsequent step, for example, after performing a user selection, the actual user support information can be requested.

[0043] The readable and / or decodable information can be provided by the pattern described above on the marker surface.

[0044] Alternatively or additionally, it may also be provided that the user support information is requested at least partially based on a user selection and / or from a memory and / or from a remote computer.

[0045] For example, the user selection may select one of the available user support information from the requested overview as previously described.

[0046] Alternatively or additionally, it may also be provided that the user support information is obtained as image information, in particular by supplementing an image template with further information.

[0047] The image information can be particularly efficiently blended into the image (which in turn preferably contains image information) or into the user's field of vision.

[0048] Preferably, the image information is two-dimensional image information. Alternatively or additionally, it can also be provided that information about the geometry of the device and / or its components is stored for the marker, in particular in the data-based collection, and that the geometry information is included in determining the display position and / or in the perspective adjustment.

[0049] The geometry information can be obtained or retrieved, for example from a memory or from a remote unit, using, for example, the marker information.

[0050] The geometric information can include, for example, relative positions (in particular to a reference point of the device), dimensions, orientations, and / or surface contours of the device and / or its components. It is advantageous if the marker is oriented in a known manner and / or positioned relative to the reference point, in particular arranged on the device.

[0051] This allows user support information to be displayed very reliably at the location or component to which it belongs based on context.

[0052] The geometric information can be a model, such as a mesh model or a CAD model, of at least parts of the device and / or its components. The model can be a simplified representation of the device and / or its components. This allows the model to be stored and used more efficiently in memory. Often, a model with only a rough representation of the components (e.g., approximated as cuboid objects) is sufficient.

[0053] The geometry information and / or the model may be part of the data-based collection, in particular part of the information stored in the data-based collection for the individual components.

[0054] Alternatively or additionally, it can also be provided that the marker has a surface and the user support information is displayed in the image or in the field of vision of the user in such a perspectively correct manner, in particular as a two-dimensional representation, that at least part of the user support information is perceived by the user as lying within a plane, which plane runs parallel to the marker surface or encloses a defined solid angle with it.

[0055] It is advantageously possible to determine a perspective distortion of the marker in the image by evaluating the marker in the image and, based on the determined perspective distortion, to adjust the perspective of the user support information. This allows the user support information or parts thereof to be displayed in the image or in the user's field of vision with a perspectively correct representation.

[0056] For the purposes of this application, a plane runs parallel to the marker surface even if the marker surface lies in the respective plane.

[0057] Alternatively or additionally, it may also be provided that the dimensioning of the user support information within the image or the field of view of the user is carried out at least partially on the basis of a structure of the marker contained in the image, such as an edge of the marker to which a previously known length value is assigned.

[0058] This makes it particularly easy to adjust the display behavior of the user support information (in the image or in the user's field of vision) by adjusting the marker size (in reality). Alternatively or additionally, the user support information can also be assigned to a specific component of the device.

[0059] The specific component can thus be distinguished from other components of the device.

[0060] Alternatively or additionally, it may also be provided that the specific component is a component of the device with a defect or impending maintenance.

[0061] Alternatively or additionally, it can also be provided that the device has several components identical to the specific component.

[0062] This allows the user support information to be reliably displayed in relation to the correct one of several identical components in the image or in the user's field of vision.

[0063] Alternatively or additionally, it can also be provided that the evaluation of the marker comprises determining, at least partially based on the representation of the marker in the image, at least some or all of the necessary information for defining a, in particular three-dimensional, reference coordinate system with respect to the marker, in particular viewed from the perspective selected for recording the image, and wherein the determined information is used at least partially in each case to determine the display position of the user support information and / or to adapt the perspective of the user support information.

[0064] The representation of the marker in the image can be characterized, for example, by a position, an orientation, and / or a geometric progression of the marker or parts thereof in the image. Therefore, corresponding features can be determined, based at least in part on which the information necessary for defining the reference coordinate system can be determined. For example, the progression of two edges of the marker could be evaluated as the geometric progression of the marker in the image, wherein it can preferably be known in advance that the two edges run parallel to each other on the physical marker. If the edges in the image are inclined relative to each other, a viewing angle (i.e., an angle at which the image was captured relative to a frontal direction) can be determined from this, and, for example, a perspective adjustment of the user support information can be at least partially carried out.

[0065] The reference coordinate system can be viewed as a coordinate system of the marker. For example, the reference coordinate system can have its origin on the marker surface, in particular, centered on the marker surface. This allows a coordinate system to be established as a reference coordinate system based on the image, so that the reference coordinate system and the marker can be perceived with the correct perspective in the image. In other words, for multiple images from different perspectives, the marker would be perceived as stationary with respect to the respective reference coordinate system superimposed on the image.

[0066] However, one skilled in the art will understand that the reference coordinate system itself does not have to be visible and is certainly not a physical part of the device. Rather, the display position and / or the perspective adjustment can be determined at least partially based on the information obtained, with which the reference coordinate system can in principle be defined.

[0067] Preferably, the position of the marker on the device and / or the position of the component relative to the marker is known (e.g., in the same world coordinate system). Thus, in principle, a "marker coordinate system" can be established using the determined information. Above all, however, the determined information can advantageously be used to determine the display position and / or to adjust the perspective of the user support information.

[0068] For example, a location specified in coordinates of the reference coordinate system can be mapped to a location in the image (in particular taking into account the information determined to define the reference coordinate system) by projection onto the image plane of the image.

[0069] Alternatively or additionally, it can also be provided that the reference coordinate system is a coordinate system of the device and / or the reference coordinate system can be converted in a previously known manner into a coordinate system of the device and vice versa.

[0070] Within the coordinate system of the device, for example, positions and / or extensions of the individual device components can be defined as geometric information. For example, corresponding geometric information can be stored in the data-based collection for the individual components. Information about the position and / or extension of a specific component (to which the user support information to be displayed is assigned) can then be retrieved from the data-based collection in order to at least partially determine the display position based on this information. This allows, for example, a display position that is "near" the respective component to be reliably determined. Conversely, the component intended for that component can also advantageously be determined using the data-based collection based on a given position.

[0071] Alternatively or additionally, it can also be provided that the user support information and / or the specific component is assigned an output position which indicates a position within the reference coordinate system or which is converted into a position within the reference coordinate system, and wherein the display position of the user support information is determined based on the position within the reference coordinate system.

[0072] Specifying the output position in a coordinate system other than the reference coordinate system is advantageous because when the origin of the reference coordinate system changes, for example, due to a change in the position of the marker on the device, the fade-in positions stored for the individual components do not need to be adjusted; instead, only a reference between the two coordinate systems needs to be adjusted. This allows for greater flexibility.

[0073] Alternatively or additionally, it can also be provided that (i) the reference coordinate system has its origin on a surface of the marker and / or (ii) the surface of the marker lies in a plane spanned by two coordinate axes of the reference coordinate system and preferably the third coordinate axis is parallel or antiparallel to a normal vector of the surface of the marker.

[0074] The reference coordinate system can therefore be the already mentioned coordinate system of the marker.

[0075] Alternatively or additionally, it can also be provided that the user support information is assigned to at least one information class from several possible information classes and the form of presentation of the user support information is selected at least partially depending on the information class. By means of appropriate information classes, different forms of presentation can be provided, for example, for user support information of varying importance. This advantageously makes it possible to promote the attention of the user of the augmented reality application, if necessary. For example, user support information that represents a general note ("low priority") can be presented differently than user support information that represents an error state of the device or one of its components ("high priority").

[0076] For example, user support information indicating a critical state of the device or one of its components can be emphasized by an appropriate presentation form, such as by displaying it with a prominent font, a prominent font color, a prominent font size, and / or prominent symbols. For example, user support information indicating additional information about the device or one of its components can be de-emphasized by an appropriate presentation form, such as by displaying it with a subdued font, a subdued font color, a subdued font size, and / or subdued symbols.

[0077] By using appropriate information classes, different presentation formats can be provided for different types of user support information, for example. This advantageously makes it easier for the user of the augmented reality application to grasp the user support information.

[0078] For example, for user support information indicating a value of an operating parameter of the device, a display format resembling an instrument display can be selected. For example, for user support information representing historical maintenance information, a display format in the form of a notepad can be selected.

[0079] Alternatively or additionally, it can also be provided that, depending on the information class, one of the following forms of representation is selected as the form of representation of the user support information in the image or in the field of vision of the user: an information panel, a diagram, an instrument display, a highlight, in particular a border, of an area of ​​the device or one of its components, in particular the specific component, a visual representation of a component, in particular the specific component, or a combination of several of the above options.

[0080] An information board can be used advantageously to display text passages, such as a test report, or characteristic values ​​of the device or one of its components.

[0081] A diagram can be advantageously used to represent the temporal progression of a quantity, such as an operating parameter, of the device or one of its components.

[0082] A highlight, in particular a border, can be used advantageously to identify a certain (in particular the specific) component of the device.

[0083] Alternatively or additionally, it can also be provided that the user support information is assigned to at least one of the following information classes: component identification, maintenance information, operating information, low priority, medium priority, high priority and / or process information.

[0084] Process information advantageously refers to a process that is at least partially carried out using the device. Process information can also refer to circumstances that lie outside the device itself. For example, process information can relate to information about a material processed using the device, such as a bulk material. Process information can also refer to parts of the process in which the device is not involved, such as upstream or downstream parts of the process.

[0085] Alternatively or additionally, it can also be provided that the user support information is a table-like overview with information on one or more operating parameters of the device or the specific component and / or on one or more process parameters of a process at least partially carried out with the device.

[0086] This allows the user of the augmented reality application to grasp the user support information particularly quickly.

[0087] A process parameter advantageously refers to a process that is at least partially carried out using the device. A process parameter can also refer to circumstances that lie outside the device itself. For example, a process parameter can relate to information about a parameter of a material processed using the device, such as a bulk material. A process parameter can also refer to parts of the process in which the device is not involved, such as upstream or downstream parts of the process.

[0088] Alternatively or additionally, it can also be provided that the user support information is a graphical representation, such as a diagram, a chart, an instrument display and / or a pictogram, of one or more operating parameters of the device or the specific component or of one or more process parameters of a process at least partially carried out with the device.

[0089] Alternatively or additionally, it can also be provided that at least one operating parameter is selected from the following parameters: feed rate, performance, energy consumption and / or S0II / I st comparison of a device size.

[0090] Alternatively or additionally, it can also be provided that at least one process parameter is information, such as a designation or a property, about a bulk material processed with the device within the process.

[0091] Processing a bulk material can, for example, involve handling, processing, and / or converting the bulk material. For example, the bulk material is processed when it is ground, mixed, sieved, or crushed.

[0092] Alternatively or additionally, it can also be provided that the user support information is a graphic illustration of the specific component (for example, screw, spiral, intelligent feeder) and wherein preferably the specific component is not visible in the recorded image or in the field of vision of the user.

[0093] This makes it particularly advantageous to display the specific component installed in the device but not visible in the recorded image (because it is covered by other parts of the device, for example). This also makes it possible to display hidden components without having to open the device and thus without potentially interrupting ongoing operation of the device. This allows users of the augmented reality application to quickly and easily obtain an overview of the device and its installed components. The user support information displayed as a graphic illustration of the specific component is preferably displayed in the image in the orientation that the specific component also has in the device and / or offset from its installation location.This allows the specific component to be displayed offset from its installation location, similar to an exploded view, using the displayed user support information.

[0094] Alternatively or additionally, the user support information may also be an overview with documentation on the device or the specific component or a reference to such an overview or to such documentation.

[0095] This allows for quick access to the relevant documents. It also allows for secure assignment of documents to the correct component, preventing them from being mixed up.

[0096] Alternatively or additionally, it can also be provided that the user support information is a highlight, in particular a border, of an area occupied by parts of the device or by the specific component and / or for highlighting a contour of parts of the device or the specific component, in each case in the image or in the field of vision of the user.

[0097] This allows the user of the augmented reality application to be directed particularly reliably to a specific area of ​​the device or to a specific (especially the specific) component. For example, a component that needs to be replaced (especially the specific) can be marked with this.

[0098] This means that, especially when there are several similar or identical components, component swaps can be reliably avoided, for example during maintenance work.

[0099] Alternatively or additionally, it can also be provided that the user support information is a visualization of at least one measuring section and / or at least one measuring location in the image or in the field of vision of the user.

[0100] This allows the user of the augmented reality application to be shown particularly reliably at which location (or at which multiple locations), for example, one or more measuring sensors must be provided to carry out a measurement, such as a voltage measurement, a vibration measurement or a temperature measurement, or between which locations a measurement must be carried out, such as a distance measurement.

[0101] For example, after installing a new component, the user of the augmented reality application can reliably measure the distances displayed by the visualization on the device or its components.

[0102] Alternatively or additionally, it can also be provided that the user of the augmented reality application is guided step by step through at least one work process, in particular through at least one maintenance process and / or through at least one measuring process, and for this purpose, at least some, preferably at every, step an updated user support information is displayed in the image or in an at least partially newly recorded image or in the field of vision of the user.

[0103] This allows the augmented reality application user to highlight different components at each step, display different graphic illustrations, different measurement path visualizations, and / or different measurement location visualizations. This makes the workflow more efficient, safer, and more reliable.

[0104] Alternatively or additionally, it can also be provided that a pixel or image area in the image is selected by the user, in particular by a selection via a touch-sensitive screen, and the specific component is selected on the basis of this user selection.

[0105] This represents an advantageous and particularly simple way to request information about a specific component by selecting the respective component in the image and then displaying the requested information as user support information in the image or in the field of view.

[0106] Preferably, the user support information is associated with the selected component as previously described.

[0107] Alternatively or additionally, it can also be provided that the pixel or image area of ​​the image selected by the user selection is assigned to a component of the device via an assignment, in particular based on the data-based collection, and that this component is selected as the specific component.

[0108] The assignment can advantageously be carried out by including the data-based collection, the reference coordinate system and / or the coordinate system of the device.

[0109] Advantageously, the selected pixel or image area of ​​the image is synchronized via the respective coordinate system with the information stored in the data-based collection, in particular geometry information, of the individual components.

[0110] For example, for the selected pixel or image area of ​​the image, a location or a multidimensional area in the reference coordinate system and / or in the coordinate system of the device can be determined and, in particular at least partially based on the data-based collection, a component of the device provided at the location and / or at least partially in the area can be selected as the specific component.

[0111] Alternatively or additionally, it can also be provided that different components of the device are assigned to different pixels or image areas in the image.

[0112] Here, too, the assignment can advantageously be carried out by including the data-based collection, the reference coordinate system and / or the coordinate system of the device, as previously described.

[0113] Alternatively or additionally, it can also be provided that a plurality of images, in particular as frames of a video stream, are recorded and for each image the display position and / or the perspective adjustment of the user support information is / are newly determined or carried out and the user support information is displayed accordingly in the respective image or in the field of vision of the user.

[0114] For example, the individual images of the plurality of images are recorded sequentially. For example, a user can move at least partially along or around the device and repeatedly record an image.

[0115] Preferably, the same marker is identified and evaluated in all images. However, it can also be advantageous for at least partially different markers to be identified and evaluated in different images. Preferably, in the case of multiple different markers, the individual markers have at least partially different patterns and / or marker information. Different patterns and / or marker information can, for example, uniquely identify the device and / or determine information about the imaged side of the device (e.g., front, back, left, right).

[0116] For example, if a device with the unique identifier "123" has a different marker arranged on all four sides, the marker information of the markers arranged on the right, left, front and rear of the device could be "123-right", "123-left", "123-front" and "123-rear", respectively.

[0117] Alternatively or additionally, it can also be provided that the device is a device from the field of bulk material handling, bulk material processing and / or bulk material processing.

[0118] For example, the device can be a conveying, measuring, weighing, grinding, mixing, screening, drying and / or dosing device for bulk material.

[0119] The object is achieved by the invention according to a second aspect in that an augmented reality device for providing at least one item of user support information relating to at least one device or one of its components to a user, the device comprising at least one detection unit configured to record at least one image of a section of the environment in which the device is at least partially visible, at least one display unit configured to receive the user support information to be provided and to display it in the image or in a field of view of the user of the augmented reality device that at least partially contains the section of the environment, at least one evaluation unit configured to identify a marker assigned to the device in the image and to evaluate the identified marker,and at least one processing unit configured to determine, at least in part based on at least one result of the evaluation, (i) a position at which the user support information is displayed in the image or in the field of view of the user as the display position and / or (ii) to perform a perspective adjustment of the user support information or parts thereof.

[0120] All advantages described with regard to the method according to the first aspect of the invention also apply accordingly to the device according to the second aspect of the invention. Therefore, reference can be made to the previous explanations at this point.

[0121] The features described with reference to the method according to the first aspect of the invention can also be provided correspondingly in the device according to the second aspect of the invention, individually and in any combination, unless the context indicates otherwise. For this purpose, the device and / or the acquisition unit, the display unit, the evaluation unit, and the processing unit can each be configured to execute corresponding method steps.

[0122] The acquisition unit, the display unit, the evaluation unit, and the processing unit can each be implemented, for example, in software, hardware, or a combination of both. The respective unit can be a data processing device. Alternatively or additionally, the respective unit can comprise a memory, a processor, a receiving device, a transmitting device, or any combination thereof.

[0123] The following implementations can advantageously be provided individually and in any combination: The detection unit can be or comprise a camera. The display unit can be or comprise a screen, in particular with a receiving module configured to receive the user support information. The evaluation unit can comprise means of digital image recognition and / or processing and preferably apply these to the image in order to identify the marker in the image and evaluate the identified marker. The processing unit can be or comprise a computing unit. The aforementioned units can advantageously exchange data with each other, at least in part.

[0124] Alternatively or additionally, the device may also be designed as or be part of a smartphone, a tablet, a head-mounted display or glasses.

[0125] Short description of the drawings

[0126] Further features and advantages of the invention will become apparent from the following description, in which preferred embodiments of the invention are explained with reference to schematic drawings.

[0127] Showing:

[0128] Fig. 1a is a schematic cross-sectional view of a dosing device for bulk material;

[0129] Fig. 1b is a schematic front view of the dosing device from Fig. 1a;

[0130] Fig. 2 shows an augmented reality device according to the second aspect of the invention;

[0131] Fig. 3a is a recorded image of the dosing device from Fig. 1b;

[0132] Fig. 3b the image from Fig. 3a with a first

[0133] User support information;

[0134] Fig. 3c the image from Fig. 3a with a second

[0135] User support information;

[0136] Fig. 3d the image from Fig. 3a with a third

[0137] User support information; and

[0138] Fig. 4 is a flowchart of a method according to the first aspect of the invention.

[0139] Description of the embodiments

[0140] Fig. 1a shows a schematic cross-sectional view of a dosing device 1 for dosing bulk material. Fig. 1b shows a schematic front view of the dosing device 1 from Fig. 1a.

[0141] A bulk material 3 to be dosed is fed to the dosing device 1, where it is received in a receiving container 5 of the dosing device 1 and forms a bulk material surface 7. The bulk material 3 passes from the receiving container 5 via a bottom opening 9 into a discharge element 13 arranged rotatably within a horizontal section 11, by means of which the bulk material 3 is then discharged from the dosing device 1 in a conventional manner and leaves the dosing device 1 via a vertical discharge 15. The discharge element 11 is coupled to a motor 17 and, controlled by a motor controller of the motor 17, can be rotated at an adjustable, variable speed. During material discharge, a change in the weight of a system of the dosing device 1 weighed by two weighing units 19 and 21 is detected and used to regulate the speed of the discharge element 11. The weighing units 19 and 21 are in turn supported on a support frame 23.

[0142] The bulk material 3 can be processed in further process steps both before the dosing process and after the dosing process, whereby the elements required for this are not shown in Fig. 1a.

[0143] A marker 25 with a defined stripe pattern (Fig. 1b) is arranged on the dosing device 1 on the frontal outer side of the horizontal section 11 (visible in Fig. 1b). The marker 25 is arranged within a coordinate system 27 of the device 1 at a defined location and with a defined orientation on the dosing device 1. The coordinate system 27 is a purely definitional construct and not a physical part of the device 1 and is shown in the figures only for illustration purposes.

[0144] The device 1 and its components are represented by a virtual model defined within the coordinate system 27, which is stored in a data-based collection. Position, orientation, and extent information is stored for each component. Using the data-based collection, a spatial volume occupied by this component within the coordinate system 27 can be determined for each component. Conversely, the data-based collection can be used to determine the component intended for a specific spatial volume. The level of detail with which the components of the device 1 are stored in the data-based collection can vary. For example, a component can be approximated by just a single cuboid, or the external contours of a component can be recorded in detail using a large number of voxels (volume pixels; 3D pixels).

[0145] Fig. 2 shows an augmented reality device 29 according to the second aspect of the invention.

[0146] The augmented reality device 29 is designed in the form of a smartphone and has a capture unit in the form of a camera 31, a screen 33, a display unit 35, an evaluation unit 37, and a processing unit 39. The camera 31 can capture an image of a section of the environment, in which the device 1 is at least partially visible. The screen 33 displays only a purely schematic image of a device.

[0147] Fig. 3a shows an image 41 of a section of the environment, recorded with the augmented reality device 29, in which the dosing device 1 from Fig. 1b is visible. The recording perspective of the image 41 is directed from a frontal view (as selected in Fig. 1b) offset to the left and above the device 1. The perspective is also visible in the rotated coordinate system 27 of the device 1, which is also shown for illustration purposes. In the image 41 shown, perspective effects are only indicated schematically; however, for reasons of simplicity of representation, parallel lines, for example, are still all parallel, even if they would actually run obliquely in the image 41 due to the selected perspective due to perspective distortion during the imaging process.In order to be able to provide a user of the augmented reality device 29, for example a service employee, with targeted information on the individual components of the device 1 (for example on the discharge element 13, the motor 17 or one of the weighing units 19, 21), user support information can be displayed in the image 41.

[0148] For this purpose, the marker 25 is identified in the recorded image 41 and the identified marker 25 is evaluated. In this case, the representation of the marker 25 in the image 41 is evaluated. For example, perspective effects of the marker 25 (such as a non-parallelism of edges of the marker 25 that are parallel in reality that occur in the image) and / or the representation of the known pattern of the marker 25 in the image 41 is evaluated (for example, a comparison between a distorted representation of the pattern in the image and the (known) undistorted pattern can be carried out at least as part of the evaluation). Based on this determined information, a reference coordinate system 43 can be defined with respect to the marker 25, which is viewed from the perspective selected for the image 31. The reference coordinate system 43 is defined in three-dimensional space.Once the reference coordinate system 43 is defined, the perspective from which the image 41 was captured can be determined. The reference coordinate system 43 has its origin in the center of the marker 25, and the marker surface lies within the YZ plane of the reference coordinate system 43. For illustrative purposes, the reference coordinate system 43 is shown at the location of the marker in Fig. 3a. The reference coordinate system 43 does not necessarily have to be visible; in principle, it is sufficient to further process the determined information, which allows the reference coordinate system 43 to be established.

[0149] Since the front side of the horizontal section 11, on which the marker 25 is arranged, lies in a plane parallel to the YZ plane of the coordinate system 27, the X, Y, and Z axes of the two coordinate systems 27 and 43 are parallel to one another, and only the zero points of the two coordinate systems 27 and 43 are offset from one another (with a known displacement vector). In principle, however, the reference coordinate system 43 could be rotated, tilted, and offset from the coordinate system 27 as desired, depending on how the surface of the marker 25 extends within the coordinate system 27. In any case, the coordinates of a location specified in coordinates of the reference coordinate system 43 can be converted into coordinates of the coordinate system 27, and vice versa.

[0150] This means that for each pixel of the image 41 showing a part of the device 1, a corresponding location on / at the device 1 can be assigned both in coordinates of the reference coordinate system 43 and in coordinates of the coordinate system 27. This is possible because the orientation of the reference coordinate system 43 is known or can be determined based on the determined information (which is necessary to define the reference coordinate system 43), and thus the virtual model of the device 1 described above can also be rotated and tilted such that it is viewed from the same perspective as the device 1 is recorded in the image 41. The parts of the model then visible from the respective perspective can be projected onto the image plane (i.e., the drawing plane of Fig. 3a), whereby an assignment between locations on / at the device 1 and pixels of the image 41 can be determined.

[0151] The pattern on marker 25 is a 2D code (such as a QR code or an April Day) in which a unique identifier of device 1 is encoded. The unique identifier is read from image 41 as marker information from marker 25 by decoding the 2D code. Based on the unique identifier, user support information is requested from a remote computer (e.g., a cloud computer). For this purpose, the marker information can be sent to the remote computer, whereupon the user support information is received by the remote computer. In addition to the marker information, a component of device 1 for which user support information is desired can also be specified.For example, an image area in the image 41 displayed on the screen 33 can be marked by touching and in this way (by including an assignment of different image areas to the respective components of the device 1) a component of the device 1 for which user support information is desired can be selected.

[0152] The user support information is assigned both to a component of device 1 and to an information class. Assigning the user support information to a component of device 1 enables a display position for the user support information to be determined component-specifically, for example, by incorporating the data-based collection. This allows the user support information to be displayed in the image 41 such that it is placed in the image near the location of the respective component or with reference thereto. An output position specified in coordinates of the coordinate system 27 or the reference coordinate system 43 can be converted into coordinates of the image 41, just as described above for the reverse direction.By assigning user support information to an information class, different types of user support information can be presented differently.

[0153] In the following, three scenarios are used to describe different possibilities for supporting a service employee as a user of the augmented reality device 29 during maintenance of the device 1.

[0154] In a first scenario, for example, a check of the motor 17 can be performed, and the user support information for this purpose can be a speed of the motor 17. The user support information can then be assigned to the motor 17 (component motor) and the information class "operating parameter."

[0155] In the data-based collection, an output position for the user support information of the "operating parameters" information class is stored for the motor 17 in the coordinate system 27 of the device 1. This output position is transferred to the reference coordinate system 43 using the information determined by the marker evaluation, thus determining the display position in the image 41. Furthermore, a perspective adjustment of the user support information is performed using the determined information.

[0156] Fig. 3b shows an image 45, which is the image 41 from Fig. 3a with the user support information 47 for the first scenario displayed therein. Due to the information class, the user support information 47 is selected as an instrument display with a pointer position corresponding to the current engine speed. The instrument display is also visually associated with the engine 17 in the image 45 by a reference line, making it easily comprehensible for the user. Furthermore, the user support information 47 is displayed in a perspectively adjusted manner, with the instrument display being perceived by the user of the device 29 as parallel to a plane of the marker surface when viewing the image 45.

[0157] In a second scenario, for example, the weighing unit 19 may be defective, and the user support information may be a highlight of the defective weighing unit 19. The user support information may then be assigned to the weighing unit 19 (weighing unit component) and the information class "Highlighting."

[0158] In the data-based collection, an output position for the user support information of the information class "Highlighting" is stored for the weighing unit 19 in the coordinate system 27 of the device 1. This output position is transferred to the reference coordinate system 43 using the information determined by the marker evaluation, thus determining the display position in the image 41. Furthermore, a perspective adjustment of the user support information is performed using the determined information.

[0159] Fig. 3c shows an image 49, which is the image 41 from Fig. 3a with the user support information 51 for the second scenario displayed therein. The

[0160] Due to the information class, user support information 51 is selected as a three-dimensional border around the weighing unit 19. The border is placed directly at the location of the weighing unit 19. Furthermore, the user support information 51 is displayed in a perspective view. This allows the user to be provided with helpful support during maintenance, for example, so that the correct weighing unit 19 (and not the identical weighing unit 21) is reliably serviced.

[0161] In a third scenario, for example, a check of the discharge device 13 can be performed, and for this purpose, the user support information can be a visual representation of the discharge device 13 (not visible during operation of the device 1). The user support information can then be assigned to the discharge device 13 (discharge device component) and the information class "visual representation."

[0162] In the data-based collection, an output position for the user support information of the information class "visual representation" is stored in the coordinate system 27 of the device 1 for the output device 19. This output position is transferred to the reference coordinate system 43 using the information determined by the marker evaluation, thus determining the display position in the image 41. Furthermore, a perspective adjustment of the user support information is performed using the determined information.

[0163] Fig. 3d shows an image 53, which is image 41 from Fig. 3a with the user support information 55 for the third scenario displayed therein. The user support information was selected as a visual representation of the discharge element 19 due to the information class. The visualization is also assigned to the location where the discharge element 19 is located in the device 1 (i.e., within the horizontal section 11) by means of dashed reference lines. This makes the position of the component easy for the user to grasp. Furthermore, the visualization is presented with an adjusted perspective, and the impression of the spatial course of the visualized discharge element corresponds to the actual course of the discharge element 13. Although the installed discharge element 13 is not visible in image 41, the user support information 55 can still make this component 13 visible, thus providing the user with helpful support.

[0164] In an alternative embodiment, the user support information could also be displayed in a field of view of the user that at least partially contains the environmental section. For this purpose, instead of the augmented reality device 29, the user could use a modified augmented reality device, for example, in the form of glasses that have a camera and a projection unit for displaying the user support information. Fig. 4 shows a flowchart 100 of a method according to the first aspect of the invention.

[0165] At 101, an image of a section of the environment in which a device is at least partially visible is captured. This can be done, for example, using the camera 31 of the augmented reality device 29. The device can be the device 1 described with reference to Figs. 1a and 1b.

[0166] At 103, a marker associated with the device is identified in the image, and the identified marker is evaluated. This can be done using the evaluation unit 37 of the augmented reality device 29.

[0167] At 105, user support information to be displayed in the image is obtained. For example, marker information can be read from the marker or determined based on the marker, and the user support information can be requested based on the marker information.

[0168] At 107, based at least partially on a result of the marker evaluation, a position at which the user support information is displayed in the image or in the user's field of vision is determined as the display position, and a perspective adjustment of the user support information or parts thereof is performed. This can be done, for example, by means of the processing unit 39 of the augmented reality device 29.

[0169] At 109, the user support information is displayed in the image or in a field of view of a user of the augmented reality application that at least partially contains the environmental section. This can be done, for example, using the display unit 35 of the augmented reality device 29. The captured image with the user support information displayed therein can then be output on the screen 33.

[0170] The method is thus advantageously suited for displaying user support information relating to at least one device or one of its components in an augmented reality application. For example, the types of user support information described with reference to Figs. 3b-3d can be displayed.

[0171] The features disclosed in the foregoing description, in the drawings and in the claims may be essential to the invention in its various embodiments, both individually and in any combination.

[0172] List of reference symbols

[0173] 1 dosing device

[0174] 3 Bulk goods

[0175] 5 receiving containers

[0176] 7 Bulk material surface

[0177] 9 Floor opening

[0178] 11 Horizontal section

[0179] 13 Discharge organ

[0180] 15 Vertical drop

[0181] 17 Engine

[0182] 19 Weighing unit

[0183] 21 Weighing unit

[0184] 23 Supporting frame

[0185] 25 markers

[0186] 27 Coordinate system of the device

[0187] 29 Augmented Reality Facility

[0188] 31 Camera

[0189] 33 screen

[0190] 35 display unit

[0191] 37 Evaluation unit

[0192] 39 processing unit

[0193] 41 Image

[0194] 43 Reference coordinate system

[0195] 45 Image

[0196] 47 User Support Information

[0197] 49 Image

[0198] 51 User Support Information

[0199] 53 Image

[0200] 55 User Support Information

[0201] 100 Flowchart 101 Taking an image of an area of ​​the environment in which a

[0202] Device is at least partially visible

[0203] 103 Identifying a marker associated with the device in the image and evaluating the identified marker

[0204] 105 Obtaining a user support message to be displayed in the image

[0205] information

[0206] 107 Determining a display position for the user support

[0207] Information in the image or in the field of view of a user and making a perspective adjustment of the user support information or parts thereof, each at least partly based on a result of the evaluation of the marker

[0208] 109 Displaying the user support information in the image or in a field of view of the user that at least partially contains the surrounding area

[0209] X, Y, Z Coordinates

Claims

Patent claims 1. A method for displaying at least one item of user support information relating to at least one device or one of its components in an augmented reality application, wherein at least one image of a section of the environment in which the device is at least partially visible is recorded, wherein the user support information to be displayed is obtained and displayed in the image or in a field of view of a user of the augmented reality application that at least partially contains the section of the environment, wherein a marker associated with the device is identified in the image, and the identified marker is evaluated, and wherein, at least partially based on at least one result of the evaluation, (i) a position at which the user support information is displayed in the image or in the field of view of the user,as the display position and (ii) a perspective adjustment of the user support information or parts thereof is made.

2. Method according to claim 1, wherein the display position is also determined as a function of the user support information to be displayed, in particular a categorization of the user support information to be displayed, and / or the component to which the user support information to be displayed is assigned.

3. Method according to one of the preceding claims, wherein at least some, preferably all, components of the device together with the information respectively assigned to them are stored in a data-based collection and the data-based collection is included in the determination of the display position and / or in the perspective adjustment, wherein preferably the information assigned to the components each represents or makes it possible to determine at least one display position for a user support information assigned to the respective component.

4. Method according to one of the preceding claims, wherein the marker contains readable and / or decodable information, such as a unique identifier of the device, as marker information and the marker information is read and / or decoded from the marker and the user support information is requested, in particular from a memory and / or from a remote computer, at least partly based on the marker information.

5. Method according to one of the preceding claims, wherein information on the geometry of the device and / or its components is stored for the marker, in particular in the data-based collection, and the geometry information is included in the determination of the fade-in position and / or in the perspective adjustment.

6. Method according to one of the preceding claims, wherein the user support information is associated with a specific component of the device, wherein preferably the specific component is a component of the device with a defect or impending maintenance and / or the device has a plurality of identical components as the specific component.

7. Method according to one of the preceding claims, wherein the evaluation of the marker comprises determining, at least in part, based on the representation of the marker in the image, at least some or all of the information necessary for defining a, in particular three-dimensional, reference coordinate system with respect to the marker, in particular viewed from the perspective selected for recording the image, and wherein the determined information is used at least in part to determine the display position of the user support information and / or to adapt the perspective of the user support information, wherein preferably the reference coordinate system is a coordinate system of the device and / or the reference coordinate system can be converted into a coordinate system of the device and vice versa in a previously known manner.

8. The method according to claim 8, wherein the user support information and / or the specific component is assigned an output position which indicates a position within the reference coordinate system or which is converted into a position within the reference coordinate system, and wherein the display position of the user support information is determined based on the position within the reference coordinate system.

9. Method according to one of the preceding claims, wherein the user support information is assigned to at least one information class of several possible information classes and the form of presentation of the user support information is selected at least partly as a function of the information class.

10. Method according to one of the preceding claims, wherein the user support information (i) is a table-like overview with information on one or more operating parameters of the device or the specific component and / or on one or more process parameters of a process at least partially carried out with the device, (ii) a graphical representation, such as a diagram, a chart, an instrument display and / or a pictogram, of one or more operating parameters of the device or the specific component or of one or more process parameters of a process at least partially carried out with the device, (iii) is a graphic illustration of the specific component and preferably wherein the specific component is not visible in the captured image or in the user’s field of view, (iv) is an overview of documentation relating to the device or specific component or a reference to such an overview or documentation, (v) highlighting, in particular by enclosing, an area occupied by parts of the device or by the specific component and / or highlighting a Contour of parts of the device or the specific component, each in the image or in the field of vision of the user, and / or (vi) a visualization of at least one measuring section and / or at least one measuring location in the image or in the field of vision of the user.

11. Method according to one of the preceding claims, wherein the user of the augmented reality application is guided step by step through at least one work process, in particular through at least one maintenance process and / or through at least one measuring process, and for this purpose, at least some, preferably at every, steps an updated user support information is displayed in the image or in an at least partially newly recorded image or in the field of vision of the user.

12. Method according to one of the preceding claims, wherein a pixel or image region in the image is selected by the user, in particular by a selection via a touch-sensitive screen, and the specific component is selected based on this user selection. wherein preferably (i) the pixel or image region of the image selected by the user selection is assigned to a component of the device via an assignment, in particular based on the data-based collection, and this component is selected as the specific component and / or (ii) different components of the device are assigned to different pixels or image regions in the image.

13. Augmented reality device for providing at least one item of user support information relating to at least one device or one of its components to a user, the device comprising at least one detection unit configured to record at least one image of a section of the environment in which the device is at least partially visible, at least one display unit configured to receive the user support information to be provided and to display it in the image or in a field of view of the user of the augmented reality device that at least partially contains the section of the environment, at least one evaluation unit configured to identify a marker assigned to the device in the image and to evaluate the identified marker, and at least one processing unit configured toat least partially based on at least one result of the evaluation (i) to determine a position at which the user support information is displayed in the image or in the field of view of the user as the display position and (ii) to perform a perspective adjustment of the user support information or parts thereof.

Images