Method for operating an augmented reality system

Abstract

A method for generating an augmented reality image (ARB), wherein: - a real image (RB) of a real object is captured using a real camera; - an edge image of the real object is generated from the real image (RB) of the real object, wherein the position of a virtual image component (VIRT) relative to the real image (RB) captured by the real camera is determined using the edge image; and - the virtual image component (VIRT) is combined with at least a part of the real image (RB) to form an augmented reality image (ARB), wherein, for markerless tracking of the edge image, - an initialization phase is provided for the initialization of the markerless tracking, in which an initialization edge image (KB) of the real object is generated depending on the pose of the real object from the perspective of the real camera, by - providing a database in which images of the object in different poses are stored.where each image is assigned a pose, - a real initialization image (RB) of the real object is captured using the real camera, whereby the image from the database that is most similar to the initialization image is automatically selected, and - the pose assigned to the selected image from the database is selected to generate the initialization edge image (KB) by transforming the edge image of the real object into the selected pose.

Description

The invention relates to a method for generating an augmented reality image and a method for generating an augmented reality image or for displaying a virtual image component on a transparent display. Markerless tracking is performed using an edge image. According to WO 01 / 96829 A1, the term augmented reality is defined as the overlay of a real environment with a computer-generated environment, i.e., a virtual world. Augmented reality is thus a type of human-technology interaction that overlays information into the user's field of vision, thereby expanding their perception. DE 10 2012 014 995 A1 discloses a method for generating an augmented reality image, wherein a real image of a real environment is captured using a real camera, wherein the position of a virtual image component relative to the real image captured by the real camera is determined by means of a feature image, and wherein the virtual image component is combined with at least a part of the real image to form an augmented reality image, wherein an artificial (or synthetic) image of at least one object from the environment to be captured is generated using CAD data, which advantageously depicts or represents or includes material properties of the object's surfaces, and wherein the feature image is generated or created depending on the artificial image. In particular, it is provided that the feature image is generated or created depending on a plurality of artificial (or synthetic) images of the object. German patent DE 10 2004 061 842 A1 discloses a method for determining the position and orientation of mobile devices with respect to real objects, wherein a mobile device receives measurement instructions from a server and only the required measurement data is sent from the client to the server. The image is calculated using so-called enriched information by the client. German patent DE 10 2006 055 893 A1 discloses a method for determining a data model of virtual information superimposed on an image of a real object. First, a three-dimensional CAD model is provided to replicate the real object. This is followed by the generation of different synthetic two-dimensional views of the CAD model, with each generated view being used for edge extraction to determine at least one extracted edge in the respective view. Finally, the edges extracted from the respective views are converted into a three-dimensional contour model that corresponds to the data model. The paper by Hoermann et al., "Robot Localisation Using 3D-Models and an Off-Board Monocular Camera," published in Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1006-1013, addresses robot localisation, one of the most common tasks in autonomous systems. Unlike most vision-based methods that perform localisation using cameras mounted on the robot, this paper proposes a system that uses an external camera for localisation. To this end, a similarity measure is used between a camera image and a synthetic image generated from a 3D object model. Unlike other methods, a prior training phase is unnecessary because we use a model for shading the appearance based on the surface curvature of the 3D model.Assuming a reasonably planar area observed by an external camera, an initial position estimate in the camera image (based on 2D blob tracking) allows the 3D model to be rendered near the object in the image. From this initial estimate, the model is then compared with the real image, allowing it to converge to the actual vehicle pose. The German patent application DE 10 2012 014 995 A1 relates to a method for generating an augmented reality image (ARB), wherein a real image (RB) of a real environment is captured using a real camera, wherein the position of a virtual image component (VIRT) relative to the real image (RB) captured by the real camera is determined using a feature image (MB), and wherein the virtual image component (VIRT) is combined with at least a part of the real image (RB) to form an augmented reality image (ARB), and wherein an artificial image of at least one object from the environment to be captured is generated using CAD data, which advantageously depicts material properties of surfaces of the object, and wherein the feature image (MB) is generated depending on the artificial image. Document US 2015 / 0023602 A1 concerns systems and methods for the rapid detection or differentiation of objects. The systems under consideration include an object model database in which recognition models are stored that are associated with known modeled objects. The object identifiers can be indexed in the object model database based on recognition features derived from keyframes of the modeled object. Such objects are subsequently recognized by a recognition engine. The recognition engine can generate a recognition strategy based on a current context, with the recognition strategy containing rules for executing one or more recognition algorithms on a digital representation of a scene.The recognition engine can recognize an object from the object model database and then attempt to identify contextually relevant keyframe bundles, which can then be used to track the object or query a content database for content information. Document WO 2015 / 036056 A1 relates to a method for determining a model that describes at least part of a real object, comprising the following steps: providing a first image of at least part of the real object to a first processing device, wherein the first image is captured by a camera and at least part of the first image is sent to a second processing device that is physically separate from the first processing device; determining, on the second processing device, a registration between the first image and a first model provided on the second processing device, corresponding to at least part of the first image received on the second processing device; determining the geometric information for at least one pixel in the first image according to the registration; and sending the geometric information from the second processing device to the first processing device.and by using at least some of the geometric information received at the first processing device, together with at least some of the first image, to determine on the first processing device a second model containing geometric and material information that describes at least some part of the real object. Publication GB 2 411 532 A concerns a method for determining the position and orientation of a camera that does not rely on special markers. A set of reference images is stored, along with camera poses and feature information for each image. An initial estimate of the camera position is obtained by comparing the current camera image with the set of reference images. A more refined estimate can be obtained by using features that match in the current image and in a subset of similar reference images, particularly the 3D positions of these features. A consistent 3D model of all stored feature information does not need to be provided. Edge images or edge models are used for so-called markerless tracking. This involves an initialization phase in which a predefined pose is assumed to align the vehicle's edge model with the real vehicle. The object of the invention is to improve or improve the operation of an augmented reality system. In particular, the initialization of the augmented reality system should be simplified. It is especially desirable that the initialization can be carried out even without prior knowledge of the augmented reality system's process. This problem is solved by a method for generating an augmented reality image with the features of claim 1 and by a method for displaying a virtual image component on a preferably transparent display with the features of claim 2. Preferred embodiments of the invention are the subject of the further claims.According to the invention, in a method for generating an augmented reality image, a real image of a real object is captured using a real camera; an edge image of the real object is generated from the real image of the real object, wherein the position of a virtual image component relative to the real image captured by the real camera is determined using the edge image; and the virtual image component is combined with at least a part of the real image to form an augmented reality image. For markerless tracking of the edge image, an initialization phase is provided, in which an initialization edge image of the real object is generated depending on the pose of the real object from the perspective of the real camera by providing a database in which images of the object in different poses are stored, with each image being assigned a pose.- a real initialization image of the real object is captured using the real camera, whereby the image from the database that is most similar to the initialization image is automatically selected, and - the pose associated with the selected image from the database is selected to generate the initialization edge image by transforming the edge image of the real object into the selected pose, According to the invention, in a method for displaying a virtual image component on a preferably transparent display, a real image of a real object is captured by means of a real camera and is visible through the display. An edge image of the real object is generated from the real image of the real object, the position of a virtual image component relative to the real image captured by the real camera is determined by means of the edge image, and the virtual image component is displayed by means of the display. For markerless tracking of the edge image, an initialization phase is provided, in which an initialization edge image of the real object is generated depending on the pose of the real object from the perspective of the real camera. This is achieved by providing a database in which images of the object in various poses are stored.where each image is assigned a pose, - a real initialization image of the real object is captured using the real camera, whereby the image from the database that is most similar to the initialization image is automatically selected, and - the pose assigned to the selected image from the database is selected to generate the initialization edge image by transforming the edge image of the real object into the selected pose. In one embodiment, the camera and the display are integrated into a single device. Such a device could be, for example, a smartphone or a tablet. Furthermore, an initialization edge image is an edge image used for initialization, where the term "edge image" is used synonymously with the term "edge model". A database is therefore provided or created beforehand, in which images of the object in different poses are stored, with each image being assigned a pose. Subsequently, a real initialization image of the real object is captured using the actual camera. The image from the database that most closely resembles this initialization image is automatically selected. A real initialization image is a real image used during the initialization process. The pose associated with the selected image from the database is then chosen. A pose, as defined by the invention, is specifically a pose defined in DIN EN ISO 8373 (Industrial Robot Dictionary). A pose, as defined by the invention, is specifically a combination of position and orientation in three-dimensional space. In a further advantageous embodiment of the invention, the pose comprises the distance between the real camera and the real object. In a further advantageous embodiment of the invention, the real object is a motor vehicle, wherein the pose comprises the rotation angle of the motor vehicle about its vertical axis from the perspective of the real camera, or wherein the pose comprises the yaw angle from the perspective of the real camera. The position of the vehicle's vertical axis or the definition of the yaw angle can be found, for example, in the book "Bosch - Driving Safety Systems", 2nd edition, ISBN 3-528-03875-6, page 239. In a further advantageous embodiment of the invention, the pose comprises the distance between the real camera and the motor vehicle and the rotation angle of the motor vehicle about its vertical axis from the perspective of the real camera (or the yaw angle from the perspective of the real camera).In a further advantageous embodiment of the invention, the pose comprises only the distance between the real camera and the motor vehicle and the rotation angle of the motor vehicle about the vehicle's vertical axis from the viewpoint of the real camera (or the yaw angle from the viewpoint of the real camera). Another objective of the invention is to simplify or make more rational the manufacture of technical components, in particular the manufacture of motor vehicles or motor vehicle components. The aforementioned task is also solved – particularly in conjunction with the aforementioned features – by a method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, wherein a component to be assembled is marked or indicated by means of the virtual image component according to the method described above, and wherein the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component. The aforementioned task is also solved – particularly in conjunction with the aforementioned features – by a method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, wherein the mounting location of a component to be mounted is marked or indicated by means of the virtual image component according to the method described above, and wherein the component to be mounted is mounted in the technical component, in the motor vehicle or in the motor vehicle component at the mounting location. The aforementioned task is also solved – particularly in conjunction with the aforementioned features – by a method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, wherein an assembly sequence for mounting a component to be mounted is determined or verified using the virtual image component according to the method described above, and wherein the component to be mounted is mounted in the technical component, in the motor vehicle or in the motor vehicle component according to the assembly sequence. A virtual image component within the meaning of the invention(s) may comprise or consist of alphanumeric characters. A virtual image component within the meaning of the invention(s) may comprise alphanumeric characters and a colored background for these characters. A virtual image component within the meaning of the invention(s) may consist of alphanumeric characters in combination with a colored background for these characters. A motor vehicle within the meaning of the invention is, in particular, a land vehicle that can be used individually in road traffic. Motor vehicles within the meaning of the invention are not limited to land vehicles with internal combustion engines. Further advantages and details will become apparent from the following description of exemplary embodiments. Figure 1 shows an exemplary embodiment of a system for displaying an augmented reality image, Figure 2 an exemplary embodiment of an initialization module, Figure 3 an exemplary embodiment of a database, Figure 4 an exemplary embodiment for a virtual image displayed by means of a display designed as a transparent display, Figure 5 an exemplary embodiment of a method for initializing a system for displaying an augmented reality image, Figure 6 an exemplary embodiment of a method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, Figure 7 a further exemplary embodiment of a method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, and Figure 8...8 Another embodiment of a method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component. Fig. 1 shows a system for displaying an augmented reality image ARB. The system comprises a head-mounted display (HMD) 10 worn by a user 1, with a display 11 for displaying the augmented reality image ARB. A camera associated with the display 11 is mounted in the head-mounted display (HMD) 10 to capture a real-world image of the user's surroundings or an object, such as the motor vehicle designated by reference numeral 25. Instead of a head-mounted display (HMD), a tablet, for example, can also be used, with the camera and display integrated into the tablet. The real-world image RB output by the camera is the input signal to a markerless tracking system 12, which determines the orientation (position / position signal) POS of the real-world image RB as a function of a (stored) feature image MB.The orientation (position / position signal) POS of the real image RB and optionally (see below) the real image RB are output signal of the tracking system 12 and input signal to a scene generator 13. The system for displaying an augmented reality image (ARB) also includes a database 14 with virtual image components or another source of virtual image components. A virtual image component (VIRT) from this database 14 or other source is supplied to the scene generator 13. The scene generator 13 combines this component with the real image (RB) to form an augmented reality image (ARB), which is displayed by means of the display 11. The display 11 may be designed as a transparent display. In this case, the augmented reality image (ARB) is not a superimposition of the real image (RB) with the virtual image component (VIRT), but rather the virtual image component (VIRT) is positioned at a specific location so that it can be displayed at that location by means of the transparent display. The superimposition between reality and the virtual image component occurs in the eye of the user (Fig. 1).Figure 5 shows an embodiment of a virtual image VIRT, which is displayed by means of the display 11 designed as a transparent display. In this embodiment, the real object to which the virtual image VIRT is assigned is a motor vehicle 25'. An initialization module 15, shown in detail in Fig. 2, is provided for initializing the augmented reality system. The initialization module 15 comprises a database 153 containing various views of the vehicle 25 with an associated description of the pose in the image. The images of the vehicle 25 are stored in the database 153 in a tree structure. These images can be real photographs or artificially generated images, for example, images created using a CAD system. In this case, a pose comprises only the distance A between the vehicle 25 and a camera, as well as the angle z of the vehicle about its vertical axis with respect to the camera.The initialization module 15 also includes a comparison module 151, which compares an image RB of the vehicle 25 captured by the camera (initialization image) with the images of the vehicle stored in the database 153. During initialization, the actual image RB is referred to as the initialization image. The initialization module 15 also includes a module 152, which generates an edge image KB or initialization edge image KB depending on the pose, and this edge image is used for initialization. Fig. 4 shows a method for operating the initialization module 15. In step 31, the application is started (i.e., initialized) by a user. This is followed by step 32, in which the user aligns the camera or a corresponding mobile device with a camera with the vehicle 25. In step 33, images are captured using the camera (of the mobile device) and compared with the images stored in the database 153. In step 34, the pose most similar to the captured image is selected and an edge model or edge image of the vehicle 25 is transformed into the stored pose. The pose is represented by the distance A or Ai between the camera and the vehicle and by the angle z or zj about the vehicle's vertical axis. Based on the transformed edge model, or edge image, the system then...In step 35, the transformed edge image undergoes an initialization process in a known manner. After completion of the initialization process, markerless tracking based on the edge model or edge images is performed in a known manner. It may be intended that the pose includes further parameters, such as open doors or open flaps, angular positions of open doors or flaps, or the construction condition of a motor vehicle. Fig. 6 shows an embodiment of a method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, wherein in step 41 a component to be mounted is marked or characterized by means of the virtual image component according to the method described above, and wherein in step 42 the component to be mounted is mounted in the technical component, in the motor vehicle or in the motor vehicle component. Fig. 7 shows an embodiment of a method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, wherein in step 51 the mounting location of a component to be mounted is marked or indicated by means of the virtual image component according to the method described above, and wherein in step 52 the component to be mounted is mounted in the technical component, in the motor vehicle or in the motor vehicle component at the mounting location. Fig. 8 shows an embodiment of a method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, wherein in step 61 an assembly sequence for mounting a component to be mounted is determined or verified using the virtual image component according to the method described above, and wherein in step 62 the component to be mounted is mounted in the technical component, in the motor vehicle or in the motor vehicle component according to the assembly sequence.

Claims

A method for generating an augmented reality image (ARB), wherein: - a real image (RB) of a real object is captured using a real camera; - an edge image of the real object is generated from the real image (RB) of the real object, wherein the position of a virtual image component (VIRT) relative to the real image (RB) captured by the real camera is determined using the edge image; and - the virtual image component (VIRT) is combined with at least a part of the real image (RB) to form an augmented reality image (ARB), wherein, for markerless tracking of the edge image, - an initialization phase is provided for the initialization of the markerless tracking, in which an initialization edge image (KB) of the real object is generated depending on the pose of the real object from the perspective of the real camera, by - providing a database in which images of the object in different poses are stored.where each image is assigned a pose, - a real initialization image (RB) of the real object is captured using the real camera, whereby the image from the database that is most similar to the initialization image is automatically selected, and - the pose assigned to the selected image from the database is selected to generate the initialization edge image (KB) by transforming the edge image of the real object into the selected pose. Method for displaying a virtual image component (ARB, VIRT) on a preferably transparent display (11), wherein: - a real image (RB) of a real object is captured by means of a real camera, which is visible through the display (11); - an edge image of the real object is generated from the real image (RB) of the real object, wherein the position of a virtual image component (VIRT) relative to the real image (RB) captured by means of the real camera is determined by means of the edge image; and wherein the virtual image component (VIRT) is displayed by means of the display (11), wherein, for markerless tracking of the edge image, - an initialization phase for the initialization of the markerless tracking is provided, in which an initialization edge image (KB) of the real object is generated depending on the pose of the real object from the viewpoint of the real camera by providing a database.in which images of the object are stored in different poses, with each image assigned a pose; - a real initialization image (RB) of the real object is captured using the real camera, whereby the image from the database that is most similar to the initialization image is automatically selected; and - the pose assigned to the selected image from the database is selected to generate the initialization edge image (KB) by transforming the edge image of the real object into the selected pose. Method according to one of the preceding claims, characterized in that the pose includes the distance between the real camera and the real object. Method according to one of the preceding claims, characterized in that the real object is a motor vehicle, wherein the pose comprises the rotation angle of the motor vehicle about the vehicle's vertical axis from the view of the real camera, or wherein the pose comprises the yaw angle from the view of the real camera. Method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, characterized in that a component to be mounted is marked or characterized by means of the virtual image component (VIRT) according to a method according to one of the preceding claims, and that the component to be mounted is mounted in the technical component, in the motor vehicle or in the motor vehicle component. Method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, characterized in that the mounting location of a component to be mounted is marked or indicated by means of the virtual image component (VIRT) according to a method according to one of claims 1 to 4, and that the component to be mounted is mounted in the technical component, in the motor vehicle or in the motor vehicle component at the mounting location. Method for manufacturing a technical component, in particular a motor vehicle or a motor vehicle component, in particular a method according to claim 5 or 6, characterized in that an assembly sequence for assembling a component to be assembled is determined or verified by means of the virtual image component (VIRT) according to a method according to one of claims 1 to 4, and that the component to be assembled is assembled in the technical component, in the motor vehicle or in the motor vehicle component in accordance with the assembly sequence.

Images