Method and device for determining a glasses pose of pair of smart glasses by means of outside-in tracking

Abstract

The invention relates to a method for determining a pose of at least one pair of smart glasses in an interior of a mobile machine, comprising the following steps: - capturing a camera image of an interior of the mobile machine; - evaluating a data-based glasses-type recognition model on the basis of the captured camera image in order to determine a relevant glasses type for at least one pair of smart glasses imaged in the camera image, the data-based glasses-type recognition model being trained to assign a camera image to a relevant glasses type of one or more pairs of smart glasses; - selecting, in each case, one of a plurality of data-based pose recognition models depending on the recognized glasses type, the data-based pose recognition models each being trained for one glasses type, in order to determine a glasses pose of the corresponding pair of smart glasses according to the camera image; - determining, in each case, the glasses pose of the pair of smart glasses of the corresponding glasses type by means of the selected pose recognition model.

Description

[0001] Description

[0002] Method and device for determining a spectacle pose of data glasses using outside-in tracking

[0003] The invention relates to data glasses, in particular for use in mobile devices. The invention further relates to measures for tracking a glasses pose based on a pose recognition system external to the glasses.

[0004] Data glasses, also called head-mounted displays, are known that, with the aid of a display device, can display an image on one or two display surfaces in the wearer's field of vision. In so-called augmented reality data glasses, the display surfaces can correspond to reflective surfaces that direct images into the wearer's eyes. The viewing openings of the data glasses are transparent, allowing the real environment to be perceived through the data glasses in the usual way. The display surfaces are located in the viewing openings, allowing information to be displayed, such as text, symbols, graphics, video displays, and the like, to be displayed overlaid on the perception of the environment.

[0005] The information is usually presented to the wearer of the data glasses as information objects in a contact-analog manner, i.e., presented in such a way that these information objects are superimposed on a specific assigned object in the real environment or are oriented towards it, or that the information object to be displayed is displayed in a specific orientation of the data glasses or their wearer. In order to display the information objects in a contact-analog manner on the display surfaces of the data glasses, it is necessary to know the position of the object in the environment and the user's line of sight. When wearing the data glasses, the user's line of sight is permanently assigned to their pose, i.e., the 3D position and the 3D orientation of the data glasses.

[0006] The pose of the smart glasses can be determined by an external pose recognition device, in which an indoor camera is aimed at the head of the smart glasses wearer and the smart glasses and determines the pose of the smart glasses by evaluating the camera image. The indoor camera is usually an infrared camera to enable accurate detection of the area of ​​the smart glasses even in poor lighting conditions. This type of tracking method is called outside-in tracking and is usually based on a data-based pose recognition model, which is designed as a data-based pattern or object recognition model.

[0007] The pose recognition model can be implemented in a conventional manner as an artificial neural network or the like and trained to assign one or more camera images depicting a pair of smart glasses to a pair of smart glasses pose using a conventional pattern recognition method. The pair of smart glasses pose has a reference to a vehicle coordinate system, which serves as the reference frame. This pair of smart glasses pose is then transmitted to the smart glasses so that a contact-analog display of an information object can be performed. In this regard, the pair of smart glasses pose provided externally is often enhanced using sensor data fusion with acquisitions from motion sensors internal to the smart glasses in order to provide the pair of smart glasses with the lowest possible latency.

[0008] To detect the pose of smart glasses from camera images, the data-based pose recognition model can be trained to determine the pose from a camera image depicting smart glasses worn on a user's head. Such pose recognition models are trained with training datasets that best represent reality. However, smart glasses for use in vehicles are offered by various manufacturers and have individual, slightly different shapes, which are expressed in different lens shapes, frame shapes, frame colors, and the like. This requires pose recognition models that rely on pattern recognition based on the camera image to recognize the various characteristics of the smart glasses in order to reliably determine a pair of smart glasses' pose.To achieve this, the different data glasses shapes from different manufacturers must be taken into account when training the pose recognition model.

[0009] Therefore, a pose recognition model must be trained with the different glasses shapes from different manufacturers to enable reliable pose determination.

[0010] It is an object of the present invention to provide an improved method for determining a spectacle pose based on a pose recognition model that is easy to train and has a reduced complexity.

[0011] This object is achieved by the method for determining a spectacle pose of data glasses in a mobile device using a data-based model system according to claim 1 and by the method for training a model system with a spectacle type recognition model for the individual spectacle type recognition of data glasses and a plurality of pose recognition models for a pose determination of the data glasses according to the independent claim.

[0012] Further embodiments are specified in the dependent claims. According to a first aspect, a method for determining a pose of at least one pair of data glasses in an interior of a mobile device is provided, comprising the following steps:

[0013] Capturing a camera image of an interior of the mobile device, in particular using an interior camera;

[0014] Evaluating a data-based glasses type recognition model based on the captured camera image to determine a respective glasses type for at least one pair of data glasses depicted in the camera image, wherein the data-based glasses type recognition model is trained to assign a camera image to a respective glasses type of one or more pairs of data glasses;

[0015] Respectively selecting one of several data-based pose recognition models depending on the respectively recognized type of glasses, wherein the data-based pose recognition models are each trained on a type of glasses in order to determine a glasses pose of the corresponding data glasses depending on the camera image;

[0016] Determining the respective glasses pose of the data glasses of the corresponding glasses type based on the selected pose recognition model.

[0017] One idea of ​​the above method is to provide a model system that determines a glasses pose of smart glasses in the interior of a mobile device using an interior camera external to the glasses. The model system comprises a glasses type recognition model that first searches the camera image for any smart glasses located there and assigns a glasses type depending on a shape and / or other characteristic of a recognized smart glasses. The glasses type recognition model therefore only needs to be trained to recognize smart glasses of a specific glasses type and to assign them to the recognized glasses type. Depending on the recognized glasses type, a corresponding pose recognition model can be selected from a plurality of data-based pose recognition models, each assigned to a specific glasses type, and then used to determine a glasses pose of the displayed smart glasses.

[0018] The data-based pose recognition model is trained specifically for the specific type of glasses and therefore does not consider other types or shapes of glasses. Since only one type of glasses, i.e., a specific shape of data glasses, needs to be considered for training the data-based pose recognition model, this pose recognition model can be simpler and less complex, and requires a correspondingly smaller number of training data sets that only consider poses of the specific type of glasses. The implementation for evaluating such a pose recognition model is also less complex.

[0019] According to a further embodiment, the glasses type recognition model can be used to determine a bounding box of the respectively recognized data glasses simultaneously with the glasses type recognition, wherein the respective pose recognition model is applied to a section of the camera image that is determined by the bounding box.

[0020] Thus, the glasses type recognition model can determine a bounding box and thus enable a search space restriction, so that the pose recognition model is selected by specifying the glasses type and can then perform the pose determination in a simpler way in a corresponding section of the captured camera image by specifying the search frame.

[0021] It can be provided that the pose of the at least one data glasses is transmitted to the respective data glasses, so that a contact-analog display can be shown in the respective data glasses depending on their pose.

[0022] Furthermore, model parameters of the selected pose recognition model can be retrieved from a database and implemented to provide the corresponding pose recognition model. The database can be located in a cloud, i.e., outside the mobile device.

[0023] According to a further aspect, a device for determining a pose of at least one pair of data glasses in an interior of a mobile device is provided, comprising: an interior camera designed to capture a camera image of an interior of the mobile device; a processor unit designed to evaluate a data-based glasses type recognition model based on the captured camera image in order to determine a respective glasses type for at least one pair of data glasses depicted in the camera image, wherein the data-based glasses type recognition model is trained to assign a camera image to a respective glasses type of one or more pairs of data glasses;o to select one of several data-based pose recognition models depending on the respectively detected glasses type, wherein the data-based pose recognition models are each trained for a specific glasses type in order to determine a glasses pose of the corresponding data glasses depending on the camera image; and o to determine the glasses pose of the data glasses of the corresponding glasses type based on the selected pose recognition model;

[0024] Furthermore, a communication unit can be designed to transmit the glasses pose to the data glasses.

[0025] Embodiments are explained in more detail below with reference to the accompanying drawings. Figure 1 shows a display system with data glasses for use in a vehicle;

[0026] Figure 2 is a flowchart illustrating a method for pose recognition of data glasses in the interior of a vehicle; and

[0027] Figure 3 different shapes of data glasses.

[0028] Figure 1 shows a schematic representation of a display system 1, particularly for use in a motor vehicle. The display system 1 comprises an assistance system 2, which has a communication connection 4 with data glasses 3. The communication connection 4 is designed as a data transmission channel, e.g., in the form of a wireless communication connection or a wired communication connection. The communication connection 4 is capable of transmitting any type of data and information between the assistance system 2 and the data glasses 3, for example, based on packet-based data transmission. The communication connection 4 can be based, for example, on WiFi, Bluetooth, Bluetooth Low Energy, or a comparable standardized radio protocol.

[0029] The assistance system 2 can be part of a vehicle assistance system and, in particular, can be provided in a fixed location in the motor vehicle. The assistance system 2 can be equipped with a communication unit 21 that enables the communication connection 4 between the data glasses 3 and the assistance system 2.

[0030] The assistance system 2 can further be provided with an interior camera 22 as a camera directed at the data glasses 3, which is directed at a likely area of ​​the head of a user / wearer of the data glasses 3. The interior camera 22 can comprise, for example, an RGB camera or an infrared camera.

[0031] The assistance system 2 can have a processor unit 23. The processor unit 23 can implement a data-based model system, which is described in more detail below. The model system includes a glasses type recognition model and a plurality of pose recognition models, each trained to determine a glasses pose based on a camera image from the interior camera 22. The glasses pose is then transmitted to the data glasses 3 via the communication unit 21.

[0032] For display in the data glasses 3, object information is generated which specifies the object position, object content and object type of the display of the at least one virtual display object in the data glasses 3.

[0033] The data glasses 3 comprise two transparent viewing panels 32, which are enclosed in a frame 31 in a conventional manner. The frame 31 is provided with temples 33, so that the data glasses 3 can be worn on the head of a user in a conventional manner.

[0034] One or both viewing panels 32 (lenses) are further provided with a transparent display surface 35, through which a suitable device, such as a display device 36 arranged on the frame 31, can project a display image for displaying virtual display objects into the eye of the wearer of the data glasses 3. The display device 36 can comprise a microprocessor or a comparable computing unit and a display unit, such as a projection device or the like. The display unit can be configured to direct the electronically generated display image onto the display surface 35 and project / display it there.

[0035] Due to the transparent design of the display surface 35, the electronically generated image can be superimposed (augmented) on the real environment perceivable through the display surface 35. Using the display device 36, a virtual display object, such as text, a symbol, video information, a graphic, or the like, can be displayed on one or both display surfaces 35. The object information defines the object position and representation of the at least one virtual display object in relation to the motor vehicle, i.e., in the vehicle coordinate system (reference system of the motor vehicle).

[0036] Furthermore, the data glasses 3 can be provided with a control unit 37. The control unit 37 can be designed separately or together with the microprocessor of the display device 36. The control unit 37 can be designed in a suitable manner to execute or support data glasses functions and functions of the display system 1. For this purpose, the assistance system 2 can be connected to the data glasses 3 in order to transmit to the data glasses 3 the object information relating to virtual display objects to be displayed in a contact-analog or non-contact-analog manner and the glasses pose determined externally by the glasses in relation to the vehicle coordinate system. For this purpose, the data glasses 3 can comprise a communication unit 39 which enables communication with the assistance system 2, in particular to detect a glasses pose ui determined externally by the glasses.

[0037] For example, the control unit 37 can read movement information using a glasses inertial sensor 38. The movement information can be used to correct the latency-related information of the glasses pose determined externally by sensor fusion and to provide a corrected glasses pose in the data glasses 3. This can then be used for the contact-analog display of information objects.

[0038] Figure 2 schematically illustrates, using a flowchart, a method for pose recognition of data glasses 3 in a vehicle as a mobile device. Pose recognition is performed remotely from the glasses, particularly in the assistance system 2. The model system, consisting of a glasses type recognition model and a plurality of pose recognition models, each assigned to a glasses type, is stored in the form of model parameters in a database of the assistance system 2.

[0039] The method initially provides for capturing a camera image using the interior camera 22 in step S1.

[0040] In step S2, this camera image is checked or searched for one or more smart glasses. Using an image recognition method or pattern recognition method in the form of the data-based glasses type recognition model, a type of at least one pair of smart glasses can be recognized in the camera image based on predefined shapes of smart glasses, as shown, for example, in Figures 3a to 3c. Using the implemented pattern recognition method, the type of the at least one pair of smart glasses 3 worn by one or more users in the vehicle can thus be classified in the camera image.

[0041] The glasses type recognition model can be designed as an artificial neural network, in particular as a convolutional neural network (CNN), in order to recognize the glasses type based on the shape of the glasses or the characteristics of the data glasses 3.

[0042] Additionally, using the glasses type recognition model, a corresponding position in the interior of the vehicle where the respective data glasses 3 are located can be detected, e.g., in the form of a bounding box surrounding the detected data glasses. This makes it possible to provide information about the at least one position of the at least one data glasses 3.

[0043] In a subsequent step S3, depending on the detected eyeglass type, a pose recognition model trained for precisely this eyeglass type is selected from a plurality of pose recognition models each assigned to a particular eyeglass type. The pose recognition model is also data-based and allows the pose of the data glasses 3 of the specific eyeglass type to be determined in the vehicle interior based on one or more camera images. The pose of the data glasses 3 is specified relative to a vehicle coordinate system.

[0044] In step S4, a pose of the data glasses 3 can be determined using the respectively selected pose recognition model based on the camera image in a manner known per se.

[0045] By specifying the bounding box for the specific data glasses of the corresponding glasses type, the selected pose recognition model can perform pose determination more reliably and quickly, since the search area in the camera image can be restricted to the bounding box.

[0046] The previously selected pose recognition model can now be used continuously to determine the glasses pose.

[0047] It can be provided that the eyeglass type is only determined when a pair of smart glasses is detected for the first time in the camera image. Alternatively, the eyeglass type can be detected regularly, so that a correspondingly different pose recognition model can be selected immediately in the event of a change. In particular, the eyeglass type can be determined if the pose recognition model has not been able to determine a pose for the smart glasses 3 for a predetermined period of time, such as 5 seconds.

[0048] List of reference symbols

[0049] Display system

[0050] Assistance system

[0051] Data glasses

[0052] Communication connection

[0053] Communication unit

[0054] Interior camera

[0055] Processor unit

[0056] Frame

[0057] Viewing windows

[0058] Glasses temples

[0059] Display area

[0060] Display device

[0061] Control unit

[0062] Glasses inertial sensors

[0063] Communication unit

Claims

Claims Method for determining a pose of at least one pair of data glasses (3) in an interior space of a mobile device, comprising the following steps: - Capturing (S1) a camera image of the interior of the mobile facility; - Evaluating (S2) a data-based eyeglass type recognition model based on the captured camera image to determine a respective eyeglass type for at least one pair of smart glasses depicted in the camera image, wherein the data-based eyeglass type recognition model is trained to assign a camera image to a respective eyeglass type of one or more smart glasses; - Selection (S3) of one of several data-based pose recognition models depending on the type of glasses detected, wherein the data-based pose recognition models are each trained on a type of glasses to determine a glasses pose of the corresponding data glasses depending on the camera image; - Determining (S4) the spectacle pose of the data glasses (3) of the corresponding spectacle type based on the selected pose recognition model. Method according to claim 1, wherein a bounding box of the respective detected data glasses (3) is further determined using the spectacle type recognition model, wherein the respective pose recognition model is applied to a section of the camera image that is determined by the bounding box. A method according to claim 1 or 2, wherein the pose of the at least one pair of smart glasses (3) is transmitted to the respective pair of smart glasses (3) so that a contact-analog display can be shown in the respective pair of smart glasses depending on their pose. A method according to any one of claims 1 to 3, wherein model parameters of the respective selected pose recognition model are retrieved from a database and implemented to provide the corresponding pose recognition model. A method according to any one of claims 1 to 4, wherein the model parameters of the pose recognition model are retrieved from a cloud database depending on a detected type of smart glasses. A device for determining a pose of at least one pair of smart glasses (3) in an interior space of a mobile device, comprising: - an interior camera (22) designed to capture a camera image of the interior of the mobile device; - a processor unit (23) configured to: o evaluate a data-based eyeglass type recognition model based on the captured camera image to determine a respective eyeglass type for at least one pair of smart glasses depicted in the camera image, wherein the data-based eyeglass type recognition model is trained to assign a camera image to a respective eyeglass type of one or more smart glasses; o select one of several data-based pose recognition models depending on the respective eyeglass type detected, wherein the data-based pose recognition models are each trained on a specific eyeglass type to determine an eyeglass pose of the corresponding smart glasses depending on the camera image; and to determine the spectacle pose of the data glasses (3) of the corresponding spectacle type based on the selected pose recognition model. Device according to claim 6, wherein a communication unit (21) is configured to transmit the spectacle pose to the data glasses (3).

Images