Method for operating smart glasses in a motor vehicle and system comprising a motor vehicle and smart glasses

Abstract

Method for operating smart glasses (13) in a motor vehicle (11), wherein a processor circuit (14), which is provided in the smart glasses (13) or coupled to the smart glasses (13) via a communication link (15), receives image data (24) from at least one camera (21) of the smart glasses (13), which represent an environment (23) of the smart glasses (13), characterized in that a digital model (27) of at least one predetermined orientation object (32) is stored in the processor circuit (14) and is checked by means of a predetermined matching routine (28) whether the model (27) corresponds to an image region (25) of the environment (23) represented in the image data (24), wherein the image data (24) are processed before the check by means of a predetermined processing routine (B), by which a level of detail of the representation of the environment (23) is reduced,and, upon detection of a match, a relative position (34) assigned to the image region (25) via a coordinate mapping (33) is determined, and an image content (18) of the data glasses (13) is controlled depending on the determined relative position (34) of the at least one orientation object (32).

Description

[0001] The invention relates to a method for operating smart glasses in a motor vehicle. Such smart glasses can be designed as so-called VR glasses (VR - Virtual Reality) or AR glasses (AR - Augmented Reality). The invention also includes a smart glasses device comprising such smart glasses and a processor circuit. Finally, the invention includes a system comprising such a smart glasses device and a motor vehicle.

[0002] A smart glasses device (also known as a head-mounted display or HMD) can be worn on the user's head, so that a display device of the smart glasses, such as a pixel-based screen or a projection unit, is positioned in front of the user's eyes to display image content within their field of vision. To give the user the impression of looking into a space, meaning that the displayed image content, such as virtual objects, is located in this virtual space and not fixed to the user's head, the display content shifts when the head moves. This ensures that the displayed virtual objects maintain a virtual relative position to the user, even when the user turns their head.For example, if the user turns their head to the left by a certain angle, an object that is stationary in the virtual space within the display content is moved to the right by the same angle, so that from the user's point of view it appears to be stationary and does not follow the head movement.

[0003] For this to work, the smart glasses must detect the user's head movement. US patent 2018 / 0081426 A1 discloses that accelerometers can be integrated into the smart glasses to detect head movement caused by acceleration forces. To differentiate between acceleration forces caused by the head movement itself and those caused by the movement of the vehicle, which can be measured even with the head still in the smart glasses, an additional mobile device, such as a tablet PC, is used to measure the vehicle's acceleration forces independently of the head. This requires an additional inertial sensor outside the smart glasses.

[0004] Controlling a display content shown through data glasses, depending on the course of a motor vehicle journey, is known, for example, from WO 2018 / 057980 A1.

[0005] When using inertial sensors or acceleration sensors to detect head movement, complex compensation measures are necessary in accelerated spaces, such as the interior of a motor vehicle.

[0006] The invention is based on the objective of operating data glasses in a motor vehicle, in particular during a journey of the motor vehicle, when acceleration forces may occur due to vehicle acceleration and / or cornering.

[0007] The problem is solved by the subject matter of the independent claims. Advantageous embodiments of the invention are described by the dependent claims, the following description, and the figures.

[0008] The invention provides a method for operating smart glasses in a motor vehicle. The operation can include detecting the head movement of a user wearing the smart glasses. In a moving and / or accelerating frame of reference, such as a vehicle interior, it can happen that not only the user's head, but the entire user is moved and / or accelerated, even though the head itself remains still. This can lead to a distorted or undesirable change in the displayed content because it is not caused, as intended, solely by the movement of the head (relative to or in proportion to the user's torso).

[0009] In this process, a processor circuit receives image data from a camera in the smart glasses. The field of view of at least one camera is directed towards the surroundings of the smart glasses, i.e., away from them, so that at least one object in the vicinity of the smart glasses is detected, filmed, or captured. The image data thus represents the surroundings of the smart glasses and shows the at least one object. The receiving processor circuit, which receives the image data, can be located within the smart glasses themselves, or it can be connected to them via a communication link, i.e., located outside the smart glasses. A suitable communication link can be implemented, for example, using a WLAN radio module (WLAN - Wireless Local Area Network). The communication link can also be wired, for example, via a USB cable connection (USB - Universal Serial Bus).

[0010] The invention provides that the movement of the smart glasses in the vehicle is not detected by an accelerometer alone or independently of an accelerometer. Instead, a digital model of at least one predetermined orientation object is stored in the processor circuit. Such an orientation object could be, for example, the interior wall of the vehicle interior and / or a seat back and / or a dashboard and / or a center console of the vehicle. Using a predetermined matching or comparison routine, the processor circuit checks whether this model corresponds to an image region of the environment as depicted in the image data. The image data represents an image or video of the environment, and the matching routine checks whether the at least one orientation object is contained or depicted in this image or video.This part of the image is the described image region. When such a match between the model and the image region is detected, the processor circuit determines a relative position of the orientation object with respect to the glasses. This relative position results from the image region for which the match with the model was detected.

[0011] For this purpose, the image region is assigned to the relative position via a coordinate mapping. In particular, it is possible to determine, based on the so-called extrinsic and / or intrinsic camera data and / or the camera's mounting position, which spatial region or direction corresponds to a specific image region, for example, a pixel or a group of pixels. If an orientation object is detected in a particular image region of the camera image, the coordinate mapping can be used to determine which coordinates in the environment or which direction vector relative to the smart glasses the image region corresponds to. If the relative position and / or the direction vector of at least one orientation object relative to the smart glasses is known, the processor circuitry controls the image content of the smart glasses—that is, the image content viewed by the user—depending on the determined relative position and / or direction vector of the at least one orientation object.In particular, the relative position and / or direction vector can be used to detect movement of the head on which the smart glasses are attached or worn, in relation to the vehicle interior. For this purpose, a component moving with the vehicle is used as the respective orientation object. If the head is held still, so that the smart glasses are also motionless in relation to the vehicle interior, this can be detected based on the relative position with respect to at least one orientation object. Therefore, even if the vehicle accelerates and / or curves, this does not lead to an erroneous change in the image content, as would occur if the user were to make a corresponding head movement in the vehicle interior.The method, as described, is particularly reliable for controlling the image content if at least one orientation object is part of the vehicle interior.

[0012] The invention offers the advantage that, even with regard to the acceleration forces acting in a motor vehicle, it is possible to distinguish, independently of an acceleration sensor, whether the user is moving their head inside the vehicle or not.

[0013] The invention also includes embodiments that offer additional advantages.

[0014] In one embodiment, the model is a digital data model that is independent of the actual vehicle, but rather describes a prototype of the vehicle. In other words, the model consists of pre-configured model data stored in the processor circuitry. Such model data can, for example, be extracted from a CAD model (CAD - Computer-Aided Design). Using model data from a prototype offers the advantage that no configuration phase is required to generate the model of the specific vehicle. Furthermore, the model is more accurate than one that could be created by a user of the smart glasses during a configuration phase.

[0015] Nevertheless, a specific or concrete vehicle may present a different appearance in its interior compared to the prototype, for example, because a user has placed an object on the center console and / or attached a decorative item to a seat back or rearview mirror, such as hanging a stuffed animal there. To still be able to recognize a match with the model, one embodiment provides that the image data is processed before verification using a predetermined processing routine, which reduces the level of detail in the image data. Such a processing routine can be achieved, for example, using a so-called Gaussian filter or a blurring filter. This allows objects of a predetermined size in the image data to be rendered unrecognizable to such an extent that they no longer have a significant influence on the matching routine.This can be adjusted, for example, by changing the size of the Gaussian filter. The processing routine can also or alternatively include a color and / or brightness adjustment in the image data to, for example, at least partially compensate for the influence of current lighting conditions. This means that placing or arranging an object in the vehicle interior will no longer affect or disrupt the matching process.

[0016] To fit or recognize a model within image data, one embodiment involves the matching routine performing this check based on a SLAM algorithm (Simultaneous Localization and Mapping). Additionally or alternatively, a computer vision algorithm (CV algorithm) can be used. Other examples of matching algorithms include an artificial neural network and correlation.

[0017] In one embodiment, the model describes the vehicle interior, and in particular, a section of at least one window pane is marked in the model as irrelevant for verification. Verification here refers to checking whether the model corresponds to a specific image area. The reason for marking the window area is that the exterior of the vehicle is visible through the window pane from within the vehicle interior. In other words, the exterior of the vehicle, and thus its surroundings, can be recognized in such a window area of ​​the camera image. Marking this window area as irrelevant has the advantage that the calculation of the relative position with respect to the vehicle interior can be performed robustly, since no image information from the exterior can interfere with the verification by the matching routine.

[0018] In one embodiment, a motion signal is determined or generated based on the temporal change in the determined relative position, which describes a movement of the smart glasses in relation to the motor vehicle. The relative position with respect to the at least one orientation object is thus repeatedly measured, and the movement of the smart glasses can be inferred from a change in the relative position. This allows not only the head position with respect to the at least one orientation object, but also the head movement itself to be determined or detected.

[0019] This motion detection can be understood as the temporal derivative of the individual relative positions detected sequentially. Additionally, it can be provided that an acceleration signal from the smart glasses' accelerometer is acquired, from which a movement can also be determined by temporal integration. However, it must be noted that this acceleration signal can describe both the movement of the head relative to the vehicle interior and the movement of the vehicle relative to the external environment. To distinguish between these, according to the embodiment, a vehicle acceleration signal is received from an acceleration sensor of the vehicle itself—that is, an acceleration sensor that operates independently of the smart glasses.A difference signal is then calculated from the vehicle acceleration signal and the glasses acceleration signal. From this, a plausibility signal is derived, describing an estimated movement of the smart glasses within the vehicle, as determined from the difference signal of the two acceleration signals. This plausibility signal is compared and / or combined with the motion signal, which is generated as described from the temporal change of the determined relative position. A comparison allows for the detection of whether a movement of the head within the vehicle interior actually occurred. A more robust motion signal can be generated through a combination of these signals, particularly by basing the motion signal on both a time derivative and a time integration.

[0020] To carry out the method according to the invention, at least the described combination of smart glasses and processor circuitry is necessary. This combination is provided by a smart glasses device of the invention. The processor circuitry is configured to execute an embodiment of the method according to the invention. The processor circuitry can be integrated into the smart glasses as described or coupled to the smart glasses via the described communication link. A processor circuitry can, for example, be based on at least one microprocessor. The at least one microprocessor can be connected or coupled to a data memory in which program instructions can be stored that, when executed by the processor circuitry, cause it to carry out the embodiment of the method according to the invention.

[0021] The combination of the described data glasses device with a motor vehicle results in a system that is also part of the invention. The system can include the motor vehicle, in particular a car, especially a passenger car or truck.

[0022] The invention also includes realizations or implementations in which the features of several different embodiments described are combined.

[0023] The described motor vehicle is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus.

[0024] The following are exemplary embodiments of the invention described. This is illustrated by: Fig. 1 a schematic representation of an embodiment of the system according to the invention; and Fig. 2 a flowchart for an embodiment of the method according to the invention.

[0025] The exemplary embodiments described below are preferred embodiments of the invention. In these exemplary embodiments, the described components each represent individual features of the invention, which can be considered independently of one another and each further develops the invention independently. Therefore, the disclosure is intended to include combinations of features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0026] In the figures, identical reference symbols denote functionally equivalent elements.

[0027] Fig. 1 shows a system 10 , a motor vehicle 11 and a data glasses device 12can include the data glasses device. 12 can in the motor vehicle 11 be provided. The data glasses device 12 can a pair of smart glasses 13 and a processor circuit 14 include the processor circuit. 14 can be used in the data glasses 13 be integrated or, as exemplified in Fig. 1 shown, with the data glasses 13 via a communication link 15 be coupled. To provide or operate the communication link. 15 can the data glasses 13 and the processor circuit 14 each one communication circuit 16 exhibiting features that may be based on, for example, Wi-Fi technology. The data glasses 13 and the processor circuit 14 can be achieved through the communication circuit 16It can also be connected via a communication cable (not shown), for example a USB cable (USB - Universal Serial Bus).

[0028] The data glasses 13 can during a journey 17 of the motor vehicle 11 It can be operated in the data glasses. 13 one the data glasses 13 User wearing an image on their head (not shown) 18 a virtual environment 19 displayed, in which the user can look around by turning and / or tilting their head and thus also the data glasses 13 in a vehicle interior 20 of the motor vehicle 11 relative to the motor vehicle 11 moves. Each of which is located in the virtual environment. 19 resulting change in perspective into the virtual environment 19 into, that is, the change of the image content 18 , this can be done by observing the movement of the data glasses 13relative to the motor vehicle 11 are equivalent to.

[0029] The processor circuit 14 This can be done using the position and / or movement of the smart glasses 13 in relation to the motor vehicle 11 capture and the image content 18 adjust accordingly. The smart glasses can do this. 13 at least one camera 21 exhibiting, whose respective detection range 22 into an environment 23 the data glasses 13 and thus into the vehicle interior 20 is directed inwards. In other words, the detection range indicates 22 from the data glasses 13 way towards the vehicle interior 20 Image data 24 which at least has a camera 21 can be added to the processor circuit 14 for example via the communication link 15 be transferred. Using an editing routine B, the level of detail in the representation of the environment can be adjusted.23 in the image data 24 be reduced.

[0030] In the processor circuit 14 can be determined from the image data 24 an image region 25 , which are in the image data 24 The diagram shows how things can be evaluated. In the processor circuit 14 can model data 26 a digital model 27 the vehicle interior 20 be provided. Unlike the actual vehicle interior. 20 can be done in the model 27 However, user accessories U, such as a mobile phone and / or a drinking cup and / or decorations, in the model 27 be disregarded. The model 27 can, for example, be based on CAD data of a vehicle design. 11 based on image data. 24 and the model data 26 can be done through a matching routine 28 the processor circuit 14to determine whether the image data 24 an image region 25 included is the one that comes with the model 27 agrees. In Fig. 1. Hatching symbolizes a disk area, indicating that the matching routine is running. 28 This area should be ignored.

[0031] In Fig. 1 is shown as an example of how both the image data 24 as well as the model data 26 an A-pillar 29 and a steering wheel 30 and a rearview mirror 31 These can be described, but these are only examples. Overall, these are orientation objects. 32 , which are processed by the matching routine 28 in the vehicle interior 20 They can be recognized again. Based on the relative position of these recognized orientation objects. 32 , that is, the agreement of the model 27 with the corresponding image region 25, can be done using a coordinate mapping 33 a relative position 34 the data glasses 13 inside the vehicle 20 in relation to the motor vehicle 11 This can be determined, for example, as a coordinate point. 35 and / or as an alignment vector or orientation vector 36 be described.

[0032] By repeatedly capturing the relative position 34 and calculating a change over time 37 Can also a movement of the data glasses 13 inside the vehicle 20 can be tracked. Based on the relative position 34 and / or the movement of the data glasses 13 VR image data can then be used 38 the virtual environment 19 calculated and / or adjusted. The resulting VR image data 38 can be displayed in the data glasses in the usual manner. 13 as image content 18 the user of the data glasses 13on a display unit 39 displayed, for example on a stereoscopic screen and / or a projector (for AR glasses).

[0033] To determine the relative position 34 To make it more plausible, an accelerometer can also be used. 40 of the motor vehicle 11 a vehicle acceleration signal 41 be received, which is related to the change over time. 37 the relative position 34 can be aligned, compared, or combined.

[0034] Fig. Figure 2 illustrates a method that is implemented using the processor circuit. 14 to generate the VR image data 38 can be carried out.

[0035] In one step S10 Sensor measurements can be performed, which enables the generation of image data. 24 using at least one camera 21and optionally the generation of the vehicle acceleration signal 41 using the accelerometer 40 can include in one step S11 can the model data 26 for the model 27 of at least one orientation object 32 be provided.

[0036] In one step S12 can be done using the matching routine 28 to determine which image region 25 in the image data 24 with the model 27 which corresponds to at least one orientation object.

[0037] In one step S13 can the relative position 34 be estimated, with both the output of the matching routine being considered. 28 as well as the vehicle acceleration signal 41 from one step S12' can be used.

[0038] The resulting change over time 37the user's virtual perspective in the virtual environment 19 can be entered in one step S14 will be recalculated. In one step S15 can the resulting VR image data 38 calculated using the new virtual perspective. In one step S16 Can the VR image data then be used? 38 via the display unit 39 be reproduced so that the user can see it in the data glasses 13 the updated image content 18 is visible. The vehicle acceleration signal 41 can be used to perform VR image data without recalculating the VR image data 38 also between two calculation steps of the VR image data 38 an adjustment of the image content 18 to generate a difference signal between a glasses acceleration signal from an accelerometer of the data glasses 13 (not shown) and the vehicle acceleration signal 41The calculation is performed and the difference is used to check whether the user is wearing the data glasses. 13 relative to the motor vehicle 11 has moved. It can then be an adjustment of the perspective to the virtual environment. 19 , that is, an adjustment of the image content 18 , by pixel shifting the calculated VR image data 38 without recalculating the results. This also allows for adjustments based on movement of the smart glasses. 13 in the motor vehicle 11 reactions will occur due to the computational effort or computing power of the processor circuit. 14 not by recalculating the VR image data 38 can be taken into account.

[0039] This enables the use of an augmented reality (AR) head-mounted display (HMD) and an application in an accelerated reference system, particularly in a moving vehicle.

[0040] The AR-HMD is based on a technology that allows the HMD to spatially orient itself within its environment. For example, if a hologram of a virtual object is positioned at a specific location in the environment, and the user moves their head, the HMD detects this movement (6 degrees of freedom) and uses it to compensate for the head movement when displaying the hologram. This can happen in real time, so that it appears to the user as if the hologram is not moving from its position. No inertial sensors are required in the HMD to detect this movement. The HMD is equipped with at least one camera (optical or infrared).

[0041] To use an AR-based head-mounted display (HMD) in a moving vehicle, the HMD must be able to spatially orient itself within its immediate surroundings. An advantage of using a moving car is that the environment, i.e., the vehicle interior, remains constant. The crucial advantage for the car manufacturer is that they possess a precise 3D model of the vehicle interior. This allows them to perform so-called "model tracking" (e.g., using the computer library https: / / visionlib.com), which calculates the relative position of the car interior to the head / HMD position at any given time. The tracked vehicle interior is then defined as the new reference coordinate system for the HMD. This allows the HMD to spatially orient itself within the vehicle interior relative to this model and thus position the holograms precisely (relative to the vehicle interior).Because this technology is based on computer vision (CV) methods, it is also fast (real-time capable). This technology even makes it independent of the HMD's inertial sensors, requiring only an optical camera.

[0042] Additionally, an external IMU can be used. To use an AR-based HMD in a moving vehicle, the HMD must be able to spatially orient itself within its immediate surroundings. The crucial advantage for the automotive manufacturer is that they can provide the exact movement of the vehicle through its inertial sensors (IMUs in the vehicle). This allows the head movement of the HMD relative to the vehicle interior to be determined by calculating the relative motion: IMU_relative = IMU_HMD - IMU_vehicle. In the AR-based HMD, the values ​​of IMU_HMD are then replaced by the relative values, i.e., IMU_relative. This gives the HMD consistent acceleration values ​​relative to the vehicle interior and enables it to spatially orient itself within the vehicle, despite external acceleration.

[0043] Ideally, one can combine solution approaches 1 (camera and model) and 2 (relative IMU) to create maximum stability regarding the spatial positioning of a hologram.

[0044] The proposal is particularly distinguished by the fact that it: - independent of the inertial sensors of the HMD and the car (no IMUs - Inertial Measurement Units are required), - a precise 3D CAD model tracking is used to orient the HMD in the moving car, - the 3D CAD model can be chosen with high precision (i.e., it can come from the car manufacturer).

[0045] Overall, the examples show how AR-based HMD can be deployed in a moving vehicle. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 2018 / 0081426 A1

[0003] WO 2018 / 057980 A1

[0004]

Claims

[1] Method for operating smart glasses (13) in a motor vehicle (11), wherein a processor circuit (14) provided in the smart glasses (13) or coupled to the smart glasses (13) via a communication link (15) receives image data (24) from at least one camera (21) of the smart glasses (13) which represents an environment (23) of the smart glasses (13), characterized by, that in the processor circuit (14) a digital model (27) of at least one predetermined orientation object (32) is stored and a predetermined matching routine (28) is used to check whether the model (27) matches an image region (25) of the environment (23) represented in the image data (24), and if a match is detected, a relative position (34) assigned to the image region (25) via a coordinate mapping (33) is determined and an image content (18) of the data glasses (13) is controlled depending on the determined relative position (34) of the at least one orientation object (32). [2] Method according to claim 1, wherein the model (27) independently describes a prototype of the motor vehicle (11) and model data (26) of the model (27) are stored in the processor circuit (14). [3] Method according to one of the preceding claims, wherein the image data (24) are processed before verification by means of a predetermined processing routine (B) by which a level of detail of the representation of the environment (23) is reduced. [4] Method according to any of the preceding claims, wherein the matching routine (28) performs the verification based on a SLAM algorithm. [5] Method according to one of the preceding claims, wherein the model (27) describes a vehicle interior (20) of the motor vehicle (11) and a pane area of ​​at least one window pane, through which an exterior area of ​​the motor vehicle is visible from the vehicle interior, is marked in the model as an area irrelevant for verification. [6] Method according to one of the preceding claims, wherein a motion signal is determined based on a temporal change (37) of the determined relative position (34), wherein the motion signal describes a movement of the data glasses (13) in relation to the motor vehicle (11). [7] Method according to claim 6, wherein a glasses acceleration signal is received from an acceleration sensor of the data glasses (13) and a vehicle acceleration signal (41) is received from an acceleration sensor (40) of the motor vehicle (11), and a difference signal is calculated from the vehicle acceleration signal (41) and the glasses acceleration signal, and a plausibility signal describing an estimated movement of the data glasses (13) relative to the motor vehicle (11) is calculated and compared and / or combined with the movement signal. [8] Data glasses device (12) comprising a data glasses device (13) and a processor circuit (14), wherein the processor circuit (14) is configured to perform a method according to one of the preceding claims. [9] System (10) comprising a motor vehicle (11) and a data glasses device (12) according to claim 8.

Images