Method for operating a display unit for extended reality and display unit

Abstract

The invention relates to a method for operating a display unit for extended reality, comprising: introducing the display unit into an interior space of a vehicle; determining a motion of the display unit in world coordinates and generating corresponding first coordinate data; continuously capturing images with a camera of the display unit, which comprise at least a portion of the interior space of the vehicle, and generating image data corresponding to the images; determining a motion of the display unit in a coordinate system of the interior space of the vehicle based on the image data and generating corresponding second coordinate data; determining a motion of the vehicle in the world coordinates from a difference between the first coordinate data and the second coordinate data and generating corresponding third coordinate data; and determining a position and an orientation of the display unit in the interior space of the vehicle taking into account the third coordinate data. The invention also relates to a display unit for extended reality.

Description

Technical Field

[0001] This invention relates to a method for operating a display unit for extended reality. The invention also relates to a display unit for extended reality. Background Technology

[0002] Extended Reality (XR) is a collective term for technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), which integrate digital content into the physical world or create immersive digital environments. Display units used in Extended Reality include, for example, head-mounted displays (HMDs) (such as XR headsets or smart glasses) and mobile devices (such as smartphones or tablets). For digital content to be seamlessly integrated into the physical world, the display unit must accurately and in real-time determine its position and orientation. Known display units calculate their position and orientation using data from various sensors, such as motion data from an inertial measurement unit (IMU) installed within the display unit and images from the display unit's cameras. To calculate the position and orientation of the display unit, motion is determined in the local coordinate system of one of the sensors and then transformed, i.e., merged, into a common coordinate system. Problems can arise if this common coordinate system is not fixed in position, such as a coordinate system that moves relative to the world coordinate system within the interior space of a vehicle. In such a highly dynamic environment, phenomena such as virtual objects suddenly "moving away" or becoming stuck can occur. Summary of the Invention

[0003] The objective of this invention is to provide a method for operating a display unit for extended reality, and a display unit for extended reality, which, in particular, reliably and accurately determines the position and orientation of the display unit in a highly dynamic environment.

[0004] This task is solved by the method having the features of claim 1 and by the subject matter of the independent claims. Advantageous further improvements are given in the dependent claims.

[0005] The proposed method for operating a display unit for extended reality includes at least the following steps: introducing the display unit into the interior space of a vehicle; determining the motion of the display unit in world coordinates; generating first coordinate data corresponding to the motion of the display unit in world coordinates; continuously acquiring images using a camera on the display unit, these images including at least a portion of the interior space of the vehicle; generating image data corresponding to the images; determining the motion of the display unit in the coordinate system of the vehicle's interior space based on the image data; generating second coordinate data corresponding to the motion of the display unit in the coordinate system of the vehicle's interior space; determining the motion of the vehicle in world coordinates from the difference between the first and second coordinate data; generating third coordinate data corresponding to the motion of the vehicle in world coordinates; and determining the position and orientation of the display unit in the vehicle's interior space, taking into account the third coordinate data.

[0006] The proposed method enables the use of display units within the interior space of a moving vehicle. To determine the position and orientation of the display unit within the moving vehicle's interior space, the motion of the display unit in world coordinates is first determined, i.e., its motion relative to a fixed reference frame in which the vehicle moves. Then, image data is used to determine the motion of the display unit in the coordinate system of the vehicle's interior space, i.e., its motion relative to a reference frame that does not move relative to the vehicle. The difference between these two motions determines the motion of the vehicle relative to the fixed reference frame. The vehicle's motion is then used to determine the position and orientation of the display unit within the vehicle's interior space. For example, the vehicle's motion can be subtracted from the display unit's motion relative to the fixed reference frame to determine the motion of the display unit within the vehicle's interior space.

[0007] Determining the position and orientation of the display unit within the vehicle's interior space using image data is independent of the vehicle's motion. However, image data from cameras with the required resolution can only be acquired and processed at rates of 10Hz to 30Hz, with a maximum of 50Hz, because determining motion from images is computationally intensive. In contrast, inertial measurement units (IMUs) have sampling rates from 100Hz to 500Hz, and therefore can determine the display unit's motion significantly more accurately and with significantly less computational power. However, IMUs can only determine motion relative to a fixed reference frame. Instead, in the proposed method, the motion of the display unit within the vehicle's interior space, determined using image data, is used to determine the vehicle's motion. Since the typical acceleration of the vehicle is less rapid and less violent than that of the display unit, the lower processing rate of the image data does not adversely affect this objective. The motion of the display unit within the vehicle's interior space can then be determined with the same accuracy as the motion of the display unit in a fixed reference frame, for example, with the high sampling rate of the IMU. Therefore, the position and orientation of the display unit can be reliably and accurately determined even in highly dynamic environments.

[0008] In this document, means of transport are understood in particular as motor vehicles, such as passenger cars or trucks. However, means of transport can also refer to other road vehicles, aircraft, or water vehicles.

[0009] In one implementation, the motion of a vehicle in world coordinates is determined from the difference between first and second coordinate data using filters and / or a trained machine learning model. For example, the difference between the first and second coordinate data can be processed using interpolation, smoothing, bandpass filters, or Kalman filters to obtain the vehicle's motion in world coordinates. Alternatively or additionally, a trained machine learning model can be used for similar functionality. With the aforementioned methods, measurement noise can be reduced, jumps compensated for, outliers filtered out, and missing values ​​appropriately supplemented. This allows for the particularly accurate determination of the vehicle's motion in world coordinates.

[0010] In one implementation, the motion of the display unit in world coordinates is determined using an inertial measurement unit (IMU) of the display unit. The IMU has a sampling rate ranging from 100Hz to 500Hz. This enables particularly accurate determination of the display unit's motion in the world coordinate system. The position and orientation of the display unit within the vehicle's interior space are ultimately determined by the display unit's motion in the world coordinate system. Therefore, using an IMU also allows for particularly accurate determination of the display unit's position and orientation within the vehicle's interior space.

[0011] In one implementation, images continuously acquired by the display unit's camera cover an area outside the vehicle. The motion of the display unit in world coordinates can be determined while taking the image data into account. In this implementation, the image data is used to determine the motion of the display unit in a fixed reference frame. This can be achieved, in particular, in conjunction with an inertial measurement unit (IMU) to further improve the accuracy of determining the motion of the display unit in world coordinates. Alternatively, the display unit's camera can be used alone to determine not only the motion of the display unit relative to the interior space of the vehicle but also the motion of the display unit relative to a fixed reference frame. This implementation, for example, does not require an IMU.

[0012] In one implementation, first sub-image data is generated from image data using image segmentation. This first sub-image data corresponds to image regions of the image, each of which includes a portion of the region outside the vehicle. The motion of the display unit in world coordinates is determined while considering the first sub-image data. In this implementation, it is first determined which image regions are respectively shown as portions of the region outside the vehicle. These image regions are then used to determine the motion of the display unit relative to a fixed reference frame, for example, using optical flow. Image segmentation prevents, for example, the erroneous use of points within the vehicle's interior space to determine the motion of the display unit in world coordinates.

[0013] In one implementation, the position of a first image point in successive images is determined based on image data to determine the motion of the display unit in world coordinates. The first image point corresponds to a point in a region outside the vehicle. For example, the motion of the first image point in successive images is tracked to infer the camera's motion and therefore the motion of the display unit. This tracking is robust and therefore can be used to determine the motion of the display unit relative to a fixed reference frame with very high accuracy. The first image point can be determined based on first image data, for example, by selecting only image points in image regions that include a portion of the region outside the vehicle.

[0014] In one implementation, the first image point is determined by identifying, for a plurality of image points in successive images, which of these image points correspond to points in the region outside the vehicle. Cluster analysis is, in particular, a less computationally intensive alternative to image segmentation, but can also be used as an adjunct. In this implementation, the locations of these plurality of image points are first determined, for example, in two-dimensional or three-dimensional coordinates relative to the camera. These locations and / or parameters derived therefrom, such as velocity, form the basis of the cluster analysis. The image points corresponding to points in the region outside the vehicle are then separated from those corresponding to points in the interior space of the vehicle by the cluster analysis. The image point corresponding to the point in the region outside the vehicle is used as the first image point.

[0015] In one embodiment, second sub-image data is generated from image data using image segmentation. This second sub-image data corresponds to image regions of the image, each of which includes at least a portion of the vehicle's interior space. The motion of the display unit in the coordinate system of the vehicle's interior space is determined taking into account the second sub-image data. In this embodiment, images continuously acquired by the display unit's camera preferably include regions outside the vehicle. First, it is determined which image regions respectively represent a portion of the vehicle's interior space. These image regions are then used to determine the motion of the display unit relative to the vehicle's interior space, for example, using optical flow. Image segmentation prevents, for example, the erroneous use of points outside the vehicle's interior space to determine the motion of the display unit in the vehicle's interior space coordinate system.

[0016] In one implementation, the position of a second image point in successive images is determined based on image data to determine the motion of the display unit in the coordinate system of the vehicle's interior space, the second image point corresponding to a point within the vehicle's interior space. For example, the motion of the second image point in successive images is tracked to infer the camera's motion and thus the motion of the display unit. This tracking is robust and therefore can be used to determine the motion of the display unit relative to the vehicle's interior space with great accuracy. The second image point can be determined based on second image data, for example, by selecting only image points in image regions that include a portion of the vehicle's interior space.

[0017] In one implementation, the second image point is determined by identifying, for a plurality of image points in successive images, which of these image points correspond to points within the vehicle's interior space, using cluster analysis. Also in this implementation, cluster analysis constitutes a computationally less intensive alternative to image segmentation, but can also be used as an adjunct. In this implementation, the locations of these plurality of image points are first determined, for example, in two-dimensional or three-dimensional coordinates relative to the camera. These locations and / or parameters derived therefrom, such as velocity, form the basis of the cluster analysis. Then, the image points corresponding to points in the region outside the vehicle are separated from those corresponding to points within the vehicle's interior space by cluster analysis. The image point corresponding to the point within the vehicle's interior space is used as the second image point.

[0018] The present invention also relates to a display unit for extended reality. The display unit includes a camera configured to continuously acquire images and generate image data corresponding to the images. The display unit further includes a determining unit configured to determine the motion of the display unit in world coordinates and generate corresponding first coordinate data. The determining unit is further configured to: determine the motion of the display unit in the coordinate system of the vehicle's interior space based on the image data corresponding to an image of at least a portion of the vehicle's interior space, and generate corresponding second coordinate data. The determining unit is further configured to: determine the motion of the vehicle in world coordinates from the difference between the first and second coordinate data, generate corresponding third coordinate data, and determine the position and orientation of the display unit in the vehicle's interior space, taking into account the third coordinate data.

[0019] The display unit has the same advantages as the claimed method. In particular, the display unit can be further improved using the features described in conjunction with the method in this document. Furthermore, the method can be further improved using the features described in conjunction with the display unit in this document.

[0020] In one embodiment, the display unit includes an inertial measurement unit configured to generate acceleration data corresponding to the motion of the display unit. A determination unit can be configured to determine the motion of the display unit in world coordinates, taking the acceleration data into account. The inertial measurement unit has a sampling rate ranging from 100Hz to 500Hz. This enables particularly accurate determination of the motion of the display unit in the world coordinate system.

[0021] The display unit may include multiple cameras, particularly those oriented in different directions. In this implementation, the image data includes images captured sequentially by the different cameras. Attached Figure Description

[0022] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Wherein:

[0023] Figure 1 A schematic diagram of a display unit for extended reality according to one embodiment is shown;

[0024] Figure 2 A schematic diagram of a vehicle illustrating one embodiment of a method for operating a display unit for extended reality;

[0025] Figure 3 The diagram is shown to further clarify the basis. Figure 2 The curve of the method;

[0026] Figure 4 A schematic diagram illustrating a vehicle used to explain another implementation of a method for operating a display unit for extended reality; and

[0027] Figure 5 The diagram is shown to further clarify the basis. Figure 4 The curve of the method. Detailed Implementation

[0028] Figure 1 A schematic diagram of a display unit 100 for extended reality according to one embodiment is shown. The display unit 100 is, for example, a head-mounted display or a mobile terminal device such as a smartphone or tablet computer.

[0029] Display unit 100 includes camera 102 configured to continuously acquire images and generate image data corresponding to those images. When display unit 100 is configured as a head-mounted display, camera 102 is, for example, oriented such that when a user uses display unit 100 as intended, camera 102 acquires images of the area in front of the user. Camera 102 can acquire images, for example, at a rate of 10Hz to 30Hz, or up to 50Hz. Thus, all images acquired by camera 102 can be processed in real time using portable hardware.

[0030] The display unit 100 also includes a determination unit 104, which, by way of example, includes an inertial measurement unit 106. The determination unit 104 is configured to determine the motion of the display unit 100 in world coordinates, i.e., the motion relative to a fixed reference frame. For this purpose, the determination unit 104 uses, for example, acceleration data generated by the inertial measurement unit 106 that corresponds to the motion of the display unit 100 relative to the fixed reference frame. Alternatively or additionally, the determination unit 104 may use image data generated by the camera 102 to determine the motion of the display unit 100 in world coordinates. The determination unit 104 generates first coordinate data corresponding to the motion of the display unit 100 in world coordinates.

[0031] Furthermore, the determining unit 104 is configured to: based on image data of at least a portion of the interior space 206 of the vehicle including the vehicle 200 (see... Figure 2 The determination unit 104 determines the motion of the display unit 100 relative to the vehicle's interior space 206 in the coordinate system. For example, the determination unit 104 uses methods such as optical flow and / or tracking to determine the motion of the camera 102 from image data and thus the motion of the display unit 100 relative to the vehicle's interior space 206. The determination unit 104 generates second coordinate data corresponding to the motion of the display unit 100 in the coordinate system of the vehicle's interior space 206.

[0032] The determining unit 104 is also configured to: determine the motion of the vehicle 200 in world coordinates based on the difference between the first coordinate data and the second coordinate data, generate corresponding third coordinate data, and determine the position and orientation of the display unit 100 within the vehicle's interior space 206, taking the third coordinate data into account. The determining unit 104 determines the motion of the vehicle 200 relative to the fixed reference frame based on the difference between the motion of the display unit 100 relative to the fixed reference frame and the motion of the display unit 100 relative to the vehicle's interior space 206. Then, the determining unit 104 can, for example, subtract the motion of the vehicle 200 from the motion of the display unit 100 relative to the fixed reference frame to determine the position and orientation of the display unit 100 within the vehicle's interior space 206 with high accuracy.

[0033] The advantage of this approach is that it enables the precise determination of the position and orientation of the display unit 100 within the vehicle's interior space 206, without requiring image data to be processed at a high rate, such as 100 Hz or higher. The image data processing rate only needs to be high enough to determine the motion of the vehicle 200. Since the motion of the vehicle 200 is less dynamic and less violent than the motion of the display unit 100, a lower processing rate, such as 10 Hz to 30 Hz, is sufficient. The rate at which the position and orientation of the display unit 100 within the vehicle's interior space 206 is determined depends on the rate at which the motion of the display unit 100 in world coordinates is determined. Therefore, by using the inertial measurement unit 106, a rate of 100 Hz or higher can be achieved, enabling the display unit 100 to seamlessly integrate digital content into the physical world.

[0034] Display unit 100 may include additional components capable of enabling extended reality displays. For example, display unit 100 may include one or more display elements 108, such as screens, transparent or semi-transparent displays, or projection systems, configured to insert virtual content into the user's field of view and overlay it onto the user's physical environment. Display unit 100 may also include one or more control units configured to manipulate the display elements to insert virtual content. These control units may include interfaces that allow them to communicate with control and processing units remote from display unit 100, such as with the control and processing unit of vehicle 200.

[0035] The display unit 100 can be used to implement a method for operating the display unit 100 for extended reality. The implementation of this method will be aided by the following... Figures 2 to 5 Detailed description.

[0036] Figure 2 A schematic diagram of a vehicle 200 is shown, illustrating one embodiment of a method for operating a display unit 100 for extended reality. Furthermore, Figure 2 The first coordinate axis 202 is shown, which belongs to the world coordinates of a fixed reference frame. The vehicle 200 moves relative to the world coordinates. Figure 2 A second coordinate axis 204 is also shown, which belongs to the coordinate system of the vehicle's interior space 206. The coordinate system of the vehicle's interior space 206 is fixed relative to the vehicle's interior space 206 and moves with the vehicle 200.

[0037] This method is purely exemplary and relies on... Figure 1 The display unit 100 is described herein. The display unit 100 is incorporated into the interior space 206 of the vehicle 200. For example, a user incorporates the display unit 100 into the interior space 206 of the vehicle. Figure 2 In this example, the display unit 100 is used by a passenger in the rear seat 208 of the vehicle 200.

[0038] In this method, images are continuously acquired using the camera 102 of the display unit 100, and these images include at least a portion of the interior space 206 of the vehicle. According to... Figure 1 In this embodiment, the image captured by camera 102 also includes a region 210 outside the vehicle 200. Camera 102 generates image data corresponding to the image and provides it for further processing.

[0039] Furthermore, the method determines the motion of the display unit 100 in world coordinates. Based on... Figure 1 In this implementation, the motion is determined by means of the camera 102 of the display unit 100. For example, image data generated by the camera 102 is processed by the determination unit 104 to determine first image points in successive images, the first image points corresponding to points 212 in the region 210 outside the vehicle 200. For example, the determination unit 104 may perform image segmentation to determine image regions in the images acquired by the camera 102, each corresponding to a portion of the region 210 outside the vehicle 200. The determination unit 104 may then use image points in these image regions as first image points. The processing of these image regions may be implemented in the form of first sub-image data.

[0040] The point in the area 210 outside the vehicle 200 is preferably point 212 on a fixed element. Figure 2 In the image, point 212 is, by way of example, a point on tree 214. Then, based on the first image point and using known methods, such as tracking or optical flow, the motion of display unit 100 in world coordinates can be determined by determination unit 104. Furthermore, determination unit 104 generates first coordinate data that corresponds to the motion of display unit 100 in world coordinates.

[0041] In this method, the motion of the display unit 100 in the coordinate system of the vehicle's interior space 206 is also determined based on image data. For example, image data generated by the camera 102 is processed by the determination unit 104 to determine second image points in successive images, the second image points corresponding to point 216 in the vehicle's interior space 206. For example, the determination unit 104 can use image segmentation to determine image regions in the images acquired by the camera 102, each corresponding to a portion of the vehicle's interior space 206. The determination unit 104 can then use image points in these image regions as second image points. The processing of these image regions can be implemented in the form of second sub-image data. As an alternative to or supplement to image segmentation, cluster analysis can be used to determine the first and second image points from the image data.

[0042] Point 216 within the interior space 206 of the vehicle is preferably fixed in position. Figure 2 In the illustrated embodiment, point 216 is merely an exemplary point on the front seat 218 of the vehicle 200. Then, based on the second image point and using known methods, such as tracking or optical flow, the determination unit 104 can determine the motion of the display unit 100 in the coordinate system of the vehicle's interior space 206. Furthermore, the determination unit 104 generates second coordinate data corresponding to the motion of the display unit 100 in the coordinate system of the vehicle's interior space 206.

[0043] Then, for example, determining unit 104 determines the motion of vehicle 200 in world coordinates from the difference between the first coordinate data and the second coordinate data. Then, for example, determining unit 104 generates third coordinate data corresponding to the motion of vehicle 200 in world coordinates and provides it for further processing. Then, taking into account the third coordinate data, the position and orientation of display unit 100 in the vehicle's interior space 206 are determined, for example, by determining unit 104.

[0044] Figure 3 The diagram is shown to further clarify the basis. Figure 2 The method is illustrated in graphs 300, 302, 304, 306, 308, 310, and 312. Time is represented on the vertical axis of each graph 300, 302, 304, 306, 308, 310, and 312. The horizontal axis of each graph 300, 302, 304, 306, 308, 310, and 312 represents an exemplary position component of the motion.

[0045] The first graph 300 illustrates, purely exemplarily, the motion of the display unit 100 determined by the inertial measurement unit 106 of the display unit 100. Since the inertial measurement unit 106 measures the acceleration of the display unit 100 in a fixed reference frame, this motion includes the motion of the vehicle 200 and the motion of the display unit 100 relative to the interior space 206 of the vehicle. The inertial measurement unit 106 has a high sampling rate, for example, in the 100Hz range, therefore the first graph 300 is shown as a solid line.

[0046] The second graph 302 is purely exemplary, illustrating the motion of the display unit 100 relative to the vehicle's interior space 206, determined based on images acquired by the camera 102. The second graph 302 therefore corresponds to the first coordinate data. The sampling rate of the camera 102 is lower than that of the inertial measurement unit 106. Furthermore, image data is difficult to process at a rate of 100Hz using portable hardware. Therefore, the determined motion of the display unit 100 relative to the vehicle's interior space 206 is shown as dots. The actual motion of the display unit 100 relative to the vehicle's interior space 206 is shown as dashed lines.

[0047] The third graph 304 is purely exemplary, illustrating the motion of the display unit 100 relative to a fixed reference frame, determined based on images acquired by the camera 102. The third graph 304 therefore corresponds to the second coordinate data. The determined motion of the display unit 100 relative to the fixed reference frame is shown by dots. The actual motion of the display unit 100 relative to the fixed reference frame is shown by dashed lines.

[0048] In the fourth curve 306, the points shown in the second curve 302 and the third curve 304 are superimposed. (As in...) Figure 3 As can be seen, the motions are offset from each other because the motion shown in the third curve 304 includes the motion of vehicle 200. The fifth curve 308 shows the difference between the second curve 302 and the third curve 304, that is, the difference between the first coordinate data and the second coordinate data. This difference corresponds to the motion of vehicle 200 relative to a fixed reference frame. To obtain the continuous motion of vehicle 200, this difference can be smoothed, for example, by interpolation or by using a Kalman filter. Figure 3 It can be clearly seen that the movement of the vehicle 200 is not as violent as the movement of the display unit 100. The low sampling rate of the camera 102 is therefore sufficient to accurately determine the movement of the vehicle 200.

[0049] In the sixth graph 310, the first graph 300 showing the motion of the display unit 100 relative to a fixed reference frame is superimposed with the motion of the vehicle 200 relative to the fixed reference frame. The seventh graph 312 shows the difference between the first graph 300 and the motion of the vehicle 200. This difference corresponds to the motion of the display unit 100 relative to the vehicle's interior space 206. Since both the motion of the vehicle 200 and the motion of the display unit 100 relative to the fixed reference frame are determined with sufficiently high resolution, the motion of the display unit 100 relative to the vehicle's interior space 206 can be determined with high precision by this difference. Therefore, the position and orientation of the display unit 100 relative to the vehicle's interior space 206 can also be determined with high precision.

[0050] Figure 4 A schematic diagram of a vehicle 200 is shown to illustrate another embodiment of a method for operating a display unit 100 for extended reality. Figure 4 The diagram also shows a first coordinate axis 202 and a second coordinate axis 204. The first coordinate axis belongs to the world coordinate system of a fixed reference system, and the second coordinate axis belongs to the coordinate system of the internal space 206 of the vehicle.

[0051] This method is purely exemplary and relies on... Figure 1 The display unit 100 is described herein. The display unit 100 is incorporated into the interior space 206 of the vehicle 200. Figure 4 In this example, the display unit 100 is used by a passenger in the rear seat 208 of the vehicle 200.

[0052] according to Figure 4 Methods and basis Figure 2 The difference in the method is that the motion of the display unit 100 relative to a fixed reference frame is determined by the inertial measurement unit 106 of the display unit 100. For example, the determination unit 104 determines the motion of the display unit 100 relative to the fixed reference frame based on acceleration data generated by the inertial measurement unit 106 and corresponding to the acceleration of the display unit 100 relative to the fixed reference frame.

[0053] Figure 5 The diagram is shown to further clarify the basis. Figure 4 The method is illustrated in graphs 500, 502, 504, 506, 508, and 510. Time is represented on the vertical axis of each graph 500, 502, 504, 506, 508, and 510. The horizontal axis of each graph 500, 502, 504, 506, 508, and 510 represents an exemplary position component of the motion.

[0054] The first graph 500 illustrates, purely exemplarily, the motion of the display unit 100 determined by the inertial measurement unit 106 of the display unit 100. This motion includes both the motion of the vehicle 200 relative to a fixed reference frame and the motion of the display unit 100 relative to the interior space 206 of the vehicle.

[0055] The second graph 502, purely exemplarily, illustrates the motion of the display unit 100 relative to the vehicle's interior space 206, determined based on images captured by camera 102, in point form. The second graph 502 corresponds to the first coordinate data. The actual motion of the display unit 100 relative to the vehicle's interior space 206 is shown by dashed lines.

[0056] In the third curve graph 504, the points of the first curve graph 500 and the second curve graph 502 are shown superimposed. (As in...) Figure 5 As can be seen, the movements are offset from each other because the movements shown in the first graph 500 include the movement of the vehicle 200 relative to a fixed reference frame. The fourth graph 506 shows the difference between the points in the first graph 500 and the second graph 502, that is, the difference between the first coordinate data and the second coordinate data. This difference corresponds to the movement of the vehicle 200 relative to the fixed reference frame.

[0057] In the fifth graph 508, the first graph 500 showing the motion of the display unit 100 relative to a fixed reference frame is superimposed with the motion of the vehicle 200 relative to the fixed reference frame. The sixth graph 510 shows the difference between the first graph 500 and the motion of the vehicle 200. This difference corresponds to the motion of the display unit 100 relative to the interior space 206 of the vehicle.

[0058] In the use of Figures 1 to 5 In the described embodiments, at least camera 102 and determining unit 104 constitute a display unit 100 for augmented reality. Other elements and features shown in the drawings and mentioned in the foregoing description may be part of the claimed display unit 100. Similarly, the method steps described using the display unit 100 may be part of the claimed method.

[0059] List of reference numerals

[0060] 100 display units

[0061] 102 camera

[0062] 104 Determining Units

[0063] 106 Inertial Measurement Units

[0064] 108 display elements

[0065] 200 vehicles

[0066] 202 and 204 coordinate axes

[0067] 206 Interior space of vehicles

[0068] 208 rear seats

[0069] Area 210

[0070] 212 points

[0071] 214 trees

[0072] 216 points

[0073] 218 front seats

[0074] Curve graphs for 300, 302, 304, 306, 308, 310, and 312

[0075] 500, 502, 504, 506, 508, 510 curves

Claims

1. A method for operating a display unit (100) for extended reality, wherein: The display unit (100) is introduced into the interior space of the vehicle (200); Determine the motion of the display unit (100) in world coordinates and generate the corresponding first coordinate data; The camera (102) of the display unit (100) continuously acquires images, the images including at least a portion of the interior space (206) of the vehicle, and generates image data corresponding to the images; Based on the image data, the movement of the display unit (100) in the coordinate system of the vehicle's interior space (206) is determined, and corresponding second coordinate data is generated; The motion of the vehicle (200) in world coordinates is determined by the difference between the first coordinate data and the second coordinate data, and corresponding third coordinate data is generated; and The position and orientation of the display unit (100) in the interior space (206) of the vehicle are determined taking into account the third coordinate data.

2. The method according to claim 1, wherein, The motion of the vehicle (200) in world coordinates is determined by the difference between the first coordinate data and the second coordinate data, using filters and / or a trained machine learning model; and / or The motion of the display unit (100) in world coordinates is determined using the inertial measurement unit (106) of the display unit (100).

3. The method according to any one of the preceding claims, wherein, Images continuously acquired by the camera (102) of the display unit (100) include the region (210) outside the vehicle (200), and the motion of the display unit (100) in world coordinates is determined taking into account the image data.

4. The method according to claim 3, wherein, In the case of image segmentation, first sub-image data is generated from the image data, the first sub-image data corresponding to the image regions of the image, which are respectively included in a portion of the region (210) outside the vehicle (200), and the motion of the display unit (100) in world coordinates is determined in consideration of the first sub-image data.

5. The method according to claim 3 or 4, wherein, Based on the image data, the position of the first image point in successive images is determined in order to determine the movement of the display unit (100) in world coordinates, the first image point corresponding to a point (212) in the region (210) outside the vehicle (200).

6. The method according to claim 5, wherein, The first image point is determined by: using cluster analysis, determining which of the multiple image points in successive images correspond to points (212) in the region (210) outside the vehicle (200).

7. The method according to any one of the preceding claims, wherein, Images continuously acquired by the camera (102) of the display unit (100) include a region (210) outside the vehicle (200). Second sub-image data is generated from the image data using image segmentation. The second sub-image data corresponds to image regions of the image, each of which includes at least a portion of the interior space (206) of the vehicle. The motion of the display unit (100) in the coordinate system of the interior space (206) of the vehicle is determined in consideration of the second sub-image data.

8. The method according to any one of the preceding claims, wherein, Based on the image data, the position of the second image point in successive images is determined in order to determine the movement of the display unit (100) in the coordinate system of the vehicle interior space (206), the second image point corresponding to point (216) in the vehicle interior space (206). Optionally, the second image point is determined by determining, in the case of cluster analysis, which of the image points in a plurality of successive images correspond to the point (216) in the interior space (206) of the vehicle.

9. A display unit (100) for extended reality, the display unit comprising: Camera (102), the camera being configured to continuously acquire images and generate image data corresponding to the images; as well as A determining unit (104) is configured to: determine the motion of the display unit (100) in world coordinates and generate corresponding first coordinate data; Based on image data corresponding to at least a portion of the vehicle interior space (206) including the vehicle (200), the motion of the display unit (100) in the coordinate system of the vehicle interior space (206) is determined, and corresponding second coordinate data is generated; The motion of the vehicle (200) in world coordinates is determined by the difference between the first coordinate data and the second coordinate data, and the corresponding third coordinate data is generated; as well as The position and orientation of the display unit (100) in the interior space (206) of the vehicle are determined taking into account the third coordinate data.

10. The display unit (100) according to claim 9, the display unit comprising an inertial measurement unit (106) configured to generate acceleration data corresponding to the motion of the display unit (100); in, The determining unit (104) is configured to determine the motion of the display unit (100) in world coordinates, taking into account the acceleration data.

Images