System and method for correcting the representation of a virtual representation of a real object in a real environment

Abstract

System for correcting the display of a virtual representation of an object image (2) of a real object (1) in a real environment, comprising a display unit (8) for providing a virtual user environment in which virtual representations of objects (2) of real objects (1) in a user's (9) real environment can be displayed, and at least one user input unit (4) configured for user interaction with a virtual object image (2), the spatial position of which can be detected using a reference system in the real environment and displayed in the virtual environment, wherein the user input unit (4) further comprises acceleration sensors so that a collision with a real object (1) of the real environment can be detected by means of a change in the acceleration of the user input unit (4), wherein a detected collision position (6) is compared with the position of the virtual representation (2) of the real object (1).so that if there is a deviation between the real collision position (6) and the representation position of the virtual object image (2) of the real object (1), the representation of the virtual object image (2) of the real object (1) can be corrected based on the real collision position (6).

Description

[0001] The invention relates to a system and a method for correcting the representation of an object image of a real object in a real environment, which is virtually displayed in a virtual environment. The invention is particularly suitable for recalibrating object images in virtual reality.

[0002] Powerful computers enable the creation of virtual realities in which users can interact with virtually generated objects. Systems for creating virtual reality, or VR for short, usually comprise a computer, a display unit located in real space with which a user can view the virtual reality, and an input unit located in real space, for example in the form of a hand controller, to enable interaction in the virtual environment. For various applications, it is desirable to be able to display real objects from a user's real environment in the virtual environment in order to create a fusion of real and virtual objects. An important component for successful user immersion therefore concerns the realistic reproduction of the position, location, and size of real objects in the virtual environment.However, there is the problem that when reproducing virtual object images of real objects, display errors can occur that cause a deviation in the display form or display position of the virtual object images with respect to the position of the real object. To correct such deviations, a recalibration or recalibration of the system is often necessary, whereby the system is unavailable for use for the duration of the corrective measures to be carried out. For example, corrective measures are known in which a targeted collision between a sensor element and a real object is carried out in order to obtain concrete position information of real objects or their surface contours based on the collision position. However, these are usually intentional calibration processes that require an interruption of use, which leads to disruptions in immersion.

[0003] Other systems, such as Microsoft's HoloLens system, use complex camera-based spatial measurement systems to determine distances between real objects and the VR headset. Another known solution involves a motion capture camera system that observes a moving system equipped with markers to facilitate position tracking. This also takes existing inherent information about the size of the tracked objects into account by checking for collisions in the virtual image of the objects and, if necessary, correcting them through position correction. This enables precise calibration of the object position even with marker shifts.The known solutions have the disadvantage that calibrating virtual object images of real objects requires an interruption of the virtual reality operation and / or additional sensor components, which results in higher overall costs.

[0004] US 2003 / 0 227 470 A1 discloses a system for correcting the appearance of virtual objects in an augmented reality system, where virtual objects are displayed in a real environment together with real objects. In this system, correction of the appearance of real objects is not necessary, since the AR space corresponds to a representation of the real space in which the real objects are displayed.

[0005] US 2016 / 0 353 094 A1 and US 2017 / 0 099 482 A1 each provide solutions for correcting the display of virtual objects, which require active user input to capture position data.

[0006] The object of the invention is to propose a system and a method for the uninterrupted display correction of object images virtually displayed in a virtual environment.

[0007] The problem is solved by a system having the features of patent claim 1 and a method having the features of patent claim 7. Advantageous embodiments or further developments are specified in the respective dependent patent claims.

[0008] The system according to the invention for correcting the representation of an object image of a real object in a real environment, which object image is virtually displayed in a virtual environment, comprises a display unit for providing a virtual user environment in which virtual object images of real objects in a user's real environment can be displayed. A user, within the meaning of the invention, is understood to be a person who interacts with the provided virtual user environment using the system. The system further comprises at least one user input unit configured for user interaction with a virtual object image, the spatial position of which can be detected using a reference system in the real environment and displayed in the virtual environment.The user input unit is further configured to detect a collision with a real object in the real environment, wherein a detected collision position is directly compared with the position of the virtual object image of the real object, so that in the event of a deviation between the real collision position and the display position of the virtual object image of the real object, the display of the virtual object image of the real object can be corrected based on the real collision position.

[0009] In the system according to the invention, a virtual user environment is provided via the display unit, which may comprise a computer or may be coupled to a computer, in which real objects of the real user environment are displayed as virtual object images corresponding to the position, location and size of the respective real objects in order to enable haptic user interaction in the virtual environment.

[0010] Advantageously, in the system according to the invention, all collisions of the user input unit with the real environment are detected during operation, i.e., during user operation, wherein each collision position is directly compared with a display position of a virtual object image of the real object, so that in the event of a deviation between the real collision position and the display position of the virtual object image of the real object, a correction of the display of the virtual object image can be carried out. A correction can provide for a shift of the display position of the virtual object image to a display position in the virtual environment corresponding to the collision position. Furthermore, a correction can consist of adapting a display of the surface contour of the virtual object image to a contour of the real object.

[0011] The at least one user input unit is a device that can be coupled to a computer and has at least one button, switch, or lever for inputting information by a user. The user input unit is configurable such that user inputs can be visualized in the virtual environment. The user input unit can preferably be held in a user's hand. For certain applications, it can be provided that the user input unit is attached to another body part, for example, the user's leg or foot. The spatial position of the user input unit can be detected using a reference system in real space. The reference system can be formed by sensors arranged in real space, wherein the sensors serve as reference points for a coordinate system in which the position of the user input unit can be determined.

[0012] A camera-based system can also be used to determine the position of the user input unit. The position of the user input unit is determined using cameras arranged in real space, in which the user input unit is recognized, for example, by actively illuminated or passively reflective patterns or markers that are clearly visible in the camera image. Sensors arranged on the user input unit can also be used that react to laser irradiation. A device for generating a rotating laser beam is arranged at a known position in real space, and the position of the user input unit is determined based on the travel times of the laser beam as it passes over the sensors arranged on the user input unit.

[0013] According to an advantageous embodiment of the system according to the invention, the user input unit has at least one push button to detect contact with a real object, the push button being actuated, as a collision. According to a further advantageous embodiment of the system according to the invention, it can be provided that the user input unit has at least one proximity sensor for detecting collisions with objects in the real environment. It can further be provided that the proximity sensor is designed as a magnetic proximity sensor, wherein an approach or collision of the user input unit with a real object can be detected based on a change in the magnetic field. It is also conceivable for the user input unit to have several different proximity sensors, wherein capacitive, inductive and / or magnetic proximity sensors can be used.Preferably, the multiple proximity sensors can be arranged at different positions on the user input unit. This makes it possible to additionally detect a contour or profile of the real object when a collision is detected. The use of proximity sensors has the advantage that collisions can be detected before they occur, allowing the system to initiate corrective measures before a collision occurs.

[0014] According to the invention, the user input unit has acceleration sensors so that a collision can be detected based on a change in the acceleration of the user input unit. Computer-assisted software can be used that evaluates an abrupt interruption of an executed movement of the user input unit as a collision with a real object. The software is configured to distinguish between an interruption of movement caused by a normal stopping movement of a user and an interruption of movement caused by a collision. In the case of collision detection without additional acceleration sensors, a movement, acceleration, and an abrupt interruption of acceleration of the user input unit can be determined based on the position determination of the user input unit in the reference system.

[0015] Because continuous collision detection with the user input unit can be based on various sensors, the system can be integrated into existing VR systems. Sensors already present in handheld controllers of conventional systems, such as mechanical push buttons, proximity sensors, or acceleration sensors, can be used to detect collisions with real objects without the need for additional external sensors. Since it is also possible to determine movement, acceleration, and the termination of movement of the user input unit based on the position determination of the user input unit, collision detection can also be performed without additional acceleration sensors in the user input unit.

[0016] The display unit can be a device in the form of glasses that can be worn on the user's head. Preferably, the spatial position of the display unit can also be detected by sensors based on the reference system in the real environment in order to be able to reproduce the viewing position in the virtual environment accordingly. Furthermore, the wearable device of the display unit can be provided with at least one proximity sensor to detect collisions of the wearable device with a real object.

[0017] According to an advantageous development of the system according to the invention, the display unit can have a device for detecting the direction of gaze, with which a virtual object image of the virtual environment fixed by the user's gaze can be detected. By detecting an object fixation in the virtual environment, a collision event with a real object can be assigned to a virtual object image fixed by the user. The assignment of a collision event to a virtual object image fixed by the user facilitates display correction, particularly in a dense arrangement of several virtual object images, since incorrect assignment of a collision event can occur without taking object fixation into account. Camera-based eye-tracking systems, as known from the prior art, are suitable for detecting the direction of gaze.

[0018] Object identification of a virtual object image can also be achieved with a target marker displayed in the user's field of vision, which is positioned with a head movement of the user such that the target marker and the virtual object image to be fixated are congruent or the target marker at least partially covers the virtual object image to be fixated. Once the target marker and the virtual object image to be fixated are congruent, object identification can be triggered by user confirmation, for example, by pressing a button, verbally, or automatically after a predefined period of object fixation with the target marker has elapsed.

[0019] The invention further comprises a method for correcting the representation of an object image of a real object in a real environment, which object image is virtually represented in a virtual environment. This procedure involves determining the spatial position of at least one body part, for example the hand, of a user and / or a user input unit using a reference system in the real environment. A collision of the body part and / or the user input unit with a real object in the real environment is detected, and the detected collision position is directly compared with the representation position of the virtual object image of the real object. In the event of a deviation between the real collision position and the representation position of the virtual object image of the real object, the representation of the virtual object image of the real object in the virtual environment is corrected using the real collision position.

[0020] In the method according to the invention, a collision with a real object is detected, and the collision position, which is determined based on the real-space position of a body part or a user input unit, is directly compared with the display position of the virtual object image of the real object in the virtual environment. Such that, in the event of a deviation between the real collision position and the display position of the virtual object image of the real object, the display of the virtual object image of the real object in the virtual environment can be corrected based on the real collision position without having to interrupt the use of the virtual environment. The method according to the invention thus enables continuous display correction or recalibration of virtual object images during ongoing use without interrupting or significantly disrupting immersion.

[0021] The determination of the real-space position of a body part of a user can be realized with an input unit that can be attached to the body part of the user or held by the body part or with a sensor provided for this purpose, wherein the input unit or the sensor can be spatially located within the reference system of the real environment.

[0022] The reference system can be formed by sensors arranged in real space, whereby the sensors serve as reference points for a coordinate system in which the position of the user input unit can be determined.

[0023] A camera-based reference system can also be used to determine the position of the user input unit. The position of the user input unit is determined using cameras arranged in real space, in which the user input unit is recognized, for example, by actively illuminated or passively reflective patterns or markers that are clearly visible in the camera image. Sensors arranged on the user input unit can also be used that react to laser irradiation. A device for generating a rotating laser beam is arranged at a known position in real space, and the position of the user input unit is determined based on the travel times of the laser beam as it passes over the sensors arranged on the user input unit.When determining the position of the user input unit, position data of the user input unit can be provided in the form of distance-time data, wherein the distance-time data can be used to determine a speed and an acceleration of the user input unit.

[0024] According to an advantageous development of the method according to the invention, a user's gaze direction can additionally be detected in order to identify, based on the user's gaze direction, a virtual object image fixated by the user in the virtual environment. A detected collision event with a real object is assigned to the virtual object image fixated by the user. This has the advantage that a collision event with a real object can be correctly assigned to the virtual object image of the real object displayed in the virtual environment, thus avoiding errors during display correction.

[0025] According to the method according to the invention, a physical representation form and / or representation dimensions of the virtual object image of the real object can be corrected based on a detected collision event with a real object. Therefore, it can be provided that, in addition to shifting the representation position of a virtual object image to the collision position with the real object, a contour or profile of the virtual object image is corrected.

[0026] According to an advantageous embodiment of the method according to the invention, the collision position can be detected when the user input unit touches the real object. According to a further embodiment, in which a magnetic proximity sensor is used, the collision position can be detected by a magnetic sensor when the user input unit approaches a real object. With a magnetic proximity sensor, a collision is detected based on a change in the magnetic field and compared with known position data of the user input unit in order to obtain the specific collision position for comparison with the display position of the virtual object image.

[0027] According to the concept of the invention, an acceleration and / or a speed of the user input unit is detected, wherein the collision position is determined based on a change in the acceleration of the user input unit. An abrupt interruption of a movement of the user input unit is evaluated using software, wherein the software is configured to distinguish between an interruption of movement caused by a normal stop movement of a user and an interruption of movement caused by a collision.

[0028] For efficiency reasons, it may be advisable to limit the acceleration-based collision analysis with the software to objects located in the immediate vicinity of the user input unit. The immediate vicinity is defined as an area of ​​the real environment that can be reached by a user at arm's length. In the acceleration-based collision analysis to determine the collision position, the history of the acceleration values ​​of the real user input unit is recorded. The first derivative of the instantaneous velocities is examined for large negative values, i.e., movement interruptions, after suitable filtering, such as smoothing the noise components using a low-pass filter. Large negative acceleration values ​​usually occur when the user input unit physically collides with a real object.The time of an abrupt movement termination is compared with the determined position data of the user input unit and thus the collision position is determined.

[0029] Since it is not necessarily possible to determine which side or location the user input unit collided with the real object—the recorded acceleration measurements apply to the entire user input unit—the expected local collision positions from the software collision analysis are used for the nearest expected collision position, and this collision position is used to correct the virtual object representation. To identify movement aborts manually induced by the user, the time domain of the velocity and acceleration values ​​of normal movement sequences and collision events can be examined, since the movement sequences of normal movement aborts differ from collision events.To improve the reliability of detecting a collision event, it has proven advantageous to compare the second derivative of the speed values ​​of normal movement sequences and collision events, as a clear difference can be identified here. To identify a collision event, it can therefore be provided that a value range of a second derivative of speed values ​​is defined in order to enable a clear assignment of a collision event. The software is expediently set up to form a second derivative of detected speed values ​​of the user control unit. The speed and acceleration as well as an abrupt termination of movement of the user input unit can be determined with the software, preferably based on tracking the positions of the user input unit detected in real space.The path-time data provided by the user input unit during position determination can be evaluated.

[0030] It is also possible for the speed and acceleration to be determined using acceleration sensors on the user input unit.

[0031] Due to the various options for determining a collision event, the method according to the invention can be applied to existing VR systems, since hand controllers of known designs usually already have at least one proximity sensor and / or acceleration sensor that can be used to detect collisions with a real object. Thus, the method according to the invention can be integrated into existing systems without the need for additional sensors.

[0032] According to an advantageous development of the method according to the invention, it can be provided that a spatial tolerance range is specified in the virtual environment, within which a display correction for a virtual object image fixed by the user is permitted. This has the advantage that an object image fixed by the user is not incorrectly corrected in the event of a collision with a real object that cannot be assigned to the virtual object image. The tolerance range is therefore to be understood as a limited area in virtual space within which a display correction for the virtual object image is permitted.

[0033] According to a further advantageous embodiment of the method according to the invention, it can be provided that at least one further collision position of the real object is determined for the display correction of the virtual object image.

[0034] Further details, features and advantages of embodiments of the invention will become apparent from the following description of exemplary embodiments. Fig. 1: a schematic representation of a first embodiment of the invention Fig. 2: a schematic representation of a second embodiment of the invention Fig. 3: a schematic representation of a third embodiment of the invention

[0035] The same reference symbols were used in the figures to identify recurring features.

[0036] Fig. 1 shows a schematic representation of a first exemplary embodiment of the system according to the invention for correcting the representation of an object image 2 of a real object 1 in a real environment, which object image 2 is virtually represented in a virtual environment. The system has a display unit 8, which is designed in the form of glasses and is worn by a user 9 on the head 3. The display unit 8 is configured to provide a virtual user environment in which virtual object images 2 of real objects 1 in a real environment of the user 9 can be displayed. For this purpose, the display unit 8 can be coupled to a computer (not shown) or can comprise a computer. The system further comprises at least one user input unit 4 configured for user interaction with a virtual object image 2, the spatial position of which can be detected using a reference system in the real environment.The user input unit 4, which is held by the user 9 in the hand 10 as a body part of the user 9, can be represented as a virtual user input unit 5, which is illustrated by a dashed line, in the virtual environment provided by the display unit.

[0037] The virtual object image 2 shown with a dashed line is a reproduction of the real object 1 shown in the virtual environment. This enables the user 9 to haptically interact with the real object 1 in the virtual environment using the user input unit 4, which can be represented in the virtual environment as a virtual user input unit 5, for example as a virtual tool.

[0038] The user input unit 4 is further configured to detect a collision with the real object 1 in the real environment. According to a particularly simple embodiment, the user input unit 4 can have a push button (not shown) that is triggered upon contact with the real object 1, thereby detecting a collision. Based on the position detection of the user input unit 4, a collision position can be determined, which is stored in the Fig. 1 is identified by the reference symbol 6. The detected collision position 6 is directly compared with the position of the virtual object image 2, whereby a deviation between the representation of the virtual object image 2 and the collision position 6 can be determined. In the event that, as in Fig. 1 a deviation between the real collision position 6 and the display position of the virtual object image 2 is detected, a correction of the display of the virtual object image 2 is carried out on the basis of the real collision position 6 by moving the virtual object image 2 in the virtual environment in such a way that the collision of the real environment can be correspondingly displayed in the virtual environment.

[0039] According to an advantageous embodiment variant (not shown), a user input unit 4 with proximity sensors can be used for collision detection. These sensors can be inductive proximity sensors, capacitive proximity sensors, magnetic proximity sensors, optical proximity sensors, or a combination of the aforementioned proximity sensors. By using proximity sensors, collisions can be detected before they occur, allowing the system to initiate measures to correct the display of an optical object image 2 even before a collision occurs.

[0040] Furthermore, the user input unit 4 can have acceleration sensors (not shown), wherein a collision can be detected based on a change in the acceleration of the user input unit 4. In this embodiment, computer-assisted software is used that evaluates an abrupt interruption of an executed movement of the user input unit 4 as a collision with the real object 1. The software is configured to distinguish between an interruption of movement caused by a normal stop movement of a user 9 and an interruption of movement caused by a collision.

[0041] The reference system in which the spatial positions of the user input unit 4 and the display unit 8 are determined can be formed by sensors (not shown) arranged in real space, whereby the sensors serve as reference points for a coordinate system. Furthermore, the reference system serves to position virtual object images in the virtual space.

[0042] Fig. Figure 2 shows a schematic representation of a second embodiment of the system according to the invention for correcting the representation of an object image 2 of a real object 1 in a real environment, which object image 2 is virtually represented in a virtual environment. Fig. The embodiment shown in Figure 2 essentially corresponds to the one shown in Fig. 1, with the difference that the display unit 8 has a gaze direction detection device 8.1, with which a gaze 7 of the user 9 can be tracked. With the aid of the gaze direction detection device 8.1, a virtual object image 2 fixated by the user 9 in the virtual environment is identified, wherein a collision event of the user input unit 4 and the real object 1 is assigned to the identified virtual object image 2. The assignment of a collision event to a virtual object image 2 fixated by the user 9 facilitates display correction, particularly in the case of a dense arrangement of several virtual object images, since without taking the object fixation into account, an incorrect assignment of a collision event can occur, whereby the display correction is carried out incorrectly.

[0043] Fig.3 shows a schematic representation of a third exemplary embodiment of the system according to the invention for correcting the representation of an object image 2 of a real object 1 in a real environment, which object image 2 is virtually represented in a virtual environment. In this exemplary embodiment, the real object 1 has a contour 11 that is touched by the user input unit 4 held in the hand 10 by the user 9, whereby a collision is detected. Based on the position determination of the user input unit 4, a collision position 6 can be detected, based on which a representation correction of the virtual object image 2 is carried out, since the virtual object image 2 is not represented in collision with the virtual user input unit 5 in the virtual environment.Proximity sensors of the user input unit 4 (not shown) enable scanning of the contour 11, so that in addition to correcting the display position of the virtual object image 2, a correction of the virtual display of the contour 11 can also be achieved. List of reference symbols 1 real object, real object 2 virtual object imaging 3 heads 4 User input unit 5 virtual user input unit 6 Collision position 7 Direction of view, view 8 Display unit 8.1 Gaze direction detection device 9 users 10 Hand, body part 11 Contour

Claims

[1] A system for correcting the display of an object image (2) of a real object (1) in a real environment, which object image (2) is virtually displayed in a virtual environment, comprising a display unit (8) for providing a virtual user environment in which virtual object images (2) of real objects (1) in a real environment of a user (9) can be displayed, and at least one user input unit (4) configured for user interaction with a virtual object image (2), the spatial position of which can be detected using a reference system in the real environment and displayed in the virtual environment, wherein the user input unit (4) further comprises acceleration sensors, so that a collision with a real object (1) in the real environment can be detected using a change in the acceleration of the user input unit (4), wherein a detected collision position (6) is compared with the position of the virtual object image (2) of the real object (1),so that in the event of a deviation between the real collision position (6) and the display position of the virtual object image (2) of the real object (1), the display of the virtual object image (2) of the real object (1) can be corrected using the real collision position (6). [2] System according to claim 1, characterized by that the user input unit (4) has at least one proximity sensor for detecting collisions with real objects (1) in the real environment. [3] System according to claim 1 or 2, characterized by that the spatial position of the display unit (8) can be detected using the reference system in the real environment. [4] System according to one of claims 1 to 3, characterized by that the display unit (8) has a device that can be worn by a user (9) on the head (3). [5] System according to one of claims 1 to 4, characterized bythat the display unit (8) has a device for detecting the direction of view (8.1), with which a virtual object image (2) of the virtual environment fixed by the user (9) can be identified. [6] Method for correcting the representation of an object image (2) of a real object (1) of a real environment, which object image is virtually represented in a virtual environment, in which the spatial position of at least one body part (10) of a user (9) and / or of a user input unit (4) is determined using a reference system in the real environment, wherein a collision of the body part (10) and / or of the user input unit (4) with a real object (1) of the real environment is detected using a change in an acceleration of the user input unit (4) and the detected collision position (6) is directly compared with the representation position of the virtual object image (2) of the real object (1), wherein in the event of a deviation between the real collision position (6) and the representation position of the virtual object image (2) of the real object (1),the representation of the virtual object image (2) of the real object (1) in the virtual environment is corrected based on the real collision position (6). [7] Method according to claim 6, characterized by in that a viewing direction (7) of the user (9) is additionally detected in order to identify, on the basis of the user's viewing direction (7), a virtual object image (2) fixed by the user (9) in the virtual environment, wherein a detected collision event is assigned to the virtual object image (2) fixed by the user (9). [8] Method according to one of claims 6 or 7, characterized by that a physical representation form and / or representation dimensions of the virtual object image (2) of the real object (1) is / are corrected. [9] Method according to one of claims 6 to 8, characterized by that the collision position (6) is detected when the user input unit (4) touches the real object (1). [10] Method according to one of claims 6 to 9, characterized by that the collision position (6) is detected by a magnetic sensor when the user input unit (4) approaches the real object (1). [11] Method according to one of claims 6 to 10, characterized by that the second derivative of a movement speed of the user input unit (4) is formed to determine the collision position (6). [12] Method according to one of claims 6 to 11, characterized by that a spatial tolerance range is specified in the virtual environment, in which a display correction is permitted for a virtual object image (2) fixed by the user (9). [13] Method according to one of claims 6 to 12, characterized by that at least one further collision position (6) of the real object (1) is determined for the display correction of the virtual object image (2).

Images