System for validating a component of a means of transportation, method and computer program for operating such a system

Abstract

System (1) for validating a component (2) of a means of transport, with - a motion platform (3) with at least three degrees of freedom for a dynamic motion simulation of a cockpit (4) of the means of transport in which the component (2) to be validated is installed; - a mixed-reality display device (5) for displaying an augmented environment to a user (6) located in the cockpit (4); and - a graphics system (7) for calculating (11) the augmented environment, wherein the graphics system (7) is configured to calculate the augmented environment (11) in such a way that for the user (6) at least his hands and the component (2) to be validated are realistically perceptible within a virtual environment, and wherein the cockpit (4) is color-prepared and the graphics system (7) is configured to calculate the augmented environment using a chroma key technique and using a virtual mask (VM) (11), wherein the color preparation of the cockpit (4) is limited to those areas of the cockpit (4) that are not covered by a virtual mask (VM).

Description

[0001] The present invention relates to a system for validating a component of a means of transportation. The invention further relates to a method and a computer program with instructions for operating such a system.

[0002] Dynamic driving simulations are often used to verify or validate functions in automotive development. Initially, projectors were predominantly used for visualization in driving simulators. In recent years, a combination of dynamic driving simulators with virtual reality technologies has become established. In virtual reality (VR), the user is immersed in a completely virtual, computer-generated world. Head-mounted display devices (HMDs), such as VR headsets, are typically used for this purpose. The advantage of using VR headsets is that no hardware components are required for driving simulation. Changes in design, concept, or function can therefore be implemented quickly and virtually.

[0003] However, VR headsets have the disadvantage that realistic, haptic interaction options are very limited in virtual reality. A major problem is that the test subject is isolated from the real world by the VR headset. In particular, the test subject does not see their own hands, but only virtual representations of them. Due to the positional inaccuracy of these virtual representations of the hands, functions in automotive development, for example, that require realistic, haptic interactions between driver and vehicle cannot be evaluated in early development phases in virtual reality. These functions are therefore only evaluated in later development phases, with the costs for changes and optimizations during these development phases rising rapidly.

[0004] Experiments have involved using multiple hand tracking devices simultaneously to reduce the positional inaccuracy of the virtual representations of the hands. However, sufficient accuracy has not yet been achieved. Even approaches based on data gloves have not yet achieved the desired accuracy.

[0005] As an alternative to VR technologies, augmented reality technologies can be used in combination with driving simulators.

[0006] Augmented Reality (AR) involves enhancing the perception of the real world with virtual elements that are accurately registered in three-dimensional space and allow real-time interaction. Data glasses, for example, can be used to display AR displays. Data glasses are worn like regular glasses, but have one or more projection units or displays that can project information in front of the wearer's eyes or directly onto the retina. The glasses are designed so that the wearer can also perceive their surroundings.

[0007] Against this background, WO 2022 / 070033 A1 describes a device for creating and managing prototypes with a plurality of physical structures and a digital medium comprising a CPU and a display and control device. The physical structures are movable and adjustable by means of a plurality of respective actuators. The CPU is configured to control the actuators to adjust the positioning of the physical structures to predetermined positions or to display the position of the physical structures.

[0008] AU 2018 200 822 A1 describes a simulator for a vehicle. The simulator uses green screening to allow a user to see their hands or body in an augmented environment.

[0009] KR 2017 0 005 971 A describes a driving simulator. The driving simulator uses no or only a limited number of physical components. A sense of operation, a driving environment, and an environmental situation that are identical or similar to reality are provided as augmented reality images.

[0010] WO 2012 / 113686 A1 describes a simulator for training a person, comprising a training room, a partially transparent head-mounted display, and a simulation computing unit. The inner boundary surface of the training room is provided with a color suitable for color-based image isolation. The head-mounted display is equipped with two video cameras for stereoscopic recordings and a head tracker that enables the determination of the wearer's gaze direction. The simulation computing unit comprises two image generators for a stereoscopic display of virtual images on one head-mounted display.

[0011] US 2019 / 0 392 639 A1 describes a head-mounted display with a frame and a display system connected to it. A stereo camera connected to the frame generates information comprising image data and depth data. The depth data indicates a distance to a cockpit of a simulator. Based on this information, a processor identifies a cockpit model corresponding to the simulator's cockpit and generates computer-generated images based on the cockpit model.

[0012] The article J. Hönig et al.: “Mixed-Reality-in-the-Loop Simulation”, atp magazine, issue 6-7 2021, describes a solution approach for combining hardware-in-the-loop simulations with modern visualization methods to create a mixed-reality-in-the-loop simulation.

[0013] With the solutions described, the test subject can see their hands, enabling improved haptic interaction between the test subject and real objects. However, further improved solutions are desirable for validating components that require interaction during driving.

[0014] It is an object of the invention to provide improved solutions for the validation of a component of a means of transport which requires interactions during driving operation.

[0015] This object is achieved by a system having the features of claim 1, by a method according to claim 7, and by a computer program with instructions according to claim 8. Preferred embodiments of the invention are the subject of the dependent claims.

[0016] According to a first aspect of the invention, a system for validating a component of a means of transport comprises a motion platform, a mixed-reality display device, and a graphics system. The motion platform has at least three degrees of freedom for dynamic motion simulation of a cockpit of the means of transport in which the component to be validated is installed. The mixed-reality display device is configured to display an augmented environment for a user located in the cockpit. The graphics system is configured to calculate the augmented environment. The augmented environment is calculated such that the user can at least perceive his or her hands and the component to be validated in a real virtual environment.The cockpit is color-prepared and the graphics system is configured to calculate the augmented environment using a chroma key technique and a virtual mask, whereby the color preparation of the cockpit is limited to those areas of the cockpit that are not covered by a virtual mask.

[0017] According to a further aspect of the invention, a method for operating a system according to the invention for validating a component of a means of transport comprises the steps: - Controlling a motion platform of the system according to a movement to be simulated; - Calculating an augmented environment for display by a mixed reality display device of the system, wherein the augmented environment is calculated in such a way that a user can perceive at least his hands and the component to be validated in a real way within a virtual environment; and - Transmitting graphic data from the augmented environment to the mixed reality display device.

[0018] A cockpit of the system is color-prepared and the augmented environment is calculated using a chroma key technique and a virtual mask, whereby the color preparation of the cockpit is limited to those areas of the cockpit that are not covered by a virtual mask.

[0019] According to a further aspect of the invention, a computer program comprises instructions which, when executed by a computer, cause the computer to carry out the following steps for operating a system according to the invention for validating a component of a means of transport: - Controlling a motion platform of the system according to a movement to be simulated; - Calculating an augmented environment for display by a mixed reality display device of the system, wherein the augmented environment is calculated in such a way that a user can perceive at least his hands and the component to be validated in a real way within a virtual environment; and - Transmitting graphic data from the augmented environment to the mixed reality display device.

[0020] A cockpit of the system is color-prepared and the augmented environment is calculated using a chroma key technique and a virtual mask, whereby the color preparation of the cockpit is limited to those areas of the cockpit that are not covered by a virtual mask.

[0021] The term "computer" should be understood broadly. It specifically includes workstations, distributed systems, and other processor-based data processing devices.

[0022] The computer program may, for example, be made available for electronic retrieval or stored on a computer-readable storage medium.

[0023] The inventive solution combines mixed reality technologies with a dynamic driving simulator. Mixed reality (MR) combines elements of virtual reality and augmented reality, i.e., the real world is connected to virtual environments to create a new environment. The user interacts simultaneously with the real and a virtual environment. This makes it possible to integrate the desired real components into the virtual simulation. This allows the simulations to be designed so that the real components are used simultaneously with virtual components. A concrete example is a combination of a real steering wheel, a display, the driver's body, and an otherwise virtual interior. The mixed reality display device can, for example, be designed as a head-mounted display.The component to be validated can be either a physical component or a functionality or software, so that, for example, the content shown on a display can also be validated.

[0024] The combination of mixed reality and dynamic driving simulation opens up numerous new possibilities for functional validation. In a static setup, the evaluation content is significantly limited because many vehicle functions must be tested while driving. The spectrum of assessable content, particularly in automotive development, is significantly expanded by the inventive solution, as the driving simulator enables the driving task to be performed. The planned number of degrees of freedom for the dynamic motion simulation ensures that the user does not experience simulation sickness or that this is significantly reduced. Simulation sickness, often referred to as motion sickness in the literature, frequently occurs without suitable motion simulation because the user drives in the simulation but their body does not move in the real world.A major advantage over dynamic driving simulators that use VR technologies is that the direct haptic interaction with control elements enables a significantly wider range of assessable content, for example in automotive development.

[0025] According to the invention, the cockpit is color-prepared, and the graphics system is configured to calculate the augmented environment using a chroma key technique. For the application of chroma key techniques, the cockpit can be completely or partially covered with a suitable color. Alternatively, the cockpit components that are to be replaced by a virtual environment in the augmented environment can be manufactured directly in the selected color. Another possibility is for parts of the cockpit to be concealed by appropriately colored covers. The most suitable colors for chroma key techniques are green and blue tones, depending on human skin color. The mixed reality display device shows the parts of the cockpit that are executed, painted, or covered in the selected color as virtual content. The control elements that are not executed, painted, or covered in this color, e.g.the steering wheel and touch displays, however, are displayed in real life.

[0026] According to the invention, the graphics system is configured to calculate the augmented environment using a virtual mask. With a virtual mask, real and virtual content can be separated using virtual boundaries that can be defined in various two- or three-dimensional forms. This solution eliminates the need for extensive cockpit preparation, as the preparation can be limited to those areas of the cockpit that are not obscured by a virtual mask.

[0027] According to one aspect of the invention, the motion platform is a hexapod. A hexapod is characterized by six degrees of freedom and high dynamics. This enables a highly realistic driving simulation, largely avoiding the occurrence of simulation sickness. At the same time, a distinctive overall experience is achieved for the user, thus enabling excellent evaluation of the component to be validated and its function.

[0028] According to one aspect of the invention, the mixed-reality display device has a depth sensor, and the graphics system is configured to calculate the augmented environment using depth data from the depth sensor. For example, the depth sensor can be a LIDAR sensor. The depth sensor is preferably integrated into the mixed-reality display device. The use of the depth data enables objects within a specific distance range preferred by the user to be viewed as real and thus not replaced by a virtual environment. A particular advantage is that no extensive preparation of the cockpit is required.

[0029] According to one aspect of the invention, the system comprises a tracking system for tracking the cockpit. The tracking system can be an active or passive tracking system or can be configured to determine the pose of the cockpit from movement data of the motion platform. Such a tracking system is particularly advantageous when virtual masks are used. To use virtual masks in the dynamic simulator, they should move with the simulator.

[0030] Preparing the cockpit for the use of chroma key techniques requires a certain amount of effort. This is especially true when validating different vehicles in the dynamic driving simulator, as the cockpit components must be moved into new positions each time. This may make it impossible to attach a cover for the chroma key techniques due to the different layout. In such cases, it may be advantageous to combine the approaches described above. This way, it may be sufficient to color-prepare only a limited portion of the cockpit for the chroma key techniques.

[0031] A system according to the invention is particularly advantageously used for validating a component of a motor vehicle, in particular a passenger car or a commercial vehicle. The solution according to the invention allows vehicle functions, e.g., autonomous driving functions, to be validated and developed in very early development phases without physical prototypes.

[0032] Further features of the present invention will become apparent from the following description and the appended claims taken in conjunction with the figures. Fig. Figure 1 shows schematically a system for validating a component of a means of transport; Fig. 2 shows schematically a method for operating the system from Fig. 1; Fig. 3 shows schematically a system diagram of a solution according to the invention; Fig. 4 shows schematically a color-prepared cockpit; Fig. 5 shows schematically the use of a depth sensor; and Fig. Figure 6 shows schematically the use of virtual masks.

[0033] To better understand the principles of the present invention, embodiments of the invention are explained in more detail below with reference to the figures. It is understood that the invention is not limited to these embodiments and that the described features may also be combined or modified without departing from the scope of the invention as defined in the appended claims.

[0034] Fig. 1 schematically shows a system 1 for validating a component 2 of a means of transport. The system 1 has a motion platform 3 with at least three degrees of freedom. In the example shown, the motion platform 3 is a hexapod with six degrees of freedom. If necessary, the hexapod can also be mounted on a rotation platform (not shown here). The motion platform 3 enables dynamic motion simulation of a cockpit 4 of the means of transport arranged on the motion platform 3, in which the component 2 to be validated is installed. The cockpit 4 is preferably adaptable in order to be able to set the desired ergonomics for different means of transport. In the example shown, the means of transport is a motor vehicle, and the component 2 to be validated is a steering wheel of the motor vehicle. A user 6 wearing a mixed reality display device 5 is located in the cockpit 4.The mixed reality display device 5 shows the user 6 an augmented environment and can be configured, for example, as a head-mounted display. A graphics system 7, shown here as part of a control computer 9, is used to calculate the augmented environment. The graphics system 7 is configured to calculate the augmented environment such that the user 6 can at least perceive his or her hands and the component 2 to be validated in reality within a virtual environment. For this purpose, the cockpit 4 is color-coded. The graphics system 7 is configured to calculate the augmented environment using a chroma key technique. In addition, the mixed reality display device 5 can have a depth sensor (not shown here), e.g., a LIDAR sensor. In this case, the graphics system 7 is configured to calculate the augmented environment using depth data from the depth sensor.In addition, the graphics system 7 is configured to calculate the augmented environment using virtual masks, with the color preparation of the cockpit being limited to those areas of the cockpit that are not obscured by a virtual mask. For this purpose, it is advantageous if the cockpit 4 is tracked by a tracking system 8 so that the virtual masks move with the cockpit 4 or the motion platform 3. The tracking system 8 can be configured as an active or passive tracking system. Alternatively, it is possible to determine the pose of the cockpit 4 from movement data of the motion platform 3.

[0035] Fig. 2 shows schematically a method for operating the system from Fig. 1. In the method, a motion platform of the system is controlled according to a movement to be simulated 10. In addition, an augmented environment is calculated for display by a mixed reality display device of the system 11. The augmented environment is calculated 11 such that a user can actually perceive at least his or her hands and a component to be validated within a virtual environment. For this purpose, a cockpit of the system is color-prepared. The augmented environment is calculated using a chroma key technique 11. In addition, the mixed reality display device can have a depth sensor, e.g., a LIDAR sensor. In this case, the augmented environment can be calculated using depth data from the depth sensor 11.Furthermore, the augmented environment is calculated using a virtual mask 11, whereby the color preparation of the cockpit is limited to those areas of the cockpit that are not obscured by a virtual mask. For this purpose, it is advantageous if the cockpit is tracked by a tracking system so that the virtual masks move with the cockpit or the motion platform. The tracking data from the tracking system is then used to calculate the augmented environment. The various approaches can also be combined. The graphic data of the augmented environment resulting from the calculation is finally transmitted to the mixed reality display device 12 and displayed by it 13.

[0036] Fig. 3 schematically shows a system diagram of a solution according to the invention. The system 1 comprises a motion platform 3 with the cockpit 4 arranged thereon, a mixed-reality display device 5, and a graphics system 7. The graphics system 7 can be designed as a standalone system or as part of a control computer 9 and generates graphics data GD for the mixed-reality display device 5. The control computer 9 generates control data SD for the movement of the motion platform 3. The mixed-reality display device 5 can have a depth sensor 50 that provides depth data TD for the graphics system 7. The system 1 can further comprise a tracking system 8 for tracking the cockpit 4, which provides tracking data TR. Alternatively or additionally, the control computer 9 can provide movement data BD of the motion platform 3 for the graphics system 7.The graphics system 7 uses the provided data BD, TD, TR to calculate an augmented environment to be displayed by the mixed reality display device 5.

[0037] Fig. 4 schematically shows a color-prepared cockpit 4. In order to be able to calculate the augmented environment using a chroma key technique, the cockpit in the example shown is partially covered with a suitable color, which in Fig. 4 is indicated by hatching. Alternatively, the components of the cockpit 4 that are to be replaced by a virtual environment in the augmented environment can be manufactured directly in the selected color. Furthermore, parts of the cockpit 4 in the example shown are covered by a correspondingly colored cover FA. The most suitable colors for the chroma key technique are shades of green and blue, depending on human skin color. The mixed reality display device 5 shows the parts of the cockpit 4 that are executed or painted in the selected color or the covered parts as virtual content. Other components 2 that are not executed, painted, or covered in this color, e.g. the steering wheel and touch displays, are displayed in real life.

[0038] Fig. Figure 5 schematically shows the use of a depth sensor 50 of the mixed reality display device 5. This approach defines a depth range TB that is actually visible to the user 6. In addition, the cockpit 4 in the example shown is also color-prepared. Preparing the cockpit 4 for the use of chroma key techniques requires a certain amount of effort. By using the defined depth range TB, it is sufficient to color-prepare only a limited portion of the cockpit for the chroma key techniques. The required effort can thus be significantly reduced.

[0039] Fig.Figure 6 schematically shows the use of virtual masks (VM) in the calculation of the augmented environment. Virtual masks (VM) are defined for specific areas. These allow real and virtual content to be separated using virtual boundaries. The boundaries can be defined in various two- or three-dimensional forms. By using virtual masks (VM), the effort required for color-preparing the cockpit 4 can be reduced. The preparation can be limited to those areas of the cockpit 4 that are not obscured by a virtual mask (VM). List of reference symbols 1 system 2 components 3 movement platform 4 Cockpit 5 Mixed Reality Display Device 50 depth sensor 6 users 7 graphics system 8 Tracking system 9 tax calculators 10 Controlling a motion platform 11 Calculating an augmented environment 12 Transmitting graphic data 13 Displaying the augmented environment BD movement data FA Colored Cover GD graphic data SD control data TB depth range TD depth data TR tracking data VM Virtual Mask

Claims

[1] System (1) for the validation of a component (2) of a means of transport, comprising - a movement platform (3) with at least three degrees of freedom for a dynamic movement simulation of a cockpit (4) of the means of transport in which the component (2) to be validated is installed; - a mixed reality display device (5) for displaying an augmented environment for a user (6) located in the cockpit (4); and - a graphics system (7) for calculating (11) the augmented environment, wherein the graphics system (7) is configured to calculate (11) the augmented environment in such a way that at least the user's hands and the component (2) to be validated are actually perceivable within a virtual environment, and wherein the cockpit (4) is color-prepared and the graphics system (7) is configured to calculate (11) the augmented environment using a chroma key technique and using a virtual mask (VM), wherein the color preparation of the cockpit (4) is limited to those areas of the cockpit (4) that are not covered by a virtual mask (VM). [2] System (1) according to claim 1, wherein the motion platform (3) is a hexapod. [3] System (1) according to claim 1 or 2, wherein the mixed reality display device (5) comprises a depth sensor (50) and the graphics system (7) is configured to calculate (11) the augmented environment using depth data (TD) of the depth sensor (50). [4] System (1) according to claim 3, wherein the depth sensor (50) is a LIDAR sensor. [5] System (1) according to one of the preceding claims, comprising a tracking system (8) for tracking the cockpit (4). [6] System (1) according to claim 5, wherein the tracking system (8) is an active or passive tracking system (8) or is configured to determine a pose of the cockpit (4) from movement data (BD) of the movement platform (3). [7] Method for operating a system (1) according to one of claims 1 to 6, comprising the steps: - controlling (10) a movement platform (3) of the system (1) according to a movement to be simulated; - Calculating (11) an augmented environment for display by a mixed reality display device (5) of the system (1), wherein the augmented environment is calculated (11) such that at least the user's hands and a component (2) to be validated are actually perceivable within a virtual environment, wherein a cockpit (4) of the system (1) is color-prepared and the augmented environment is calculated (11) using a chroma key technique and using a virtual mask (VM), wherein the color preparation of the cockpit (4) is limited to those areas of the cockpit (4) that are not covered by a virtual mask (VM); and - Transmitting (12) graphic data (GD) of the augmented environment to the mixed reality display device (5). [8] A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to claim 7 for operating a system (1) according to any one of claims 1 to 6.

Images