System for verifying component of vehicle, method and computer program for operating such system

Abstract

The invention relates to a system (1) for verifying components of a vehicle. The invention also relates to a method for operating such a system (1) and to a computer program having instructions. The system (1) comprises a motion platform (3), a mixed reality display device (5) and a graphics system (7). The motion platform (3) has at least three degrees of freedom for dynamic motion simulation of a cockpit (4) of the vehicle in which the component (2) to be verified is mounted. The mixed reality display device (5) is configured to display an augmented environment for a user (6) located in the cockpit (4). The graphics system (7) is configured to compute an enhanced environment, the enhanced environment being computed in such a way that the user (6) can truly perceive their hands and the component (2) to be verified at least in the virtual environment.

Description

TECHNICAL FIELD

[0001] The invention relates to a system for verifying components of a means of transport. The invention also relates to a method for operating such a system and a computer program having instructions. BACKGROUND

[0002] For the purpose of securing or verifying functions in the development of a car, dynamic driving simulation is often used. Here, in a driving simulator, visualization is initially carried out mainly using projectors. In recent years, the combination of dynamic driving simulators with virtual reality technology has established itself. In virtual reality (VR), German "virtuelle Realität", the user is immersed in a completely, computer-generated virtual world. For this purpose, a head-mounted display device (HMD), for example VR glasses, is usually used. Using VR glasses has the advantage that no hardware components are required in the driving simulation. Changes in the design, concept or function can thus be implemented quickly and virtually.

[0003] However, VR glasses have the disadvantage that a realistic haptic interaction possibility can only be realized very restrictively in virtual reality. The main problem is that the respective test person is isolated from the real world by the VR glasses. In particular, the respective test person does not see his own hands, but only virtual representatives of the hands. Due to the positional inaccuracy of these virtual representatives of the hands, functions in the development of a car, which require a realistic haptic interaction between the driver and the vehicle, cannot be evaluated in virtual reality in the early development phase. These functions are therefore evaluated only in the later development phase, wherein the costs for changes and optimizations in these development phases increase dramatically.

[0004] Experimentally, a plurality of hand tracking devices have been used simultaneously to reduce the positional inaccuracy of the virtual representatives of the hands. However, sufficient accuracy has not been achieved to date. Data glove-based solutions have also not achieved the desired accuracy to date.

[0005] As an alternative to VR technology, augmented reality technology can be used in combination with a driving simulator.

[0006] Augmented reality (AR), German "erweiterte Realität", is the enrichment of the perception of the real world by virtual elements, which are registered positionally correctly in three-dimensional space and allow real-time interaction. For the purpose of presenting an AR display, data glasses can be used, for example. Data glasses are worn like ordinary glasses, but have one or more projection units or displays, by means of which information can be projected in front of the eyes of the glasses wearer or directly onto the retina. The glasses are designed here in such a way that the wearer can also perceive the environment.

[0007] Against this background, WO 2022 / 070033 A1 describes an apparatus for creating and managing a prototype with a plurality of physical structures and digital media, which digital media comprise a CPU and a display and control apparatus. The physical structures are movable and adjustable by means of a plurality of corresponding actuators. The CPU is configured in such a way that it manipulates the actuators in order to adjust the physical structures to predetermined positions or to display the positions of the physical structures.

[0008] AU 2018 200 822 A1 describes a simulator for a vehicle. The simulator uses green screen technology in order that a user can see his hand or his body in an augmented environment.

[0009] KR 2017 0 005 971 A describes a driving simulator. The driving simulator does not use or only uses a small number of physical components. The same or similar feeling of operation, driving environment and environmental conditions as in reality are provided as augmented reality images.

[0010] In the described solutions, the respective test person can see his hand, so that an improved haptic interaction between the test person and the real object is achieved. However, a further improved solution for verifying components that require interaction in a driving operation is desirable. SUMMARY

[0011] It is the task of the present invention to provide an improved solution for verifying components of a means of transport, in which interaction is required in a driving operation.

[0012] This task is solved by a system having the features of claim 1, a method according to claim 9 and a computer program having instructions according to claim 10. Preferred design solutions of the present invention are the subject of the dependent claims.

[0013] According to a first aspect of the present invention, a system for verifying components of a means of transport comprises a motion platform, a mixed reality display device and a graphics system. The motion platform has at least three degrees of freedom for dynamic motion simulation of a cockpit of the means of transport, in which cockpit the components to be verified are installed. The mixed reality display device is set up for displaying an augmented environment for a user located in the cockpit. The graphics system is set up for calculating the augmented environment. Here, the augmented environment is calculated in such a way that the user can perceive his hand and the components to be verified at least realistically within the virtual environment.

[0014] According to a further aspect of the present invention, a method for operating a system for verifying components of a means of transport according to the present invention comprises the following steps: - manipulating the motion platform of the system according to the motion to be simulated; - computing an augmented environment for display by a mixed reality display device of the system, wherein the augmented environment is computed such that the user can perceive his hands and the component to be verified at least realistically within the virtual environment; and - transmitting graphical data of the augmented environment to the mixed reality display device.

[0015] According to another aspect of the application, a computer program comprises instructions which, when implemented by a computer, cause the computer to implement the following steps for operating a system for verifying a component of a means of transport according to the application: - manipulating a motion platform of the system according to the motion to be simulated; - computing an augmented environment for display by a mixed reality display device of the system, wherein the augmented environment is computed such that the user can perceive his hands and the component to be verified at least realistically within the virtual environment; and - transmitting graphical data of the augmented environment to the mixed reality display device.

[0016] The term "computer" is to be understood broadly here. In particular, it also includes workstations, distributed systems and other processor-based data processing devices.

[0017] The computer program may, for example, be provided for electronic retrieval or stored on a computer-readable storage medium.

[0018] In the solution according to the application, mixed reality technology is combined with a dynamic driving simulator. In mixed reality (MR), German "gemischte Realität", elements of virtual reality and augmented reality are combined, i.e. the real world is combined with a virtual environment, resulting in a new environment. The user interacts with both the real environment and the virtual environment at the same time. In this way, it is possible to integrate the desired real components into the virtual simulation. Thereby, the simulation can be designed such that the real components are used at the same time as the virtual components. A specific example is the combination of a real steering wheel, a display, the driver's body and an otherwise virtual interior. The mixed reality display device may, for example, be designed as a head-mounted display. The component to be verified can be both a physical component and a function or software, so that, for example, the content displayed on the display can also be verified.

[0019] The combination of mixed reality and dynamic driving simulation opens up many new possibilities for the functional assurance. In static structures, the assessable content is significantly limited, since many vehicle functions must be tested during driving. By means of the solution according to the application, the range of assessable content is significantly expanded, especially in automotive development, since the driving simulator fulfils the driving tasks. The number of degrees of freedom set for the dynamic motion simulation ensures here that no or significantly less simulator sickness occurs for the user. Simulator sickness, often also referred to in the literature as motion sickness, often occurs without adequate motion simulation, since the user drives in the simulation, but his body is not moved in the real world. A large advantage compared to dynamic driving simulators using VR technology is that by means of the direct haptic interaction with the operating elements, a significantly greater range of assessable content is achieved, for example in automotive development.

[0020] According to one aspect of the application, the motion platform is a hexapod platform. Hexapod platforms are characterized by six degrees of freedom and high dynamics. This achieves a very realistic driving simulation, whereby the occurrence of simulator sickness is largely avoided. At the same time, a prominent overall experience for the user is achieved, and thus a very good assessability of the components or their functions to be verified.

[0021] According to one aspect of the application, the cockpit is pre-treated with a color and the graphics system is set up to calculate the augmented environment using color keying techniques. For the application of color keying techniques, the cockpit can be completely or partially painted with a suitable color. Alternatively, components of the cockpit to be replaced by the virtual environment in the augmented environment can be manufactured directly in the selected color. Another possibility is that parts of the cockpit are covered by a corresponding colored cover. Depending on the human skin color, the most suitable color for color keying techniques is the green and blue tone. The mixed reality display device displays the parts of the cockpit implemented, painted or covered in the selected color as virtual content. Conversely, operating elements not implemented, painted or covered in this color, such as the steering wheel and the touch display, are displayed realistically.

[0022] According to one aspect of the application, the mixed reality display device has a depth sensor and the graphics system is set up to calculate the augmented environment using depth data of the depth sensor. For example, the depth sensor can be a laser radar sensor. The depth sensor is preferably installed into the mixed reality display device here. Using the depth data achieves that objects in a certain distance range preferred by the user are considered real and thus not replaced by the virtual environment. Here, it is especially advantageous that no extensive preparation of the cockpit is required.

[0023] According to an aspect of the present application, the graphics system is set up to calculate the augmented environment in the case of using virtual masks. In the virtual masks, real and virtual content can be separated by means of virtual boundaries, which can be determined as different two- or three-dimensional shapes. In this solution, neither extensive preparation of the cockpit is required either.

[0024] According to an aspect of the present application, the system has a tracking system for tracking the cockpit. Here, the tracking system can be an active or passive tracking system, or set up so that the pose of the cockpit is determined from the motion data of the motion platform. Such a tracking system is particularly advantageous when using virtual masks. In order to use virtual masks in a dynamic simulator, these masks should move with the simulator.

[0025] Preparation of the cockpit for using the color key technique requires certain expenditure. This is particularly the case when different vehicles are to be verified in a dynamic driving simulator, because in this case the components of the cockpit must be moved to new positions respectively. This can lead to the fact that due to the different structures, the cover for the color key technique cannot be installed. It can be advantageous in such a case to combine the methods described above. In this way, it can be sufficient for the color key technique that only a limited part of the cockpit is color-preprocessed.

[0026] It is particularly advantageous to use the system according to the present application for verifying components of a motor vehicle, in particular a passenger car or a commercial vehicle. By means of the solution according to the present application, it is possible to verify and develop vehicle functions, for example autonomous driving functions, already in a very early development phase without a physical prototype. BRIEF DESCRIPTION OF DRAWINGS

[0027] Further features of the present application will become apparent from the following description and appended claims, considered in conjunction with the accompanying drawings.

[0028] Figure 1 A system for verifying components of a means of transport is schematically shown; Figure 2 A method for operating the system in Figure 1 is schematically shown; Figure 3 A system block diagram of the solution according to the present application is schematically shown; Figure 4 A cockpit color-preprocessed is schematically shown; Figure 5 The use of a depth sensor is schematically shown; and Figure 6 The use of a virtual mask is schematically shown.

[0029] For a better understanding of the principles of the application embodiments of the application are explained in more detail in the following with reference to the drawings. It is understood that the application is not limited to these embodiments and that features described can also be combined or modified without departing from the scope of the application as defined in the appended claims. DETAILED DESCRIPTION

[0030] Figure 1 A system 1 for verifying a component 2 of a means of transport is shown schematically. The system 1 has a motion platform 3 with at least three degrees of freedom. In the example shown, the motion platform 3 is a hexapod platform with six degrees of freedom. If required, the hexapod platform can additionally be fitted on a not shown rotary platform. The motion platform 3 effects a dynamic motion simulation of a cockpit 4 of the means of transport, which is arranged on the motion platform 3, in which the component 2 to be verified is installed. The cockpit 4 is preferably adaptable in order to be able to set the desired ergonomics for different means of transport respectively. In the example shown, the means of transport is a motor vehicle and the component 2 to be verified is a steering wheel of the motor vehicle. In the cockpit 4 there is a user 6, who wears a mixed reality display device 5. The mixed reality display device 5 shows an augmented environment to the user 6 and can for example be designed as a head-mounted display. A graphics system 7, which is shown here as a component of a control computer 9, is used to calculate the augmented environment. The graphics system 7 is set up here in such a way that it calculates the augmented environment in such a way that the user 6 can perceive his hand and the component 2 to be verified at least virtually within a virtual environment. For this purpose, the cockpit 4 can be pre-processed in terms of color. In this case, the graphics system 7 is set up to calculate the augmented environment using chroma key technology. Alternatively or additionally, the mixed reality display device 5 can have a not shown depth sensor, for example a laser radar sensor. In this case, the graphics system 7 is set up to calculate the augmented environment using depth data of the depth sensor. Finally, there is also the possibility of calculating the augmented environment using a virtual mask. For this purpose, it is advantageous to track the cockpit 4 by means of a tracking system 8 in order to move the virtual mask together with the cockpit 4 or the motion platform 3. The tracking system 8 can be designed as an active or passive tracking system 8. Alternatively, it is possible to determine the pose of the cockpit 4 from the motion data of the motion platform 3.

[0031] Figure 2 A system 1 for verifying a component 2 of a means of transport is shown schematically. The system 1 has a motion platform 3 with at least three degrees of freedom. In the example shown, the motion platform 3 is a hexapod platform with six degrees of freedom. If required, the hexapod platform can additionally be fitted on a not shown rotary platform. The motion platform 3 effects a dynamic motion simulation of a cockpit 4 of the means of transport, which is arranged on the motion platform 3, in which the component 2 to be verified is installed. The cockpit 4 is preferably adaptable in order to be able to set the desired ergonomics for different means of transport respectively. In the example shown, the means of transport is a motor vehicle and the component 2 to be verified is a steering wheel of the motor vehicle. In the cockpit 4 there is a user 6, who wears a mixed reality display device 5. The mixed reality display device 5 shows an augmented environment to the user 6 and can for example be designed as a head-mounted display. A graphics system 7, which is shown here as a component of a control computer 9, is used to calculate the augmented environment. The graphics system 7 is set up here in such a way that it calculates the augmented environment in such a way that the user 6 can perceive his hand and the component 2 to be verified at least virtually within a virtual environment. For this purpose, the cockpit 4 can be pre-processed in terms of color. In this case, the graphics system 7 is set up to calculate the augmented environment using chroma key technology. Alternatively or additionally, the mixed reality display device 5 can have a not shown depth sensor, for example a laser radar sensor. In this case, the graphics system 7 is set up to calculate the augmented environment using depth data of the depth sensor. Finally, there is also the possibility of calculating the augmented environment using a virtual mask. For this purpose, it is advantageous to track the cockpit 4 by means of a tracking system 8 in order to move the virtual mask together with the cockpit 4 or the motion platform 3. The tracking system 8 can be designed as an active or passive tracking system 8. Alternatively, it is possible to determine the pose of the cockpit 4 from the motion data of the motion platform 3. Figure 1The method in the system is shown in Fig. 1. In the method, a motion platform of the system is maneuvered 10 in accordance with the motion to be simulated. Furthermore, an augmented environment for display by a mixed reality display device of the system is calculated 11. Here, the augmented environment is calculated 11 such that the user can perceive his hands and the component to be verified at least virtually realistically within the virtual environment. For this purpose, the cockpit of the system can be color preprocessed. In this case, the augmented environment can be calculated 11 using color keying techniques. Alternatively or additionally, the mixed reality display device can have a depth sensor, for example a laser radar sensor. In this case, the augmented environment can be calculated 11 using depth data of the depth sensor. Finally, there is also the possibility of calculating 11 the augmented environment using a virtual mask. For this purpose, it is advantageous for the cockpit to be tracked by a tracking system so that the virtual mask moves with the cockpit or the motion platform. The tracking data of the tracking system are then used to calculate the augmented environment. The different methods can also be combined here. Finally, the calculated graphics data of the augmented environment are transmitted 12 to the mixed reality display device and displayed 13 there by.

[0032] Figure 3 A system diagram of the solution according to the application is shown schematically. The system 1 comprises a motion platform 3 with a cockpit 4 arranged thereon, a mixed reality display device 5 and a graphics system 7. The graphics system 7 can be designed as a separate system or as a component of the control computer 9 and generates graphics data GD for the mixed reality display device 5. The control computer 9 generates control data SD for moving the motion platform 3. The mixed reality display device 5 can have a depth sensor 50 which provides depth data TD for the graphics system 7. The system 1 can furthermore comprise a tracking system 8 for tracking the cockpit 4, which provides tracking data TR. Alternatively or additionally, the control computer 9 can provide the graphics system 7 with motion data BD of the motion platform 3. The graphics system 7 uses the provided data BD, TD, TR when calculating the augmented environment to be displayed by the mixed reality display device 5.

[0033] Figure 4 A cockpit 4 which is color preprocessed is shown schematically. In order to be able to calculate the augmented environment using color keying techniques, the cockpit is partly painted in suitable colors in the example shown, which is indicated by the different colors in the cockpit 4. The cockpit 4 is then tracked by a tracking system 8, which provides tracking data TR. The tracking data TR are used to calculate the augmented environment. In this case, the virtual mask is calculated in such a way that it is moved with the cockpit 4. The tracking data TR are used to calculate the virtual mask. The virtual mask is then displayed by the mixed reality display device 5. Figure 4The parts of the cockpit 4 that are to be replaced by the virtual environment in the augmented environment are indicated by hatching. Alternatively, the parts of the cockpit 4 that are to be replaced by the virtual environment in the augmented environment can be manufactured directly in the chosen color. Furthermore, in the shown example, parts of the cockpit 4 are covered by a respective colored cover FA. Depending on the human skin color, the most suitable color for the color key technique is a greenish or bluish color. The mixed reality display device 5 displays the parts of the cockpit 4 that are implemented or painted in the chosen color or covered as virtual content. Vice versa, the other parts 2 that are not implemented, painted or covered in that color, e.g. the steering wheel and the touch display, are displayed as real.

[0034] Figure 5 The use of a depth sensor 50 of the mixed reality display device 5 is shown schematically. In this approach, a depth range TB is defined that is visible to the user 6 as real. Additionally, in the shown example, the cockpit 4 is again pre-processed with colors. Preparing the cockpit 4 for using the color key technique requires a certain effort. By using the defined depth range TB, it is sufficient to color- pre-process only a limited part of the cockpit for the color key technique. Thus, the required effort can be reduced significantly.

[0035] Figure 6 The use of virtual masks VM in computing the augmented environment is shown schematically. Virtual masks VM are defined for certain areas. These masks allow to separate real and virtual content by means of virtual boundaries. These boundaries can be determined here as different two- or three-dimensional shapes. By using virtual masks VM, the effort for color-preprocessing the cockpit 4 can be reduced. The preprocessing can also be limited to those areas of the cockpit 4 that are not covered by the virtual masks VM.

[0036] List of reference signs: 1 system 2 part 3 motion platform 4 cockpit 5 mixed reality display device 50 depth sensor 6 user 7 graphics system 8 tracking system 9 control computer 10 maneuvering motion platform 11 computing augmented environment 12 transmitting graphics data 13 displaying augmented environment BD motion data FA colored cover GD graphics data SD control data TB depth range TD depth data TR tracking data VM virtual mask.

Claims

1. A system (1) for verifying a component (2) of a transport vehicle, comprising: - A motion platform (3) with at least three degrees of freedom is used to perform dynamic motion simulation on the cockpit (4) of the vehicle, and the component (2) to be verified is installed in the cockpit; - A mixed reality display device (5) for displaying an enhanced environment to a user (6) located in the cockpit (4); and - A graphics system (7) for computing the enhanced environment described in (11), wherein, The graphics system (7) is configured to calculate (11) the enhanced environment such that the user (6) can realistically perceive his hand and the component to be verified (2) at least within the virtual environment.

2. The system (1) according to claim 1, wherein, The motion platform (3) is a hexapod platform.

3. The system (1) according to claim 1 or 2, wherein, The cockpit (4) is color preprocessed, and the graphics system (7) is configured to calculate (11) the enhanced environment using chroma key technology.

4. The system (1) according to claim 3, wherein, The mixed reality display device (5) has a depth sensor (50), and the graphics system (7) is configured to calculate (11) the augmented environment using the depth data (TD) of the depth sensor (50).

5. The system (1) according to claim 4, wherein, The depth sensor (50) is a lidar sensor.

6. The system (1) according to any one of claims 3 to 5, wherein, The graphics system (7) is configured to compute (11) the enhanced environment using a virtual mask (VM).

7. The system (1) according to claim 6, having a tracking system (8) for tracking the cockpit (4).

8. The system (1) according to claim 7, wherein, The tracking system (8) is an active or passive tracking system, or is configured to determine the position of the cockpit (4) by motion data (BD) of the motion platform (3).

9. A method for operating the system (1) according to any one of claims 1 to 8, comprising the following steps: - The motion platform (3) of the system (1) is controlled according to the motion to be simulated; - Calculation (11) is used to display an enhanced environment via the mixed reality display device (5) of the system (1), wherein, The enhanced environment is calculated in such a way (11) that the user (6) can realistically perceive his hand and the component to be verified (2) at least within the virtual environment; and - Transmit (12) the graphics data (GD) of the augmented environment to the mixed reality display device (5).

10. A computer program having instructions that, when executed by a computer, cause the computer to perform the steps of the method for operating the system (1) according to any one of claims 1 to 8 as described in claim 9.

Images