Method for training users by means of VR-enabled terminals

Abstract

The invention relates to a method (100, 200) for training users (1) by means of virtual reality (VR)-enabled terminals (20), said method (100, 200) having the steps of: providing first data, said first data comprising images in a first image format (301) for training the users; transforming the first data into second data such that the second data comprises the images of the first data in a second image format (302), said second image format being designed to display at least one partial virtual reality on the VR-enabled terminals (20), and providing the second data for transmission to the VR-enabled terminals (20).

Description

PROCEDURES FOR TRAINING USERS USING VR-CAPABLE DEVICES TECHNICAL FIELD

[0001] The present invention relates to a method for training users by means of virtual reality (VR)-capable terminals, a computer program product, a device for data processing and a system comprising such a device. BACKGROUND OF THE INVENTION

[0002] Today, the use of virtual reality (VR) is becoming increasingly important in many areas. However, traditional training methods continue to rely on pre-produced content that offers no opportunity for interaction, thus untapped VR technology's potential. This is partly due to the fact that VR technology is complex and expensive to acquire, making it difficult to use in training. SUMMARY OF THE INVENTION

[0003] The above problem is at least partially solved by the subject matter of the independent claims of the present invention, examples of which are shown in the dependent claims.

[0004] According to a first aspect, the invention relates to a method for training users using virtual reality (VR)-capable devices, the method comprising: - Providing first data, wherein the first data comprises images for training users in a first image format, - Transforming the first data into second data (for training the users by means of the VR-capable terminals or, in other words, the second data is configured for training the users by means of the VR-capable terminals) such that the second data comprises the images of the first data in a second image format, wherein the second image format is configured for displaying at least a partial virtual reality on the VR-capable terminals, and - Providing the second data for transmission to the VR-capable devices.

[0005] Virtual Reality (VR) refers to a reality that is at least partially or completely only virtual, but at least not entirely physical or, in other words, exists in the real world. Virtual reality is therefore a reality that is at least partially or entirely only apparent, as it can be conveyed to one or more users by a VR-capable device. The conveyance can take place in particular on a visual level, for example via at least one screen of the VR-capable device, which is in the visual field of view of the user when the user is using the VR-capable device, in particular when wearing it. In addition, the conveyance of virtual reality can also take place on an auditory and / or touch level, whereby the VR-capable device can accordingly be equipped, for example, with at least one audio output means, such as a loudspeaker.A virtual reality can, for example, be represented by a partially or completely virtual environment, as can be presented to the user in particular by the VR-capable device. The user can thus experience at least the visual representation and simultaneous perception of an apparent reality with its, for example, visually visible, audible and / or touchable physical properties. The virtual environment can, in particular, be computer-generated in real time. The virtual environment can be interactive. Interactive means, in particular, that the user can interact with the virtual environment, i.e., make one or more inputs. The inputs can take any form, for example touching an object in the virtual environment or voice interaction.Interactive can also mean that the user's field of vision is adjusted through a movement, such as head movement or something similar. Furthermore, interactive can also mean that an interaction takes place with another user in the virtual environment, such as communication. An interaction can also mean, for example, an adjustment of the virtual environment. The virtual environment can be simulated by the VR-capable device. In particular, the simulation can take place as a three-dimensional environment, giving the user a particularly immersive experience in the virtual environment. By simulating the virtual environment in 3D, or in other words, stereoscopically, the physical objects depicted in the virtual environment can appear three-dimensional or tangible. Interacting with objects depicted in this way makes the user experience even more realistic and therefore more immersive.As indicated, the virtual environment can be created through visual, auditory, and / or haptic impressions to give the user the feeling of being in a real environment and to make interactions within it seem even more realistic. The user's field of vision can be completely or partially covered by the virtual environment, particularly through one or more screens of the VR-capable device. This can further enhance the immersive experience, where the user feels like they are moving around the virtual environment as if they were in a real environment and can orient themselves as if they were physically present.

[0006] At least partial virtual reality can mean that the virtual reality does not completely cover the entire visual environment, but only a part of it. This means that while some elements of the environment, such as objects, colors, backgrounds, etc., can be virtual and computer-generated, others can belong to the real environment. In particular, it is a mixture of real elements, i.e. those that are visible to the user in the real world without a VR-capable device, and virtual, i.e. computer-generated, elements. For example, elements in the virtual environment can be real, while other elements are only virtual. This makes it possible to create an immersive experience without generating the entire environment reproduced or simulated by the VR-capable device in virtual reality.The generation of virtual reality can, for example, be a replication of the real environment in the user's field of vision and / or the creation of an artificial environment that does not exist in the real environment of the user's field of vision or is transformed from it. Accordingly, at least partial virtual reality can also be an augmented reality or a mixed reality, as explained in more detail below using examples.

[0007] The term "virtual reality" is therefore to be interpreted broadly and can encompass various options discussed herein in connection with VR. At least partial virtual reality can therefore also mean that only one, some, or all of the virtual reality options mentioned here are fulfilled, which at least support the creation of the virtual environment and, in particular, make it immersive. For example, virtual reality or its virtual environment can be created or supported by an enlarged field of view, with or without simulation of a three-dimensional environment, with or without interactivity in the virtual environment, etc.

[0008] A VR-enabled device can be any device of a user that can create at least a partial virtual reality. For example, the VR-enabled device can be VR glasses, a VR headset, a smartphone, a VR-enabled computer, a VR-enabled game console, or similar. The VR-enabled device can, for example, comprise one or more screens that display the virtual environment. The screen(s) can display a single large image for display both eyes of the user (monocular screen) or separate images for each eye of the user (binocular screen), for example to create a stereoscopic or, in other words, 3D effect. Furthermore, the VR-capable device can have one or more sensors that track the user's movements. Based on the user's movement, the VR-capable device can be configured to adapt the virtual environment, for example to interact with elements of the virtual reality or environment and / or to move the user's field of vision in the virtual environment. The sensor(s) can be, for example, a gyroscope, accelerometer and / or magnetometer. Additionally, or alternatively, device-external tracking systems with one or more optical sensors can also be used to enable particularly precise motion detection.The VR-capable device can have an audio output, for example, via integrated speakers and / or headphones. The VR-capable device can have a processor and / or a graphics card. The processor and / or graphics card allow for fast calculation and display of the virtual environment, essentially in real time. They can also be responsible for receiving and further processing the secondary data.

[0009] User training images can contain a variety of information and visualizations designed to support and enhance a learning process. User training images can include, but are not limited to, instructional materials, diagrams, graphs, illustrations, infographics, animations, videos, images of objects or scenarios, and / or instructions.

[0010] The provision of the initial data can be done through various means, such as a server, computer, video player or similar.

[0011] The first data can include various images for training users. These can, for example, be images that represent training content or teaching materials. These images can, for example, include graphics that contain information, diagrams, illustrations, or other visual elements. The first data can, for example, also be a sequence of images, a video, a stream, or the like. In other words, the images of the first data can, in particular, be arranged in a sequence relative to one another or sorted so that moving images, such as a sequence of images or a video, can be generated from them. In addition to the image material, in particular video material based on the images, the first data can, for example, also include one or more audio tracks. The first data can also include textual information such as instructions, descriptions, learning materials, or the like. The first data can also include 3D models or similar.

[0012] The images of the first data can be displayed in a first image format. The first image format can, for example, include 2D images or animations presented on a conventional, in particular flat, screen or screen configured for 2D display. The first data can have any file format such as JPEG, PNG, GIF, BMP, GIFs, or similar. Alternatively or additionally, the first data can also have any video format such as MP4, AVI, MOV, MKV, or similar. The first data can also include 3D models, which can be in any relevant formats such as OBJ, FBX, STL, glTF, or similar.

[0013] The first data in the first image format are transformed into the second image format of second data, wherein the second image format is set up for the representation of at least a partial virtual reality on the VR-capable end devices. This transformation can, for example, relate to one of the options explained above as an example in order to at least support or enable the creation of the virtual reality or virtual environment. The transformation of the first data into the second data can, for example, comprise adjusting the resolution, image quality, color and contrast adjustments, adjusting the file size, for example by compression, creating separate images for each eye of the user in order to enable a stereoscopic representation, an expanded field of view or the like. For example, the impression of an expanded field of view and / or a three-dimensional perception of virtual reality orvirtual environment created by the user of the VR-capable device.

[0014] The second data comprises the images that were transformed from the first data of the first image format and generated and / or edited, in particular optimized, for display in a virtual environment. The second image format can be specifically configured to enable an immersive display in at least a partially virtual reality or environment on the VR-capable end devices. The second data can be configured for provision to the VR-capable end devices. The second data can be provided, for example, by a computer or server, for example a streaming server, via a cloud platform, through local networks, peer-to-peer transmission, or a client-server architecture or the like.

[0015] The method may be at least partially computer-implemented. Accordingly, one, several, or all steps of the method may be carried out by a data processing device. The method may be implemented in software and / or hardware. be implemented. Furthermore, the method can be carried out by computer program instructions that can be executed on a data processing device, in particular a computer. The data processing device can comprise one or more processors, a memory, a data interface, etc.

[0016] For example, the images of the first data in the first image format can be configured for display on a non-VR-capable device. Non-VR-capable devices can be devices that do not have the necessary technical specifications or functions to display virtual environments. Examples of non-VR-capable devices can be computers, smartphones, game consoles, televisions, digital projectors, or similar. For example, the first image format can be an image format for displaying monoscope images and / or with a flat or uncurved field of view.

[0017] For example, the second image format for projecting the images of the second data can be configured such that it corresponds to an expanded and / or enlarged field of view on the VR-capable end devices compared to the first image format, or in other words, results in such an expanded and / or enlarged field of view on the VR-capable end devices. In particular, the images of the second data in a second image format on the VR-capable end devices can have an expanded and / or enlarged field of view compared to the images of the first data in a first image format on a non-VR-capable end device. The field of view can refer to the user's field of view, in particular a human field of view. An expanded and / or enlarged field of view can thus refer to the area that a user can see of the virtual environment.The enlarged field of view can already be enlarged compared to the first image format to such an extent that the user has a larger field of view without having to move their head or eyes than, for example, with the first image format displayed on a non-VR-capable device. Additionally or alternatively, the expanded field of view can refer to the fact that the user can shift their field of view by turning their head or moving their eyes, whereby the displayable field of view can be larger than the field of view or field of view that the user can perceive without moving their head and / or eyes. Accordingly, the user can move within the expanded field of view by rotating their head and / or moving their eyes by varying the currently visible area in the field of view.This can be supported in particular by means of the aforementioned at least one sensor installed in the VR-capable device or externally, whereby the sensor data can be used to track head rotation and / or eye movement in order to vary the visible area in the field of view. The second image format can in particular be configured to enable a projection of the images of the second data with a field of view of at least 60°, in particular between 60° and 180°, very particularly between 60° and 140°, on the VR-capable terminals, in particular on their one or more screens.

[0018] For example, the second image format can be configured to project the images of the second data with a fisheye field of view on the VR-capable end devices. A fisheye field of view is characterized by a wide perspective. The images of the second data can, for example, be distorted to enable a wider field of view. In particular, the virtual environment in the fisheye field of view can be displayed in a curved and / or spherical format. In this context, this means that the first image format, which may be in a non-spherical format, is transformed into a spherical format. The second image format can further be configured to reproduce the predominant or substantially complete field of view of a user in the at least partially virtual reality. The images of the second data can be further adapted to take the distortion of the fisheye effect into account.By taking the distortion of the fisheye effect into account, the images of the second data can be projected with a fisheye field of view on the VR-enabled devices, especially on the screen, without resulting in a distorted view.

[0019] For example, the second image format can be configured so that a change in the position of the VR-capable device enables a shift in the current field of view on a screen of the VR-capable device within the field of view. The current field of view can shift, in particular, relative to the expanded and / or enlarged field of view. For example, movement data relating to the user, such as inclinations, rotations, for example of the head, or similar movements of the VR-capable device and / or user, can be processed by the VR-capable device. The movement data can be recorded, for example, using the aforementioned at least one sensor. The VR-capable device then adjusts the field of view on the VR-capable device according to the movement data. This means that the current field of view on the screen of the VR-capable device can change if the position of the VR-capable device is changed.In other words, the VR-capable device itself can process this motion data to adjust the field of view displayed on its screen. For example, the current field of view can be shifted immediately as soon as the position or orientation of the device changes, without the need for constant data exchange with a computer. It is also possible that the motion data can be adjusted by tilting, rotating, or similar actions. VR-capable device and / or user, are processed by the computer, in particular the computer carrying out the method with regard to at least the steps of providing the first data, transforming the first data and providing the second data. The movement data is then first sent to the computer. The computer adjusts the field of view of the screen of the VR-capable device according to the movement data and sends the data of the adjusted field of view back to the VR-capable device. The adjustment of the visible area on the screen can take place in real time, based on the movement data of the VR-capable device and / or the user. The ability to move the field of view on the screen of the VR-capable device enables the user to move around in the virtual environment.

[0020] For example, the first data can comprise monoscopic images. Transforming the first data can include transforming the monoscopic images of the first data into stereoscopic images of the second data to create a stereoscopic impression on the VR-capable end devices. In contrast to monoscopic images, stereoscopic images are slightly different for each eye and thus enable a spatial impression or a 3D effect. These stereoscopic images, or in other words, 3D images, can be created, for example, using the parallax effect, i.e., by shifting multiple elements in the image relative to one another, as part of the transformation of the first data. Alternatively or additionally, depth maps can also be obtained from the images of the first data. Techniques such as depth map extraction can be used for this purpose.This information can then be used to generate stereoscopic images as part of the transformation of the first data. The images of the second data can, in particular, be transformed such that they present different perspectives for the left and right eye, creating the impression of a 3D effect. These stereoscopic images of the second data can then, for example, be configured to display a 3D model on VR-capable devices. By converting the monoscopic images into stereoscopic images, a stereoscopic impression is created, giving the user the feeling of actually being in a three-dimensional environment.

[0021] For example, the method may further comprise receiving the second data on the VR-capable terminals and projecting the images of the second data onto a sphere, in particular a hemisphere, of screens of the VR-capable terminals. The second data resulting from the transformation of the first data may, for example, be processed by the computer mentioned herein (i.e., in particular, the computer used to describe the method). The second data can then be sent to the VR-capable device (i.e., a computer carrying out at least the steps of providing the first data, transforming the first data, and providing the second data). This can be done, for example, via a wireless and / or wired connection between the computer and the VR-capable devices. The second data can then be received by the VR-capable device. The images of the second data can be projected onto a sphere, in particular a sphere of a hemisphere, of screens of the VR-capable devices. The projected sphere, in particular a sphere of a hemisphere, can be displayed, for example, on the screens of the VR-capable devices. A hemisphere is a geometric shape that is one half of a sphere. However, spheres of a partial sphere, a full sphere, or the like are also conceivable.In this context, the hemisphere refers to the spherical surface onto which the images of the second data are projected. For example, the images of the second data can be rendered onto the surface of a projected sphere surrounding the user's field of view. A projection onto a hemisphere can, for example, help create a particularly immersive virtual reality experience by creating the impression that the virtual environment surrounds the user's field of view. A virtual camera can be located at the center of the hemisphere, simulating the user's viewing direction or perspective. A viewing direction can be further determined, for example, by position sensors of the VR-capable devices.Instead of spheres, other shapes can also be conceivable for projection in virtual reality, such as surfaces of a complete sphere, a cylinder, a cone or dynamically changing shapes, or similar.

[0022] For example, distortion correction can be performed when projecting the images of the second data onto the sphere. Since the second images are projected onto a spherical surface, distortions can occur. This distortion can occur, for example, in the corner regions of the projected second images. Distortion correction can be performed using various methods, including techniques such as geometric transformations and other image processing techniques. These methods can be used, for example, to adjust the projected images of the second data to conform to the curvatures of the sphere. Distortion correction can, for example, involve clipping these corner regions to avoid projecting these corner regions onto the sphere. UV mapping, for example, can be used for this purpose. UV mapping is a technique for projecting 2D textures onto 3D models without causing distortion or overlap.UV mapping allows the corner areas to be projected onto a curved surface such as a sphere. Distortions can be avoided or minimized. Distortion correction can also take the user's viewing perspective into account to ensure that projected images are displayed correctly from different perspectives.

[0023] For example, providing the first data can include a screen recording or a live stream of the images of the first data. During screen recording, the images of the first data of a non-VR-capable end device are recorded from its screen. The screen recording can, for example, be carried out by the computer, which can be set up to capture and record screen content. Screen recording can also be referred to as screen capturing. With this method, the screen content of a particularly non-VR-capable end device is captured. The captured screen content can be recorded and provided as first data in order to be transferred to the VR-capable end devices after it has been transformed. The first data can also be provided in real time, for example via live stream.The images of the initial data can be transmitted directly to the computer via the internet or a local network, for example. A so-called Spout protocol can be used to deliver the images of the initial data. The Spout protocol allows images to be delivered directly from the non-VR-capable device to the computer, without, for example, having to go through a buffer storage process.

[0024] For example, there may be a latency of less than 2 seconds between the provision of the first data and the second data. Latency refers to the delay or time between the two provisioning operations. Specifically, the latency may be less than 1 second or less than 0.5 seconds. The provision of the second data may therefore be in real time or virtually real time.

[0025] For example, the method may further comprise compressing the second data, wherein the compressed second data is provided for transmission to the VR-capable end devices. Compression is a process in which the amount of data is reduced, for example by removing redundant information or replacing it with more efficient representations. By compressing the second data, the amount of data can be reduced, which can lead to more efficient transmission, for example via networks or wireless connections. This can, for example, ensure fast transmission to the VR-capable end devices. Furthermore, by compressing the second data, bandwidth usage can be optimized and ensure that the transmission can be carried out efficiently even with limited network resources. This is particularly relevant for user training, since a larger number of Users are being trained, so network resources must be distributed in a particularly resource-efficient manner. For example, if a school class with 20, 30 or more students is being trained simultaneously, with each of these students using their own VR-capable device, bottlenecks can quickly arise without compression, or simultaneous training with the same second data cannot be carried out for all students if network resources are limited. In particular, training deployments where network resources are typically limited, or where a very large number of users in one location or building use the same network resources, such as in schools, can benefit from the compression process, which in these examples makes the training possible in the first place.In particular, the method can be set up for training at least five, very particularly at least ten, and further very particularly at least twenty or more users using VR-capable end devices. The second data can be made available for transmission to all VR-capable end devices, in particular substantially simultaneously or with temporal overlap. After compression, the compressed second data can be made available for transmission to the VR-capable end devices. The compression can, for example, take place on the same computer on which the provision and transformation of the first data and the provision of the second data take place. However, the compression can also take place, for example, on a separate computer, in particular a router between the computer and the VR-capable end devices.

[0026] For example, the method can further comprise transmitting the second data to the VR-capable end devices, wherein the transmission is carried out by means of a computer which transforms the first data into the second data, a router and / or multiple routers of a mesh network. For example, the computer can transmit the second data to a router, which in turn transmits it to the VR-capable end devices. All wireless transmission methods are possible. For example, routers with modern WLAN standards can be used, since the use of multiple VR-capable end devices generates high bandwidths. The use of a mesh network, for example, is also conceivable. In a mesh network, networked routers cooperate wirelessly and / or wired to enable transmission between the computer and the VR-capable end device.This would make it possible to ensure sufficiently fast transmission from the computer to a VR-capable device even with lower-performance routers. However, the second data can also be transmitted directly from the computer to the VR-capable device. It is also conceivable that the received second data could be exchanged, for example, between different VR-capable devices.

[0027] For example, transforming the first data can include setting up the images of the second data for an augmented reality or mixed reality application on the VR-capable devices. Augmented reality and mixed reality are technologies that overlay digital content into the real world by overlaying virtual objects or information over the physical environment. The VR-capable devices can be equipped with sensors such as cameras or depth sensors to capture the user’s real environment. The sensors can capture information about objects, structures, and dimensions in the real environment. This can be done, for example, through live capture of physical objects. Using the live capture function, the sensors on the VR-capable devices can capture physical objects in the real environment and use them to create 3D models, for example.The captured information can be combined in real time with the images from the second data. The sensors on the VR-capable devices can also enable tracking and positioning of the user within this environment. This enables precise placement and alignment of the virtual objects in relation to the physical environment and the user. The images from the second data can, for example, be projected into the user's real environment as a 3D model. Interaction options with the images from the second data can be achieved, for example, through gesture control or, in particular, handheld input devices.

[0028] For example, the method may further comprise receiving third data for adapting the second data from at least one of the VR-capable devices. Furthermore, the method may optionally comprise adapting the second data for the at least one VR-capable device from which the third data was received. The third data may, for example, be information about the technical specifications and performance of the VR-capable device, settings, or the like of a user based on their preferences. The third data may, for example, also be information about the user's current environment, such as lighting, room size, objects, structures, and other physical elements in the real world. However, the third data may also be interaction data that reflects the user's interactions with the VR-capable device and / or the second data.The third data can be provided by the VR-capable device and subsequently sent to the computer. The computer can analyze the received third data and use it as a basis for re-transforming and / or otherwise adapting the images of the second data. This can mean, for example, that properties and / or parameters of the second data can be changed to adapt them to user-specific settings of the VR-capable device.

[0029] For example, the third data can be based on an interaction between the user of the at least one VR-capable device and the at least partially virtual reality or, in other words, environment. The third data can, for example, reflect the user's interaction with the VR-capable device and the at least partially virtual reality or the projected images of the second data. This can, for example, be movements of the user during their interaction with the images of the second data displayed on the VR-capable device and / or the interaction with the displayed images of the second data themselves. This enables dynamic adaptation of the content of the projected images of the second data. For example, a user could have an increased interest in certain virtual objects, which they signal, for example, through gestures or control with a handheld input device.Based on this, the VR-capable device sends third data to the computer. After receiving this third data, the computer analyzes the third data and adjusts the second data accordingly. This could mean, for example, changing the position or properties of the images in the second data. For example, the projected second images could be enlarged or displayed on a different plane, or something similar.

[0030] For example, the method may comprise receiving fourth data for controlling the adaptation of the second data for the at least one VR-capable device from which the third data was received. In this context, this means that the fourth data is provided by another user, for example a teacher, in particular with control access to the computer. The fourth data may represent control data for enabling interaction with the second data, so that the users, in this example students, can interact with the VR-capable devices with the at least partially virtual reality. In a training scenario, this could mean that the teacher presents VR training material on the students' VR-capable devices in a teaching mode via a computer.In this teaching mode, students can move freely and view the VR training material from different positions, but cannot interact with the VR training material. This means that the VR training material is controlled by the computer in teaching mode. The teacher can now switch to an interactive mode in which control over the display of the VR training material is transferred to the VR-capable device. This switch can be made using the fourth data. By switching from teaching mode to interactive mode, it becomes possible for users to control, for example, the position, size, possible sections, and the showing and hiding of layers on the VR-capable devices, in this example. For example, controllers, handheld input devices, or hand tracking can be used to change the position or appearance of VR training materials, such as a 3D model.

[0031] For example, the method may comprise comparing the received third data and / or received fifth data based on the second data from at least one of the VR-capable devices with sixth data to verify a training objective of the user of the at least one VR-capable device. The fifth data may be additional information or context data based on the second data. The received third data and / or the received fifth data may be compared with the second data generated on the at least one VR-capable device. This comparison may comprise various aspects of the data, such as the accuracy, relevance, or agreement with the user's training objectives based on the sixth data. The sixth data may comprise information about the training objectives of the user of the at least one VR-capable device.These training objectives can be predefined and include, for example, learning objectives, competencies, performance standards, or similar that the user is expected to achieve. Based on the comparison of the data and the user's training objectives, a result can be determined, which can, in particular, have at least two or more evaluation results, for example, on a grading scale of 1 to 5 or 1 to 6. This result can, for example, indicate whether the received data supports the user's training objectives, whether additional measures are required, or whether the objectives have already been achieved.

[0032] The method may further comprise providing seventh data regarding a result of the comparison for transmission to the at least one VR-capable device. The seventh data may, for example, comprise information about the result of the comparison and be transmitted to the at least one VR-capable device. For example, such information may be a grade on the grading scale. This information may, for example, alternatively or additionally comprise hints, feedback, or recommendations to assist the user in achieving their training goals.

[0033] An example could be that a user interacts in at least partially virtual reality with elements that are set up as VR training material. This interaction can be sent to the computer in the form of third data and compared there with sixth data. The result of this comparison can be communicated to the user, for example, in the form of the seventh data. Furthermore, a representation of the interaction data or Third-party data from VR-enabled devices can be combined with VR training materials, for example, on a display board or dashboard. Displaying the interaction data, or third-party data, from VR-enabled devices with VR training materials on a dashboard can enable a clear and easy-to-understand visualization of user interactions with or in virtual reality. This can, for example, give teachers or administrators the opportunity to monitor user progress, their interactions, and their success in achieving training objectives in real time. A dashboard can contain various metrics and statistics to provide comprehensive insight into user performance and behavior during VR training.

[0034] According to a second aspect of the invention, a computer program product is provided containing instructions which, when executed by a computer, cause the computer to perform the method of the first aspect. The computer program product may be a computer program as such or a product, e.g., a computer-readable storage medium on which the computer program is stored.

[0035] According to a third aspect, a data processing device is provided, which comprises at least one means for executing the method. The means may be a computer, in particular the computer mentioned herein, which may, for example, comprise a processor and a computer-readable storage medium, which may, in particular, comprise or be the computer program product of the second aspect.

[0036] According to a fourth aspect, a system is provided that comprises a data processing device according to the third aspect and a plurality of VR-capable terminals. Commercially available VR glasses, for example, can be used for the VR-capable terminals. The number of VR-capable terminals in the system can be, for example, at least five, at least ten, at least twenty, or more. The VR-capable terminals can be configured for wireless communication with the data processing device, in particular connected or connectable, wherein the method according to the first aspect can be executed by the system or can be executed by the system. SHORT DESCRIPTION OF THE CHARACTERS

[0037] Exemplary embodiments of the disclosure are described below with reference to the following drawings. Figure 1 shows an example of a system for a method for training users using virtual reality (VR)-capable devices; Figures 2A - 2B show a method with method steps for providing and Transformation of first data and a method with method steps for processing second data on the VR-capable device; Figures 3 and 4 each show an example of a method for training users using VR-capable devices. DETAILED DESCRIPTION OF THE INVENTION

[0038] The figures are merely schematic representations and serve only to illustrate examples of the disclosure. Identical or equivalent elements are generally provided with the same reference numerals.

[0039] Figure 1 shows a system for a method 100, 200 (see Fig. 2) for training users 1 (see Figs. 3, 4) using virtual reality (VR)-capable end devices 20. In this example, the system comprises a computer 10, three VR-capable end devices 20, and a router 30. Depending on requirements, the number of VR-capable end devices 20 can vary and is flexibly scalable in order to train a single person or, in the context of a training course, for example, one or more students or school classes. In this example, the computer 10 carries out the method 100 for providing and transforming first data. The first data can, for example, represent various images for training user 1. The first data can be provided from various sources, including another computer, a video player, television, or the like.The first data in a first image format 301 are converted into second data in a second image format 302 during the transformation process (see Figs. 3, 4). This transformation process can be performed entirely or partially on the computer 10. Alternatively, individual transformation steps can also be performed on external devices such as a second computer or a server or streaming server.

[0040] The second data in the second image format 302 are now prepared for transmission to the VR-capable end devices 20. The computer 10 or the streaming server transmits the second data in this example to a router 30. The router 30 is configured to transmit the second data to the VR-capable end devices 20. Alternatively, the transmission of the second data can take place directly from the computer 10 to the VR-capable end devices 20, i.e., without the router 30. In this example, three VR-capable end devices 20 are shown as an example. Depending on the performance of the router 30, several VR-capable end devices 20 can be supported simultaneously, for example for one or more school classes with several students as users 1. For less powerful routers 30, the use of a mesh network can be considered in which wirelessly and / or wired networked routers 30 cooperate to enable efficient transmission between computer 10 and the VR-capable end devices 20. The router 30 shown here as an example distributes the second data to the VR-capable end devices 20. A VR-capable end device 20 can, for example, be VR glasses or a VR headset. In addition to VR headsets, other AR / VR devices such as AR glasses, smartphones with VR functionality, or AR-capable tablets can also be integrated into the system. The VR-capable end devices 20 receive the second data and process the second data for projection on their screens. The VR-capable end devices 20 can each be equipped with one or more sensors that can, for example, generate movement data of a user 1.This movement data can preferably be processed on the VR-capable terminal 20 and adjust a field of view of the user 1 in the at least partially virtual reality according to the movement data. Alternatively, the movement data can also be sent back to the router 30, which then transmits it to the computer 10. The computer 10 can, for example, process the movement data and adjust a field of view of the user 1 in the at least partially virtual reality according to the movement data. Alternatively, the adjustment of the field of view can take place on the VR-capable terminal 20 itself, wherein a field of view provided by the second data can be expanded and / or enlarged in such a way that the user 1 can change his or her field of view within the field of view through movement.

[0041] Figure 2A shows an example of a method 100 with method steps 110-160 for providing and transforming the first data. The method 100 can be carried out by a data processing device 40, which can have the computer 10, wherein the computer can comprise a computer program product 50. In a first step 110, first data are provided, wherein the first data comprises images for training the users 1 in the first image format 301. This first data can, for example, comprise teaching material in the form of images or a video, which is initially available in a common image format or video format such that it can be displayed as screen content on a screen of a non-VR-capable terminal device.

[0042] In a step 120 of method 100, a recording of this screen content is created. Screen recording can be used, for example, with screen capturing, in which all or parts of the screen are recorded. Another possibility is, for example, the use of a Spout protocol. With the Spout protocol, applications can exchange images or video with each other in real time without the images (of the video) having to be first recorded and saved.

[0043] In a step 130 of the method 100, the first data, which in particular comprise monoscopic images, can be transformed into stereoscopic images of second data to generate a stereoscopic impression on the VR-capable end devices 20. These stereoscopic images can be achieved, for example, by shifting certain elements in the images relative to one another. Depth maps can also be obtained from the first images of the first data using so-called depth map extraction. This information can then be used to generate stereoscopic images that create a virtual reality with the teaching material of the images for the left and right eye on the VR-capable end devices at least by having slightly different perspectives, thereby creating the impression of a 3D effect.These stereoscopic images of the second data can, for example, be set up to display a 3D model on the VR-capable end devices 20.

[0044] In a step 140 of the method 100, the first data can be adapted for projection onto at least one screen in the VR-capable terminal 20. This can include, for example, adjusting the resolution. Also, or alternatively, this can at least partially create or expand the virtual reality. The resulting second image format can be configured by these adaptation processes such that it can reproduce the predominant or substantially complete field of view 2 of a user 1 (see Figs. 3, 4) in an at least partially virtual reality.

[0045] In a step 150 of the method 100, the second data can be compressed in order to subsequently make it available for transmission to the VR-capable end devices 20 in a step 160. This transmission from the computer 10 to the VR-capable end devices 20 can be carried out, for example, via the aforementioned router 30. The entire process, from providing the first data to providing the second data, runs in particular with a latency of less than 2 seconds, in particular less than 1 second or less than 0.5 seconds. Furthermore, this process can run automatically.

[0046] In this example, computer 10 is used to create the first data, transmit the second data, compress it if necessary, and execute it as a streaming server if necessary. In another embodiment of the invention, two computers 10 can be used, for example. On a first of the two computers 10, for example, the screen content can be created and the first data can be transformed into second data. The second of the two computers 10 is then used, for example, to compress the second data and execute the streaming server.

[0047] Figure 2B shows an example of a method 200 with method steps 210 - 230 for Processing the second data on the VR-capable terminal 20. The VR-capable terminal 20 may, for example, have a processor and / or a graphics card that can execute the method 200.

[0048] In a first step 210 of the method 200, the second data is received on the VR-capable terminal 20. In a step 220 of the method 200, a distortion correction of the second data takes place, for example. Projecting a previously two-dimensional image onto a sphere on the VR-capable terminal 20 can, for example, result in distortions in the corner areas of the image. To prevent these corner areas from being projected onto the sphere, these corner areas are clipped by the distortion correction. UV mapping, for example, can be used for this purpose.

[0049] In a step 230 of the method 200, the second data are projected onto a sphere, in particular a sphere of a hemisphere, of a screen of the VR-capable terminal 20. Projecting the second data onto the hemisphere creates the impression of a panoramic view and / or a spatial perception for a user 1 with a VR-capable terminal 20. A field of view 2 of the user 1 can, for example, be between 60° and 140°. All steps can be performed automatically on the VR-capable terminal 20.

[0050] The sequence of the process steps in the two processes 100 and 200 is given here merely as an example and can also be performed in any other order. Furthermore, any individual or multiple process steps from processes 100 and 200 can optionally be omitted.

[0051] Figure 3 shows a schematic example of a method 100, 200 for training users 1 using VR-capable end devices 20, using the example of a teaching unit on the topic of orientation in the night sky. A computer 10 provides initial data, wherein the first data comprises images for training users 1 in the first image format 301. In this example, this is a night sky in a two-dimensional representation on a screen of computer 10. This initial data can be provided, for example, by computer 10, which can run astronomy software such as Stellarium, for example.

[0052] By means of the method 100 with the method steps 110 - 160 for providing and transforming the first data, the resulting second data are configured in a second image format 302 for displaying at least a partial virtual reality on the VR-capable terminal 20.

[0053] Specifically, for this example, this means that the initially two-dimensional night sky is captured by the screen of the computer 10 and processed in such a way that that the night sky now surrounds the field of view 2 of the user 1. By projecting the second data onto a hemisphere of a screen of the VR-capable terminal 20, the impression of a wide perspective is created that covers a predominant, or essentially complete, field of view 2 of the user 1.

[0054] Furthermore, in this example, by transforming initially monoscope images of the first data into stereoscopic images of the second data, a spatial impression or a 3D effect is achieved, so that the impression is created that the depicted planets have a volume and are located at different positions in space.

[0055] By changing the position of the VR-capable end devices 20 by the user 1, a shift of the current field of view 2 on the screen of the VR-capable end device 20 within the field of view 2 of the user 1 is also possible. For this purpose, movement data, such as inclinations, rotations, or similar movements of the VR-capable end device 20 and / or the user 1, can be processed by the computer 10. The computer then adjusts the field of view 2 of the user 1 according to the movement data.

[0056] Through the sum of the above transformations of the first data into the second data, a virtual reality or virtual environment is created on multiple VR-capable devices 20, providing an immersive experience. Users of the VR-capable devices 20 can experience virtual reality in real time and simultaneously, and thus also be trained simultaneously and in real time.

[0057] Figure 4 shows a schematic example of a method 100, 200 for training users 1 using VR-capable end devices 20, using the example of a teaching unit on the subject of biology. Data in the second image format 302, generated according to the method 100 and representing a human cell, can be seen. By interacting with the second data in the second image format 302, for example, using techniques such as gesture control or control by one or more controllers, the user 1 can, for example, change the position or appearance of the data in the second image format 302, in this case the human cell.

[0058] In the example shown in Fig. 4, user 1, for example, wishes to examine the area within the human cell (dashed lines) and signals this, for example, through gesture control. This interaction is sent in the form of third data to a computer 10 and processed by it. Alternatively, the third data can also be processed directly on the VR-capable device. Subsequently, an adapted model of the human cell is sent to the VR-capable device 20 of user 1. This makes it possible, for example, to determine the position, size, possible sections and the To show and hide levels on the VR-capable devices 20 by the user 1.

[0059] However, it is also conceivable that interaction with the data in the second image format 302 by a user 1 must first be authorized by another person, for example a teacher. In this example, this would mean that user 1, in a first mode, for example called teaching mode, can move freely and view the data in the second image format 302, i.e., the representation of the human cell, from different positions, but cannot interact with it. Control over the display of the data in the second image format 302 in this teaching mode therefore occurs from the computer 10. The teacher can now switch to a mode, for example called interactive mode, by activating fourth data, in which control over the display of the data in the second image format 302 is transferred to the VR-capable terminal 20. The fourth data can thus represent control data.In the interactive mode, changes to, for example, the position, the size, possible cuts and the showing and hiding of layers, as shown here using a human cell, can be made by user 1.

[0060] As used herein, the term "at least one" with respect to a list of one or more entities should be understood to mean at least one entity selected from among one or more of the entities in the list of entities, but does not necessarily include at least one of each entity specifically listed in the list of entities, and does not exclude any combination of entities in the list of entities. This definition also allows for the optional presence of entities other than those in the list of entities referred to by the term "at least one", whether or not related to the specifically named entities.Thus, by way of non-limiting example, “at least one of A and B” (or, equivalently, “at least one of A or B” or, equivalently, “at least one of A and / or B”) may refer in one example to at least one, optionally including more than one, A without B being present (and optionally including moieties other than B); in another example to at least one, optionally including more than one, B not being present (and optionally including moieties other than A); in yet another example to at least one, optionally including more than one, A and at least one, optionally including more than one, B (and optionally including moieties other than A). In other words, the terms “at least one,” “one or more,” and “and / or” are indefinite terms that can be used both conjunctive. can be used both singly and disjunctively. For example, each of the terms "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" can mean A alone, B alone, C alone, A and B together, A and C together, B and C together, A, B, and C together, and optionally, each of the above in combination with at least one other entity.

[0061] Other variations of the disclosed examples may be understood and practiced by one skilled in the art in practicing the claimed disclosure, given the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" does not exclude a plurality. A single processor or other unit may perform the functions of multiple elements or steps recited in the claims. The mere fact that certain steps are recited in dependent claims does not mean that a combination of those steps cannot be advantageous.A computer program may be stored / distributed on a suitable medium, such as an optical storage medium, supplied with or as part of other hardware, but it may also be distributed in other forms, for example, via the Internet or other wired or wireless communication systems. Reference to elements or method steps as first, second, third, further, etc., is merely exemplary in their order and is not limiting. Any reference signs in the claims are not to be construed as limiting the scope of the claims.

[0062] REFERENCE SYMBOL 1 user 2 Field of vision of a user 10 computers 20 VR-capable devices 30 routers 40 Data processing device 50 computer program product 100 procedures on the computer 110 - 160 procedural steps 200 procedures on the VR-capable device 210 - 230 process steps 301 first image format 302 second image format

Claims

CLAIMS 1. A method (100, 200) for training users (1) using virtual reality (VR)-capable terminals (20), the method (100, 200) comprising: - providing first data, wherein the first data comprises images for training the users (1) in a first image format (301), - transforming the first data into second data such that the second data comprise the images of the first data in a second image format (302), wherein the second image format is configured for a representation of at least a partial virtual reality on the VR-capable terminals (20), and - Providing the second data for transmission to the VR-capable end devices (20).

2. The method (100, 200) according to claim 1, wherein the images of the first data in the first image format (301) are configured for display on a non-VR capable terminal device.

3. The method (100, 200) according to claim 1 or 2, wherein the second image format for a projection of the images of the second data (302) is configured such that it corresponds to a field of view (2) on the VR-capable terminals (20) that is expanded and / or enlarged compared to the first image format.

4. The method (100, 200) according to claim 3, wherein the second image format is configured to project the images of the second data (302) with a fisheye field of view on the VR-capable terminals (20).

5. The method (100, 200) according to claim 3 or 4, wherein the second image format is configured to enable a shift of a current field of view (2) on a screen of the VR-capable terminal (20) in the field of view (2) by changing a position of the VR-capable terminals (20).

6. The method (100, 200) according to claim 5, wherein the change in a position of the VR-capable terminals (20) and the displacement of the current field of view (2) on a screen of the VR-capable terminal (20) in the field of view (2) are processed by the VR-capable terminals (20).

7. The method (100, 200) according to any one of the preceding claims, wherein the method (100, 200) further comprises receiving the second data on the VR-capable terminals (20) and projecting the images of the second data (302) onto a sphere, in particular a sphere of a hemisphere, of screens of the VR-capable terminals (20).

8. The method (100, 200) of claim 7, wherein distortion correction is performed when projecting the images of the second data (302) onto the sphere.

9. The method (100, 200) according to any one of the preceding claims, wherein providing the first data comprises a screen capture or a live streaming of the images of the first data.

10. The method (100, 200) according to any one of the preceding claims, wherein a latency of less than 2 seconds occurs between the provision of the first data and the second data.

11. The method (100, 200) according to any one of the preceding claims, wherein the method (100, 200) further comprises compressing the second data, wherein the compressed second data is provided for transmission to the VR-capable terminals (20).

12. The method (100, 200) according to any one of the preceding claims, wherein the method (100, 200) further comprises transmitting the second data to the VR-capable terminals (20), wherein the transmission is carried out by means of a computer (10) which transforms the first data into the second data, a router (30) and / or a plurality of routers (30) of a mesh network.

13. The method (100, 200) according to any one of the preceding claims, wherein transforming the first data comprises setting up the images of the second data (302) for an augmented reality or mixed reality application on the VR-capable terminals (20).

14. The method (100, 200) according to any one of the preceding claims, wherein the method (100, 200) further comprises: - receiving third data for adapting the second data from at least one of the VR-capable terminals (20), and - adapting the second data for the at least one VR-capable terminal (20) from which the third data was received.

15. The method (100, 200) according to claim 14, wherein the third data is based on an interaction between the user (1) of the at least one VR-capable terminal (20) with the at least partially virtual reality.

16. The method (100, 200) according to claim 14 or 15, wherein the method (100, 200) comprises: - Receiving fourth data for controlling the adaptation of the second data for the at least one VR-capable terminal (20) from which the third data was received.

17. The method (100, 200) according to any one of claims 14 to 16, wherein the method (100, 200) comprises: - comparing the received third data and / or received fifth data based on the second data from at least one of the VR-capable terminals (20) with sixth data to check a training objective of the user (1) of the at least one VR-capable terminal (20), and - Providing seventh data relating to a result of the comparison for transmission to the at least one VR-capable terminal device (20).

18. A computer program product (50) or computer-readable storage medium comprising instructions which, when executed by a computer (10), cause the computer (10) to carry out the method (100, 200) according to any one of claims 1 to 17.

19. A data processing device (40) comprising at least one means for carrying out the method (100, 200) according to one of the preceding claims.

20. System comprising a data processing device (40) according to claim 19 and a plurality of VR-capable terminals (20).

Images