DEVICE FOR REAL-TIME SYNCHRONIZATION BETWEEN A PHYSICAL UNIT AND A MIXED-REALITY ENVIRONMENT

Abstract

A device comprises a physical unit operationally connected to a multitude of sensors. A microcontroller is configured to map the respective baseline positions of the sensors. A virtual reality (VR) controller is designed to generate a digital twin of the physical unit. The microcontroller receives sensor data from the multitude of sensors and converts this data into a corresponding position value. The microcontroller is able to determine, in a predefined priority order, whether each position value deviates from its respective baseline position. The VR controller generates an updated digital twin by updating a corresponding virtual position value within the digital twin based on a unique letter code.The updated digital twin is transferred to a virtual reality medium or a mixed reality medium, with the operation of the physical unit being controlled in real time partly based on the updated digital twin.

Description

INTRODUCTION

[0001] This description refers generally to real-time synchronization between a physical unit and a mixed-reality environment. More specifically, it refers to a device and a corresponding method for controlling a physical unit based on real-time synchronization between the physical unit and a mixed-reality environment. A mixed-reality environment integrates real physical properties into a virtual overlay. Physical properties generally refer to the real-world attributes of an object. Mechanisms for integrating real physical properties into a virtual overlay generally use optical detectors to track these properties. However, the data from these optical detectors are dependent on environmental conditions and may be inaccurate under certain circumstances.

[0002] KR 10 2019 0 115 810 A shows a cultivator simulation system with a cultivator model to which a first and a second model tracker are attached at different positions. A tracker acquisition unit monitors the positions of both trackers and generates corresponding position information. A control unit creates a three-dimensional virtual cultivator model, controls its movement based on operating data from the simulation device, and compares the movement of the 3D model with the actual movement of the physical cultivator model using the tracker position information provided by the tracker acquisition unit.

[0003] DE 10 2023 113 630 A1 discloses a method and system for generating an enhanced machine view, in which a dynamic virtual machine model of the real machine is superimposed with camera images from its environment and synchronized with them in time. The system comprises cameras mounted on the machine as well as machine sensors that record the position and movements of the machine components.

[0004] CN 1 16 946 876 A shows a real-time monitoring system for a large metallurgical casting crane, comprising a sensor system, a visual system, a control system, a data processing system, and a display and VR unit. The inputs of the sensor and visual systems are connected to the crane, and their outputs are connected to the control system. This system feeds the data processing system, whose output is in turn connected to the display and VR unit.

[0005] DE 10 2005 061 211 A1 discloses a method for generating a human-machine user interface in which a user selects an industrial device or process by spatially moving a pointer, and the pointer position is tracked. The identity of the selected device is determined based on the user's position and head position by comparing it with predefined device or process positions and with control data. Based on this comparison, a stored virtual image is retrieved, superimposed with a real image of the device or process, and displayed on a screen. Furthermore, a virtual control interface in the form of a virtual control console is provided, with which the user interacts via the pointer to control the selected device or process. DESCRIPTION

[0006] The object of the invention is to overcome the aforementioned disadvantages. This object is achieved by the subject matter according to claim 1. Further developments are described in the dependent claims.

[0007] This document describes a device comprising a physical unit and a plurality of sensors operationally connected to the physical unit. A microcontroller is integrated into the physical unit and includes a first processor and a first tangible, non-volatile memory. The microcontroller is configured to map a respective baseline position of the plurality of sensors. A virtual reality (VR) controller is connected to the microcontroller and includes a second processor and a second tangible, non-volatile memory, the VR controller being designed to generate a digital twin of the physical unit.

[0008] The microcontroller is configured to receive sensor data from numerous sensors, converting this data into a corresponding position value. The microcontroller is configured to determine, in a predefined priority order, whether a given position value has deviated from the object's baseline position. If a position value has deviated, a data packet containing a unique letter code is generated. The VR controller can receive incoming data from the microcontroller and forward it, among other factors, based on this unique letter code. The updated digital twin is then transferred to a virtual reality or mixed reality medium, allowing the physical unit's operation to be partially controlled in real time based on this updated digital twin.

[0009] The unique letter code can contain a first letter encoding an axis of movement of the object and a second letter encoding an identity of the object, with the data packet further including a current numerical position coordinate of the object. Converting the sensor data into a corresponding position value involves transmitting the sensor data to an analog-to-digital converter for conversion into a corresponding bit value. The sensor data is sorted in a predefined priority order. The respective bit value is converted into the respective position value. The respective bit value depends on the output voltage of the multiple sensors.

[0010] In some implementations, the predefined priority order prioritizes the respective elements within a user's predefined field of view relative to the respective elements outside that field of view. The VR controller can be adapted to create a first thread and a second thread via a multithreading library, so that the first thread processes incoming data and the second thread operates a user interface of the VR controller.

[0011] The device may include a head-mounted display that communicates with the VR controller, transmitting and displaying the updated digital twin on the head-mounted display. The device may also include a monitor that communicates with the VR controller, transmitting and displaying the updated digital twin on the monitor. The device may include a Universal Serial Bus (USB) port adapted to connect to the microcontroller and the VR controller, with the VR controller adapted to receive incoming data from the USB port. The sensor array may include an accelerometer and / or a gyroscope. The sensor array includes at least one sensor positioned along each of the first, second, and third orthogonal axes.

[0012] The physical unit can comprise a seat assembly and a steering wheel assembly. The plurality of sensors can include rotary motion sensors in the steering wheel assembly, each suitable for indicating the tilt of the steering column, the telescoping of the steering column, and the rotation of the steering wheel. The plurality of sensors can include rotary motion sensors in the seat assembly, each suitable for indicating the angular adjustment of the seat back, the longitudinal adjustment of the seat, and the seat height adjustment. In some embodiments, the physical unit comprises a foot pedal assembly. The plurality of sensors can include linear actuators in the foot pedal assembly, the linear actuators being adapted to indicate an angular adjustment and a longitudinal adjustment.

[0013] This document describes a method for controlling the operation of a physical unit within a device comprising a microcontroller with a first processor and a first tangible, non-volatile memory, and a virtual reality (VR) controller with a second processor and a second tangible, non-volatile memory. The method includes connecting a multitude of sensors to the physical unit. It also includes integrating the microcontroller with the physical unit and mapping the baseline position of each sensor to the microcontroller. Finally, the method includes creating a digital twin of the physical unit via the VR controller and converting sensor data from the multitude of sensors into corresponding position values ​​via the microcontroller.

[0014] The process involves determining, in a predefined priority order, whether the respective position value of an object has deviated from its base position, using the microcontroller. The process includes generating a data packet with a unique letter code via the microcontroller whenever the position value has deviated from its base position. The process also includes receiving incoming position data from the microcontroller and forwarding this data, partially based on the unique letter code, via the VR controller. Finally, the process generates an updated digital twin by updating a corresponding virtual position value within the digital twin, based on the unique letter code, via the VR controller.The process involves transferring the updated digital twin to a virtual reality medium or a mixed reality medium and controlling the operation of the physical unit in real time, partly based on the updated digital twin, via the VR control.

[0015] A vehicle with a physical unit and a multitude of sensors operationally connected to the physical unit is presented here. A microcontroller is integrated into the physical unit and comprises a first processor and a first tangible, non-volatile memory. The microcontroller is configured to map a respective baseline position of the multitude of sensors. A virtual reality (VR) controller is connected to the microcontroller and has a second processor and a second tangible, non-volatile memory, with the VR controller being designed to generate a digital twin of the physical unit.

[0016] The microcontroller is configured to receive sensor data from a multitude of sensors, converting the sensor data into a corresponding position value. The microcontroller is configured to determine, in a predefined priority order, whether the respective position value has deviated from the object's base position. If so, it creates a data packet containing a unique letter code. This unique letter code encodes the object's axis of movement and its identity, with each data packet containing the object's numerical position coordinate. The VR controller is capable of receiving incoming data from the microcontroller and forwarding it, among other things, based on this unique letter code.The updated digital twin is transferred to a virtual reality medium or a mixed reality medium, with the operation of the physical unit being controlled in real time partly based on the updated digital twin.

[0017] The above features and advantages, as well as other features and advantages of the present description, are readily apparent from the following detailed description of the best ways of carrying out the disclosure when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a schematic, fragmentary representation of a device with a physical unit, a microcontroller, and a VR controller; Fig. 2 is a flowchart for a first procedure performed by the microcontroller of Fig. 1 can be executed; Fig. 3 is a flowchart for a second procedure, which is initiated by the VR controller of Fig. 1 can be executed; and Fig. Figure 4 is a schematic representation of an example of a modular architecture found in the device of Fig. 1 can be used.

[0018] Representative embodiments of this description are shown as non-limiting examples in the drawings and are described in more detail below. However, it should be understood that the new aspects of this description are not limited to the specific forms shown in the drawings listed above. Rather, the description is intended to cover modifications, equivalents, combinations, sub-combinations, permutations, groupings, and alternatives that fall within the scope of this description, such as those encompassed by the attached claims. DETAILED DESCRIPTION

[0019] Referring to the drawings, in which identical reference numbers refer to identical components, shows Fig. Figure 1 schematically shows a device 10 for controlling the operation of a physical unit 12. The physical unit 12 is operationally connected to a plurality of sensors S, each positioned at predefined reference points. In other words, the plurality of sensors S are positioned along several different axes of the physical unit 12 so that coordinates and positions can be tracked. As described below, these coordinates are then used to align a specific virtual target to reflect the physical unit 12. A priority order is assigned to the sensors S to determine which virtual targets should be updated first.

[0020] Mechanisms for incorporating real-world physical properties into a virtual overlay generally use optical trackers to detect these properties. However, the data from these optical trackers is dependent on the environment and can be obscured under various circumstances. With Device 10, optical trackers are no longer required. Since no optical positioning is needed, errors caused by the environment, such as reflections, sunlight, darkness, and shading, are avoided.

[0021] Device 10 can be used in various ways, including tracking and synchronizing movements in virtual reality. Device 10 can be used in design visualization, for example, to visualize how a virtual product would fit into a physical environment. Device 10 can be used in training simulations, for example, to practice tasks on virtual objects.

[0022] The device 10 can be used in industry, for example, to overlay digital instructions onto real machines or to visualize hidden components. For instance, in some cases, a user U may not be able to see a component they are installing because it may be obscured by another, larger component. The user U can use the head-mounted display 56 (virtual reality or mixed reality) to project the position of the component behind the obstruction.

[0023] In the Fig. In the example shown, the physical unit 12 is a reconfigurable seat assembly 14 comprising a seat assembly 16, a steering wheel assembly 18, and a foot pedal assembly 20, all located on a base structure 22. It is understood, however, that the physical unit 12 can take other forms and include other components not shown. The reconfigurable seat assembly 14 can be used for interior and visibility assessments for a vehicle.

[0024] Referring to Fig. 1. The plurality of sensors S can include the first, second, and third sensors S1, S2, S3 in the steering wheel assembly 18. The first, second, and third sensors S1, S2, S3 are angle and rotation sensors, or rotary motion sensors, which can indicate the rotation of the steering wheel, the tilt of the steering column, and the telescoping of the steering column. As those skilled in the art know, rotary motion sensors (the term is used interchangeably with angle and rotation sensors) measure the rotational motion of objects to determine their angular displacement, velocity, and direction of rotation. The plurality of sensors S can include the fourth and fifth sensors S4, S5 in the foot pedal assembly 20. The fourth and fifth sensors S4, S5 are linear actuators with corresponding position feedback mechanisms for angular and longitudinal adjustment. The plurality of sensors S can include the sixth, seventh, and eighth sensors S6, S7, S8 in the seat assembly 16.The sixth, seventh and eighth sensors S6, S7, S8 are rotary motion sensors that indicate the angle adjustment of the backrest, the longitudinal adjustment of the seat and the height adjustment of the seat.

[0025] As in Fig. As shown in Figure 1, the device 10 comprises a microcontroller 30 which is hardwired or directly connected to the physical unit 12 or otherwise integrated.

[0026] The microcontroller 30 has a first processor 32 and a first tangible, non-volatile memory 34 in which instructions for the selective execution of the procedure 100 are recorded, as in Fig. Figure 2 shows that memory 34 can store control-executable instruction sets, and processor 32 can execute the control-executable instruction sets stored in memory 34. Microcontroller 30 is configured to map the respective baseline positions for the plurality of sensors S.

[0027] As in Fig. As shown in Figure 1, the device 10 includes a virtual reality controller 40, referred to here as the VR controller 40, which is electronically connected to the microcontroller 30. The VR controller 40 has a second processor 42 and a second tangible, non-volatile memory 44. The memory 44 can store instruction sets executable by the controller, and the processor 42 can execute the instruction sets stored in the memory 44. The VR controller generates a digital twin 50 of the physical entity based on a baseline position of the reference points. The digital twin 50 is a surrogate of the physical entity 12.

[0028] The sensor data is used to update the digital twin 50. The VR controller 40 generates an updated digital twin by updating a corresponding virtual position value within the digital twin based on the unique letter code. In other words, if a part of the physical unit 12 moves in the real world, the corresponding virtual part also moves in the virtual world. The updated digital twin 52 is then transmitted to a virtual reality or mixed reality medium. The VR controller 40 uses a specific format of commands via a communication protocol to relay information from the physical unit 12 to a virtual target or digital twin 50, aligning the two when using a virtual reality and / or mixed reality medium to enable a seamless user experience.

[0029] In one example, the device 10 can include a screen or monitor 54 that communicates with the VR controller. Here, the updated digital twin 52 is transmitted to and displayed on the monitor 54 for viewing by the user U. In another example, the device 10 can include a head-worn display 56 that communicates with the VR controller 40. The updated digital twin 52 is transmitted to the head-worn display 56 and displayed on it for the user U. The VR controller 40 is designed to partially control the operation of the physical unit 12 based on the updated digital twin 52.

[0030] An exemplary overall architecture 200 for the device 10 is in Fig. 4 shown. Fig. Figure 4 shows a physical environment 210 with a physical unit 212 and a virtual environment 230 with a virtual unit 240. Referring to Fig. In the physical unit 212, a variety of sensors 214 (e.g., a potentiometer, an encoder) are attached to the physical unit 212 to receive sensor data, which is sent to an input processor 216 (e.g., Arduino, Raspberry Pi, etc.) for processing. As mentioned earlier, the sensors S are assigned a priority ranking. This priority ranking allows, for example, updates to virtual targets within a user's field of view U to be prioritized over updates outside the user's field of view.

[0031] As in Fig. As shown in Figure 4, the input processor 216 transfers the data to the first software processor 218 (e.g., C, C++, Python, etc.) and then to a first communication interface 220 (e.g., serial port 60). The first communication interface 220 transfers the data to a second communication interface 232 (e.g., serial port 60) in the virtual environment 230. As shown in Fig. As shown in Figure 4, the second communication interface 232 transmits the data to a virtual reality processor 234 (e.g., VRED, Unreal Engine, etc.), which creates a simulated, computer-generated environment. Using the unique letter code (encoded in the data packet) to separate the incoming data, valid data is forwarded to the corresponding virtual models, their positions being updated with the position contained in the data packet. The data is then forwarded to a second software processor 236 (e.g., for format conversion) before being transmitted to a head-mounted virtual reality display 238 to display the virtual unit 240.

[0032] In Fig. Figure 2 shows an example flowchart of the first procedure 100. Procedure 100 can be represented as computer-readable code or instructions. Procedure 100 does not have to be applied in the order shown here. Furthermore, some steps can be omitted. Procedure 100 can be executed in real time, continuously, systematically, sporadically, and / or at regular intervals.

[0033] Procedure 100 begins in block 102 of Fig. 2, where the microcontroller 30 is programmed to receive sensor data from the sensor with the highest priority. In the example shown, the priority order denotes the sensor S1 with the highest priority. In one embodiment, there is at least one sensor (from the plurality of sensors S) arranged along the first, second, and third orthogonal axes XYZ. The sensors S can include at least one accelerometer and / or a gyroscope for detecting position and orientation.

[0034] In block 104, the sensor data from sensor S1 is sent to an analog-to-digital converter to convert the sensor voltage into a bit value in a format that the microcontroller 30 can identify. The specific bit value depends on the output voltage of the multiple sensors S. In this case, the bit value depends on the output voltage of sensor S1. Immediately after the voltage-to-bit-value conversion, the bit value is converted into a current position value, e.g., in millimeters.

[0035] In the further course of block 106, procedure 100 determines whether the current position value has deviated from the respective base position value of an element. In other words, once the current position value has been determined, the microcontroller 30 searches for the positions that have changed in the predefined priority order. The priority order can, for example, begin with the positions that are directly within the user's field of vision.

[0036] If the current position value has deviated (Block 106=YES), the process 100 proceeds to Block 108, where a data packet is created containing a unique code of at least two letters that describes which object has moved, along with the current numerical position coordinate of the moving object. The unique letter code includes a first letter that encodes the object's axis of movement and a second letter that encodes the object's identity. The data packet also contains the object's current numerical position coordinate. From Block 108, the data packet is then sent to Block 110 at the VR controller 40. The data packet can be transmitted to the VR controller 40 via a Universal Serial Bus (USB) port 60, which is suitable for connecting to the microcontroller 30 and the VR controller 40.The data packet can be transmitted to the VR controller 40 via a wired communication method using a protocol available to a person skilled in the art, e.g., a wireless network 46. A first iteration of the procedure 100 is completed, and the procedure 100 can be executed again in the next iteration.

[0037] The wireless network 46 can be a short-range or long-range network. The wireless network 46 can be a communication bus, which may take the form of a serial Controller Area Network (CAN bus). The wireless network 46 can be a serial communication bus in the form of a local area network. The local area network may include, but is not limited to, a Controller Area Network (CAN), a Controller Area Network with Flexible Data Rate (CAN-FD), Ethernet, Bluetooth, Wi-Fi, and other data forms. The wireless network 46 can be a wireless local area network (LAN) that connects multiple devices via a wireless distribution method, or a wireless metropolitan area network (MAN) that connects multiple wireless LANs. Other types of network technologies or communication protocols available to the person skilled in the art may be used.

[0038] If there is no change in position (block 106=NO), procedure 100 proceeds to block 112 to obtain sensor data from the next sensor (e.g., sensor S2) in the predefined priority order. In other words, if the position of the most important elements has not changed, the lower-priority elements are checked to see if any movement has occurred, and if so, the data is sent in the same way. Procedure 100 then proceeds from block 112 to block 114. Blocks 104, 114, and 124 are similar. In block 114, the sensor data is transferred to an analog-to-digital converter (which may be integrated into the microcontroller 30) to convert the sensor voltage into a corresponding bit value. The bit value is then converted into a current position value, e.g., in millimeters.

[0039] Blocks 106, 116, and 126 are similar. Continuing at block 116, procedure 100 involves scanning for position changes relative to the respective base position value via microcontroller 30. Blocks 108, 118, and 128 are similar. If the current position value has deviated (block 116 = YES), procedure 100 proceeds to block 118, where a data packet is created containing a unique code of at least two letters that describes which element has moved, along with the current numerical position value. From block 118, the data packet is forwarded to block 110 and sent to VR controller 40. If there is no change in position (block 116=NO), procedure 100 continues with block 122 to obtain sensor data from the next sensor (e.g., sensor S3) in the predefined priority order, and proceeds in the same way (to blocks 124, 126, 128) as described above.

[0040] In Fig. Figure 3 shows an example flowchart of the second procedure 150. Procedure 150 can be executed as computer-readable code or instructions stored on the VR controller 40 and at least partially executable by it. Procedure 150 does not have to be applied in the order shown here. Furthermore, some steps can be omitted. Procedure 150 can be executed in real time, continuously, systematically, sporadically, and / or at regular intervals.

[0041] In block 152 of Fig. In block 3, the VR controller 40 receives incoming data from the microcontroller 30, including a serial port number. In block 154, the VR controller 40 determines whether the port number matches the predefined system requirements. If the port number does not match (block 154=NO), procedure 150 terminates (and the process starts again).

[0042] If the connection number matches (Block 154=YES), Method 150 proceeds to Block 156, where a serial Python library is called to process the input data. Method 150 then moves from Block 156 to Block 158. In Block 158, the validity of the input data is determined by checking the unique letter code. The input data is considered valid if the letters in the letter code have a corresponding match in the VR software. If the input data is invalid (Block 158=NO), Method 150 terminates. If the input data is valid (Block 158=YES), Method 150 proceeds to Block 160, where a multithreading library is called to process the data individually.The multithreading library creates two threads – a first thread for processing the received data and a second thread for operating the user interface 62 or the frontend of the running VR software. Additionally, the data is decoded in block 160 and sent to different categories.

[0043] Moving from block 160 to block 162, the decoded data is assigned to the various elements. This can be done via a compatible application programming interface (API) that allows the Python script to communicate with the virtual reality software in the VR controller 40. Moving from block 162 to block 164, the VR controller 40 asks the user U (e.g., via user interface 62) whether the data should be used in a head-mounted display 56. If no (block 164=NO), procedure 150 ends. If yes (block 164=YES), procedure 150 continues with block 166 to ask, via user interface 62, whether the user U wants a virtual reality platform. User interface 62 allows the selection of a virtual reality or mixed reality platform and assigns the data to the platform selected by the user U.

[0044] A mixed-reality environment allows a user to manipulate virtual elements as if they were physically present by integrating physical properties of the real world into the virtual overlay. In the context used here, user U can see their own body while interacting with the virtual world in a mixed-reality environment. For example, user U can steer the steering wheel with their own hands. In contrast, user U does not see their own body during interaction in a virtual-reality experience.

[0045] If user U selects a virtual reality platform (Block 166=YES), procedure 150 continues to Block 168 to set up a virtual reality experience. If not (Block 166=NO), procedure 150 proceeds to Block 170 to ask if user U wants a mixed reality platform. If not (Block 170=NO), procedure 150 ends. If user U selects a mixed reality platform (Block 170=YES), procedure 150 continues to Block 172 to set up a mixed reality experience.

[0046] In summary, the device 10 enables the tracking of a real-world object or physical unit 12 using a multitude of sensors S. The sensors S scan the environment to map the physical space and track the movements of the physical unit 12. A priority rank is assigned to the sensors S to determine which virtual reference points or targets in the digital twin 50 should be updated first. When a change is detected at the highest-ranked sensor, a data packet is generated. If no change is detected, the microcontroller 30 moves down the priority list until a position change is detected. The positions of each of the multitude of sensors S are labeled with a unique letter code (e.g., two letters) to identify the object being moved. The unique letter code and the position data can be formatted as a string.After labeling, the positions can be sent to the VR controller 40 via a communication protocol (serial, CAN, etc.), either via a wired connection or a wireless network.

[0047] The VR Controller 40 (e.g., via a custom Python script) receives the incoming position data and uses the assigned unique letter code and the position data to decode which part of the real-world object or physical unit 12 is moving, as well as the current position occupied by the physical unit 12. Once the VR Controller 40 (e.g., via the Python script) has this information, virtual reality software with a compatible application programming interface (API) can be used to update a digital twin of the physical unit 12 with the correct position / orientation. Using the updated digital twin, a user U can remotely visualize and evaluate the position / orientation of the physical unit 12 and modify or control the physical unit 12 accordingly.

[0048] In some embodiments, the physical unit 12 can be part of a vehicle V, which may include, among others, a passenger car, a sport utility vehicle, a light truck, a heavy truck, a minivan, a bus, a transit vehicle, a bicycle, a self-driving robot, agricultural equipment (e.g., a tractor), sports equipment (e.g., a golf cart), a train, or any other moving platform. The terms "dynamic" and "dynamic" as used here describe steps or processes that are executed in real time and are characterized by the fact that the states of parameters are monitored or otherwise determined, and the states of the parameters are updated regularly or periodically during the execution of a routine or between iterations of the routine's execution.

[0049] The Microcontroller 30 and the VR Controller 40 from Fig.1. Each contains a computer-readable medium (also called a processor-readable medium), including a non-transitory (e.g., tangible) medium involved in providing data (e.g., instructions) that can be read by a computer (e.g., by a computer's processor). Such a medium can take many forms, including, but not limited to, non-volatile and volatile media. Non-volatile media include, for example, optical or magnetic disks and other permanent storage devices. Volatile media can include, for example, dynamic random-access memory (DRAM), which can represent main memory. Such instructions can be transmitted over one or more transmission media, such as coaxial cables, copper wire, and fiber optic cables, including the wires that form a system bus connected to a computer's processor.Some forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, another magnetic medium, a CD-ROM, a DVD, another optical medium, a physical medium with hole patterns, a RAM, a PROM, an EPROM, a FLASH EEPROM, another memory chip or memory cartridge, or another medium from which a computer can read.

[0050] Lookup tables, databases, data stores, or other data repositories described here can encompass various mechanisms for storing, accessing, and querying different types of data, including a hierarchical database, a set of files in a file system, an application database in a proprietary format, a relational database management system (RDBMS), and so on. Each of these data repositories can be contained within a computer device running a computer operating system such as one of those mentioned above, and can be accessed over a network in one or more of the various ways. A file system can be accessed from a computer operating system and can contain files stored in various formats.An RDBMS can use the Structured Query Language (SQL) in addition to a language for creating, storing, editing and executing stored procedures, such as the PL / SQL language mentioned above.

[0051] The flowcharts illustrate the architecture, functionality, and operation of possible implementations of systems, procedures, and computer program products according to various embodiments of the present description. In this respect, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code comprising one or more executable instructions for implementing the specified logical function(s). It should also be noted that each block in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, can be implemented by specific hardware-based systems that perform the specified functions or actions, or by combinations of specific hardware and computer instructions.These computer program instructions can also be stored in a computer-readable medium that can instruct a control unit or other programmable data processing device to function in a certain manner, so that the instructions stored in the computer-readable medium produce a manufactured item containing instructions for implementing the function / action specified in the flowchart and / or block diagram blocks.

[0052] The numerical values ​​of parameters (e.g., of quantities or conditions) in this specification, including the appended claims, are to be understood as being modified by the term "approximately," regardless of whether "approximately" actually precedes the numerical value. "Approximately" means that the stated numerical value permits a certain degree of inaccuracy (with some approximation to the accuracy of the value; approximately or reasonably close to the value; almost). If the inaccuracy indicated by "approximately" is not otherwise understood in the art with this ordinary meaning, then, as used here, "approximately" at least indicates deviations that may arise from ordinary methods of measurement and use of such parameters. Furthermore, the description of ranges includes the description of each value and further subdivided ranges within the overall range.Each value within a range and the endpoints of a range are hereby disclosed as separate embodiments.

[0053] The detailed description and the drawings or FIGS. are supporting and descriptive of the disclosure, but the scope of the disclosure is defined exclusively by the claims. While some of the best modes and other embodiments for carrying out the claimed description have been described in detail, various alternative designs and embodiments exist for carrying out the disclosure defined in the appended claims. Furthermore, the embodiments shown in the drawings or the features of other embodiments mentioned in this description are not necessarily to be understood as independent embodiments.Rather, it is possible that each of the features described in one of the exemplary embodiments can be combined with one or a multitude of other desired features of other embodiments, leading to other embodiments that are not described in words or by reference to the drawings. Accordingly, such other embodiments fall within the scope of the appended claims.

Claims

[1] A device (10) comprising the following: a physical unit (12) and a plurality of sensors (S) that are operationally connected to the physical unit (12); a microcontroller (30) integrated into the physical unit (12) comprising a first processor (32) and a first tangible non-volatile memory (34), wherein the microcontroller (30) is configured to map a respective baseline position of the plurality of sensors (S); a virtual reality (VR) controller (40) that communicates with the microcontroller (30) and includes a second processor (42) and a second tangible non-volatile memory (44), wherein the virtual reality (VR) controller (40) is designed to create a digital twin (50) of the physical unit (12); wherein the microcontroller (30) is configured to receive sensor data from the plurality of sensors (S), the sensor data being converted into a respective position value; wherein the microcontroller (30) is configured to determine, in a predefined priority order, whether the respective position value has deviated from the respective base position of an object, and generates a respective data packet with a unique letter code if the respective position value has deviated; the controller (40) is adapted for virtual reality (VR) to: to receive incoming data from the microcontroller (30) and to forward some of the incoming data based on the unique letter code; to generate an updated digital twin (52) by updating a respective virtual position value in the digital twin (50) based on the unique letter code; and to transfer the updated digital twin (52) to a virtual reality medium or a mixed reality medium and to control the operation of the physical unit (12) in real time partly based on the updated digital twin (52), wherein the unique letter code contains a first letter that encodes an axis of movement of the object, and a second letter that encodes an identity of the object, wherein the data packet further contains a current numerical position coordinate of the object, where the predefined priority order prioritizes the respective elements in a predefined field of view of a user relative to the respective elements that are not in the predefined field of view. [2] Device (10) according to claim 1, comprising the conversion of the sensor data into a respective position value: Transmitting the sensor data to an analog-to-digital converter for conversion into a corresponding bit value, sorting the sensor data in the predefined priority order; and Conversion of the respective bit value into the respective position value, whereby the respective bit value depends on a respective output voltage of the plurality of sensors (S). [3] Device (10) according to claim 1, wherein the controller (40) is adapted for virtual reality (VR) to: to create a first thread and a second thread via a multithreading library, such that the first thread processes the incoming data and the second thread operates a user interface (62) of the controller (40) for virtual reality (VR). [4] Device (10) according to claim 1, further comprising: a head-mounted display (56) that communicates with the virtual reality (VR) controller (40), the updated digital twin (52) is transferred to the display (56) attached to the head and displayed there. [5] Device (10) according to claim 3, further comprising: a monitor (54) that communicates with the controller (40) for virtual reality (VR), the updated digital twin (52) is transferred to the monitor (54) and displayed on it. [6] Device (10) according to claim 3, further comprising: a USB port (60) adapted to be connected to the microcontroller (30) and the virtual reality (VR) controller (40), wherein the virtual reality (VR) controller (40) is adapted to be able to receive the incoming data from the USB port (60). [7] Device (10) according to claim 1, wherein the plurality of sensors (S) comprises an accelerometer and / or a gyroscope, wherein the plurality of sensors (S) comprises at least one sensor which is arranged along a first, second and third orthogonal axis. [8] Device (10) according to claim 1, wherein the physical unit (12) comprises a seat assembly (14) and a steering wheel assembly (16), wherein the plurality of sensors (S) comprise rotary motion sensors in the steering wheel assembly (16), each of which is suitable to indicate the tilt of the steering column, the telescoping of the steering column and the rotation of the steering wheel.

Images