Ergonomic reconfiguration of virtual input devices

Abstract

A mechanism for ergonomic-based reconfiguration of a virtual input device is provided. A method is implemented by a control device. The method includes obtaining user posture and / or movement information from a sensor while the user interacts with the virtual input device in a virtual environment. The method includes determining whether the user's posture and / or movement meets an ergonomic criterion by comparing the user's posture and / or movement with reference ergonomic data. The method includes reconfiguring at least one setting of the virtual input device when the user's posture and / or movement fails to meet the ergonomic criterion.

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION The embodiments presented herein relate to control devices, methods, computer programs, and computer program products for ergonomic-based reconfiguration of virtual input devices. [Background technology]

[0002] Virtual input devices, such as optical virtual keyboards, enable character input in virtual reality (VR) or augmented reality (AR) environments without the need for physical keys. Human interaction with virtual keyboards is often achieved through a touchscreen interface, but may take different forms in VR or AR environments. In this regard, an optical virtual keyboard is an example of a user interface in VR or AR that is configured to optically detect and analyze human hand and finger movements and interpret them as actions on a non-physically present input device, such as a surface with painted or projected keys. In this way, an optical virtual keyboard can emulate unlimited types of manually operated input devices (such as mice, keyboards, and other devices). Yet another example of an optical virtual keyboard is a projected keyboard, a form of computer input device whereby an image of a virtual keyboard is projected onto a surface. When a user touches a surface covered by the image of the keys, the input device records the corresponding keystroke. Furthermore, some optical virtual keyboards are based on a combination of lasers and tactile sensors, taking into account finger-on-projected-area movement combined with micro-vibration detection. For example, a projected finger tap detected simultaneously with a vibration rendered by the tap indicates a keystroke. Mechanical input units are thereby replaced by such virtual input devices potentially optimized for specific applications and for the user's physiology, while maintaining the speed, simplicity, and clarity of manual data entry.

[0003] However, it can be cumbersome for a user to find the correct settings for a virtual input device, and therefore the virtual input device is in fact optimized for a particular application and for the user's physiology. If a user has difficulty finding such correct settings, the user may either avoid making settings or avoid even using the virtual input device.

[0004] Users of virtual input devices may therefore still experience ergonomic issues for the user. Therefore, there remains a need for virtual input devices with improved ergonomics. Summary of the Invention

[0005] The purpose of the embodiments described herein is to address the above-mentioned problems and to provide techniques that improve ergonomics for users while interacting with virtual input devices.

[0006] According to a first aspect, the object is addressed by providing a control device for ergonomic-based reconfiguration of a virtual input device. The control device includes a processing circuit configured to cause the control device to acquire user posture and / or movement information from a sensor while the user interacts with the virtual input device in a virtual environment. The processing circuit is configured to cause the control device to determine whether the user's posture and / or movement meets an ergonomic standard by comparing the user's posture and / or movement with reference ergonomic data. The processing circuit is configured to cause the control device to reset at least one setting of the virtual input device when the user's posture and / or movement fails to meet the ergonomic standard. The at least one setting is reset as a function of the user's posture and / or movement and the reference ergonomic data to suggest to the user to change their posture and / or movement to more closely meet the ergonomic standard.

[0007] According to a second aspect, the object is addressed by providing a method for ergonomic-based reconfiguration of a virtual input device. The method is implemented by a control device. The method includes acquiring information about a user's posture and / or movement from a sensor while the user interacts with the virtual input device in a virtual environment. The method includes determining whether the user's posture and / or movement meets an ergonomic standard by comparing the user's posture and / or movement with reference ergonomic data. The method includes reconfiguring at least one setting of the virtual input device when the user's posture and / or movement fails to meet the ergonomic standard. The at least one setting is reconfigured as a function of the user's posture and / or movement and the reference ergonomic data to suggest to the user to change their posture and / or movement to more closely meet the ergonomic standard.

[0008] According to a third aspect, the object is addressed by providing a computer program for ergonomic-based reconfiguration of a virtual input device, the computer program comprising computer program code which, when run on a control device, causes the control device to perform the method according to the second aspect.

[0009] According to a fourth aspect, the object is addressed by providing a computer program product comprising a computer program according to the fourth aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium may be a non-transitory computer readable storage medium.

[0010] According to a fourth aspect, the object is addressed by providing a system comprising a control device according to the first aspect and a user interface device, the user interface device comprising a projection module for making a virtual input device visible on a surface, the user interface device further comprising a sensor for detecting user interaction of a user with the virtual input device.

[0011] Advantageously, these aspects allow the shape, size, format, and / or physical placement of the virtual input device to be adapted to the individual physical characteristics of the user. In this way, the user is encouraged to assume a more ergonomic position with respect to the virtual input device, thus avoiding, reducing, or at least mitigating ergonomic problems caused by interacting with the virtual input device.

[0012] In some embodiments, the user's posture and / or movements are represented by a user model that defines a digital twin of the user, and the digital twin is compared to reference ergonomic data when determining whether the user's posture and / or movements meet ergonomic criteria.

[0013] In some embodiments, the digital twin represents any or any combination of the user's arms, hands, legs, feet, neck, head and torso with relative dimensions extracted from the digital representation of the user.

[0014] In some embodiments, the threshold value depends on the relative dimensions of the user's arms, hands, head and / or torso, as derived from a digital representation of the user.

[0015] In some embodiments, the reference ergonomic data represents postures and / or movements to be used by the user, and the user's posture and / or movements fail to meet ergonomic criteria when the user's posture and / or movements deviate from the reference ergonomic data by more than a first threshold, thereby suggesting the user to change their posture and / or movements to more closely match the reference ergonomic data.

[0016] In some embodiments, the reference ergonomic data represents postures and / or movements that should be avoided by the user, and the user's posture and / or movements fail to meet the ergonomic criteria when the user's posture and / or movements deviate from the reference ergonomic data by less than a second threshold, thereby suggesting the user to change their posture and / or movements to move further away from the reference ergonomic data.

[0017] In some embodiments, the processing circuitry may be configured to cause the control device to: The device is further configured to retrieve historical data representative of the user's previous postures and / or movements, and further, determining whether the user's postures and / or movements meet ergonomic criteria is a function of the historical data.

[0018] In some embodiments, the processing circuitry may be configured to cause the control device to: aggregating tracked postures and / or movements of the user over time; Constructing a user model from the user's tracked poses and / or movements; The method is further configured to:

[0019] In some embodiments, the user's posture and / or movements are tracked by at least one sensor configured to record sensor data representing a digital representation of the user, and the digital representation of the user is analyzed to track the user's posture and / or movements with respect to a user input device in the virtual environment.

[0020] In some embodiments, the digital representation of the user relates to at least one of the user's hand position and / or movement, the user's arm position and / or movement, the user's head position and / or movement, the user's neck position and / or movement, and the user's torso position and / or movement.

[0021] In some embodiments, reconfiguring at least one setting of the virtual input device is constrained by at least one property of the physical environment in which the virtual input device is deployed.

[0022] In some embodiments, the constraints are derived by the control device from information received by the control device of the physical environment.

[0023] In some embodiments, at least one setting relates to the layout of buttons and / or keys on the virtual input device, and the at least one setting is reconfigured by changing the layout of buttons and / or keys on the virtual input device.

[0024] In some embodiments, at least one setting relates to the size and / or shape of buttons and / or keys on the virtual input device, and the at least one setting is reconfigured by changing the size and / or shape of the buttons and / or keys on the virtual input device.

[0025] In some embodiments, at least one setting relates to the size and / or shape of the virtual input device, and the at least one setting is reset by changing the size and / or shape of the virtual input device.

[0026] In some embodiments, at least one setting relates to the spatial location of the virtual input device, and the at least one setting is reset by changing the spatial location of the virtual input device.

[0027] In some embodiments, at least one setting of the virtual input device is reset in stages from a starting setting, through at least one intermediate setting, to a final setting.

[0028] In some embodiments, the virtual input device is one of a virtual keyboard, a virtual computer mouse, a virtual remote controller, or a virtual gaming controller.

[0029] In some embodiments, the virtual environment is an extended reality (XR) virtual environment.

[0030] In some embodiments, the virtual environment is either an augmented reality (AR) virtual environment, a virtual reality (VR) virtual environment, or a mixed reality (MR) virtual environment.

[0031] In some embodiments, the control device is part of or integrated with the communication device.

[0032] Other objectives, features, and advantages of the enclosed embodiments will become apparent from the following detailed disclosure, from the attached dependent claims, as well as from the drawings.

[0033] In general, all terms used in the claims should be interpreted according to their customary meaning in the art unless expressly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." should be openly interpreted as referring to at least one instance of that element, apparatus, component, means, module, step, etc., unless expressly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless expressly stated.

[0034] Furthermore, the description of technical features or method steps preceded by the term "comprising" should be understood as not excluding the presence of other technical features or method steps not recited in the appended claims.

[0035] The inventive concept will now be described, by way of example only, with reference to the accompanying drawings in which: [Brief explanation of the drawings]

[0036] [Figure 1] 1 is a schematic diagram illustrating a system according to an embodiment. [Figure 2] 1A and 1B are diagrams illustrating a user interacting with a user interface device according to an embodiment; [Figure 3] 1A and 1B are schematic diagrams illustrating a user interface device projecting a virtual input device according to an embodiment; [Figure 4] 1 is a flowchart of a method according to an embodiment. [Figure 5] 1 is a flowchart of a method according to an embodiment. [Figure 6] 2 is a schematic diagram illustrating functional units of a control device according to one embodiment; [Figure 7] FIG. 2 is a schematic diagram illustrating functional modules of a control device according to one embodiment. [Figure 8] FIG. 1 illustrates an example of a computer program product comprising a computer-readable storage medium according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0037] The inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which several embodiments of the inventive concepts are shown. However, the inventive concepts may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concepts to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature indicated by a dashed line should be considered optional.

[0038] 1 is a schematic diagram of a system 100. The system 100 comprises a user interface device 110 and a control device 600. The user interface device 110 and the control device 600 are operatively connected to each other.

[0039] In some implementations, the control device 600 is part of or integrated with the user interface device 110. In some implementations, the control device 600 is part of or integrated with a communication device, such as a mobile phone, tablet, computer, etc.

[0040] The user interface device 110 includes a projection module 114 for making a virtual input device 120 visible on the surface. In Figure 1, the virtual input device 120 is shown schematically as a virtual keyboard. In other non-limiting examples, the virtual input device 120 is one of a virtual computer mouse, a virtual remote controller, or a virtual gaming controller.

[0041] The user interface device 110 further comprises sensors 112 for detecting user interactions of a user with the virtual input device 120 in coordinates along the surface. The sensors 112 are further configured to track the user's posture and / or movement. The user's posture and / or movement may be tracked by at least one sensor 112 configured to record sensor data representing a digital representation of the user. The digital representation of the users 210a, 210b may be analyzed to track the user's posture and / or movement with respect to the user input device 120 in the virtual environment. In some non-limiting examples, the digital representation of the users 210a, 210b relates to at least one of hand positions and / or movement of the users 210a, 210b, arm positions and / or movement of the users 210a, 210b, head positions and / or movement of the users 210a, 210b, neck positions and / or movement of the users 210a, 210b, and torso positions and / or movement of the users 210a, 210b. The sensor 112 may be a radar module, a lidar module, a camera module, etc. In some examples, the sensor 112 is an inertial measurement unit (IMU) and is provided on a glove worn by the user. Thus, the functionality of the user interface device 110 may be split between at least two physical devices. Furthermore, combinations of different types of sensors 112 are also possible.

[0042] In FIG. 1 , the user is represented schematically by two hands 130 a, 130 b. Software translates the coordinates to identify actions or characters entered by the user. The virtual input device 120 is part of the virtual environment. There can be different types of virtual environments. In some non-limiting examples, the virtual environment is an extended reality (XR) virtual environment. In further non-limiting examples, the virtual environment is either an augmented reality (AR) virtual environment, a virtual reality (VR) virtual environment, or a mixed reality (MR) virtual environment. Thus, although the user interface device 110 in FIG. 1 is shown as a standalone device, the user interface device 110 can be part of a headset (such as a VR headset) or wearable computer glasses (such as AR glasses or smart glasses).

[0043] As noted above, there remains a need for a virtual input device 120 with improved ergonomics. To further illustrate this, reference is made to FIG. 2. FIG. 2 schematically illustrates a first user 120a interacting with a user interface device 120 with a virtual keyboard having a first layout 230a and a second user 120b interacting with the user interface device 120 with a virtual keyboard having a second layout 230b. The first user 210a is modeled according to a first user model 220a, and the second user 210b is modeled according to a second user model 220b. It is assumed that the first user 210a and the second user 120b are identical to each other. Compared to a standard ergonomic model in which a user's arms and hands both follow a straight line and an imaginary line between the shoulders should be close to 90 degrees relative to that line, the first layout 230a creates stress on the shoulders and wrists of the first user 120a. The second layout 230b is much closer to the standard ergonomic model. Therefore, it can be concluded from the first user model 220a and the second user model 220b that the second layout 230b provides the user with a more ergonomically appropriate virtual keyboard than the first layout 230a. Therefore, interacting with the user interface device 120 with a virtual keyboard having the first layout 230a may cause ergonomic problems for the first user 210a.

[0044] Accordingly, embodiments disclosed herein relate to a mechanism for ergonomic-based reconfiguration of virtual input devices. To obtain such a mechanism, a control device, a method implemented by the control device, and a computer program product comprising code, e.g., in the form of a computer program, are provided, which code, when run on the control device, causes the control device to implement the method.

[0045] Reference will now be made in parallel to FIGS. 3 and 4 for disclosure of ergonomic-based reconfiguration of the virtual input device 120. FIG.

[0046] 3 schematically illustrates a user interface device 110 projecting a virtual input device 120 in the form of a virtual keyboard that is changeable between a first setting 410a, e.g., a default keyboard setting, and a second setting 410b that would correspond to a more ergonomically advantageous keyboard position and layout for the current user 210a, 210b. The change between the first setting 410a and the second setting 410b is indicated schematically by arrow 420. As disclosed further below, the control device 600 is configured to adaptively change the virtual input device 120 between the first setting 410a and the second setting 410b based on a comparison of the current posture and / or movement of the user 210a, 210b with reference ergonomic data.

[0047] 4 is a flow chart illustrating an embodiment of a method for ergonomic-based reconfiguration of a virtual input device 120. The method is performed by a control device 600. The method is advantageously provided as a computer program 320.

[0048] S102: The control device 600 acquires posture and / or movement information of the users 210a, 210b from the sensors 112 while the users 210a, 210b interact with the virtual input device 120 in the virtual environment.

[0049] S110: The control device 600 determines whether the posture and / or movement of the user 210a, 210b meets ergonomic criteria by comparing the posture and / or movement of the user 210a, 210b with reference ergonomic data.

[0050] S112: The control device 600 resets at least one setting 410a, 410b of the virtual input device 120 when the posture and / or movement of the user 210a, 210b cannot meet the ergonomic criteria.

[0051] The at least one setting 410a, 410b suggests how the user 210a, 210b should interact with the virtual input device 120 in the virtual environment, in terms of posture and / or movement. Examples of settings are provided below. The at least one setting 410a, 410b is reset as a function of the posture and / or movement of the user 210a, 210b and reference ergonomic data.

[0052] Reconfiguring at least one setting 410a, 410b of the virtual input device 120 suggests to the user 210a, 210b to change their posture and / or movements to more closely meet ergonomic standards.

[0053] Referring again to FIG. 2, by tracking the posture and / or movements of the users 210a, 210b while they interact with the virtual input device 120 in the virtual environment, the control device 600 can adaptively change the virtual keyboard between the first layout 230a and the second layout 230b depending on the posture and / or movements of the users 210a, 210b, thereby helping the users 210a, 210b avoid ergonomic problems or at least resulting in fewer ergonomic problems for the users 210a, 210b.

[0054] Next, embodiments relating to further details of the ergonomic-based reconfiguration of the virtual input device 120 implemented by the control device 600 are disclosed.

[0055] In some aspects, the posture and / or movements of the users 210a, 210b are tracked by the user models 220a, 220b. In particular, in some embodiments, the posture and / or movements of the users 210a, 210b are represented by the user models 220a, 220b stored as digital twins of the users 210a, 210b. The digital twins are then compared to reference ergonomic data when determining whether the posture and / or movements of the users 210a, 210b meet ergonomic criteria. Thus, a holistic model of the users 210a, 210b defined by the digital twins can be created, and each limb of the users 210a, 210b can be digitally recreated to enable the control device 600 to calculate a representation of the current user pose and limb placement, as well as hand, arm, and finger placement and movement relative to each other. This allows the control device 600 not only to obtain a snapshot of the current ergonomic situation, but also to build an understanding of the ergonomic situation over time.

[0056] In some embodiments, the relative lengths of the scale of the limbs of the users 210a, 210b are taken into consideration to avoid using settings 410a, 410b of the virtual input device 120 that result in poor ergonomic positions for the individual users 210a, 210b. In particular, in some non-limiting examples, the digital twin represents any one or any combination of the arms, hands, legs, feet, neck, head, and torso of the users 210a, 210b with relative dimensions extracted from the digital representations of the users 210a, 210b. Similarly, in some non-limiting examples, the threshold value depends on the relative dimensions of the arms, hands, head, and / or torso of the users 210a, 210b derived from the digital representations of the users 210a, 210b.

[0057] In some embodiments, the user models 220a, 220b are compared to existing predefined models of ergonomically correct and preferred poses and limb placements, and / or a set of identified explicitly bad constellations that should be avoided (such as sharp limb angles, short distances to the trunk causing forward head posture / hunched back, etc.). If deviations from the existing predefined models are detected, an ergonomic issue is deemed to be identified, thus determining whether the posture and / or movements of the users 210a, 210b meet the ergonomic criteria.

[0058] In some aspects, the user models 220a, 220b are compared to predefined models of postures and / or movements that are ergonomically appropriate and should be used. In particular, in some embodiments, the reference ergonomic data represents the postures and / or movements that should be used by the users 210a, 210b. The postures and / or movements of the users 210a, 210b fail to meet ergonomic standards when the postures and / or movements of the users 210a, 210b deviate from the reference ergonomic data by more than a first threshold. The users 210a, 210b are thereby suggested to change their postures and / or movements to more closely match the reference ergonomic data.

[0059] In some aspects, the user models 220a, 220b are compared to predefined models of postures and / or movements that are ergonomically poor and should be avoided. In particular, in some embodiments, the reference ergonomic data represents postures and / or movements that should be avoided by the users 210a, 210b. The postures and / or movements of the users 210a, 210b fail to meet ergonomic standards when the postures and / or movements of the users 210a, 210b deviate from the reference ergonomic data by less than a second threshold. The users 210a, 210b are thereby suggested to change their postures and / or movements to further deviate from the reference ergonomic data.

[0060] In some aspects, both spatial history data and temporal history data may be used as input, e.g., it may be okay for a user to use a less optimal pose and limb placement for less than 5 minutes if the input does not repeat in a pattern over time. In particular, in some embodiments, the control device 600 is configured to perform (optional) step S104. S104: The control device 600 retrieves historical data representing previous postures and / or movements of the users 210a, 210b. Furthermore, determining whether the postures and / or movements of the users 210a, 210b meet ergonomic criteria is a function of the historical data.

[0061] In some aspects, the control device 600 aggregates positions and / or movements over time to build an understanding of the ergonomic situation over time. In particular, in some embodiments, the control device 600 is configured to perform (optional) steps S106 and S108. S106: The control device 600 aggregates the tracked postures and / or movements of the users 210a, 210b over time. S108: The control device 600 constructs user models 220a, 220b from the tracked poses and / or movements of the users 210a, 210b.

[0062] The relative positioning of each head, torso, arm, hand, finger, etc. can thereby be aggregated over time so that the control device 600 not only detects poor ergonomic constellations being used intermittently, but also takes into account their long-term (aggregated) use.

[0063] In some aspects, the reconfiguration is constrained by the physical environment in which the virtual input device 120 is deployed. For example, if the virtual input device 120 is deployed in a limited physical space, such as when a user is traveling using public transportation, there may be less available space than if the virtual input device 120 is deployed in a private or work environment specifically designed for interaction with the virtual input device 120. In particular, in some embodiments, the virtual input device 120 is deployed in a physical environment, and the reconfiguration of at least one setting 410a, 410b of the virtual input device 120 is constrained by at least one property of the physical environment. In particular, in some embodiments, the constraint is derived by the control device 600 from information received by the control device 600 of the physical environment. The at least one property of the physical environment may be sensed by the sensor 112. For example, the sensor 112 may be configured to measure or estimate proximity to objects in the physical environment and provide the measured or estimated value as information to the control device 600.

[0064] In some aspects, virtual input device 120 may be altered with respect to layout, size, shape, format, or location, regardless of whether virtual input device 120 is a virtual keyboard, a virtual computer mouse, a virtual remote controller, or a virtual gaming controller. Accordingly, virtual input device 120 may be reconfigured to alternate its physical locality with respect to other input devices used by the user within a determined appropriate positional range to help the user avoid static loads on joints, neck, etc.

[0065] According to a first example, the changes relate to the size or shape of individual buttons and / or keys of the virtual input device 120. In particular, in some embodiments, at least one setting 410a, 410b relates to the size and / or shape of buttons and / or keys on the virtual input device 120. The at least one setting 410a, 410b is then reset by changing the size and / or shape of the buttons and / or keys on the virtual input device 120.

[0066] According to a second example, the change relates to the overall size or shape of the virtual input device 120. In particular, in some embodiments, at least one setting 410a, 410b relates to the size and / or shape of the virtual input device 120. The at least one setting 410a, 410b is then reset by changing the size and / or shape of the virtual input device 120.

[0067] According to a third example, the change relates to the spatial location of the virtual input device 120. In particular, in some embodiments, at least one setting 410a, 410b relates to the spatial location of the virtual input device 120. The at least one setting 410a, 410b is then reset by changing the spatial location of the virtual input device 120.

[0068] In some aspects, the virtual input device 120 is reconfigured in stages. In particular, in some embodiments, at least one setting 410a, 410b of the virtual input device 120 is reconfigured in stages from an initial setting, through at least one intermediate setting, to a final setting. Thus, the reconfiguration may be performed gradually over a period of time so as not to disturb the user during an input session. This means, for example, that the virtual keyboard may gently and gradually shapeshift from one keyboard layout to another.

[0069] In some embodiments, upon determining that the posture and / or movement of the user 210a, 210b cannot meet the ergonomic criteria, but before resetting at least one setting 410a, 410b of the virtual input device 120, the control device 600 may provide instructions to the user 210a, 210b to make the user 210a, 210b aware that the posture and / or movement of the user 210a, 210b cannot meet the ergonomic criteria. In some cases, the instructions are accompanied by instructions to give the user 210a, 210b an opportunity to cancel or accept the proposed solution. Accepting the proposed solution may also speed up changing the configuration of the virtual input device 120 and may even allow, for example, to proceed directly to a new keyboard layout without an intermediate keyboard layout.

[0070] FIG. 5 is a flowchart of a method for ergonomic-based reconfiguration of a virtual input device 120 implemented by a control device 600 according to at least some of the embodiments, aspects, and examples disclosed herein.

[0071] S201: The control device 600 detects that a user has attempted to start a session with the virtual input device 120.

[0072] S202: The control device 600 commands the user interface device 110 to make the virtual input device 120 visible.

[0073] S203: The control device 600 tracks the posture and / or movement of the users 210a, 210b while they interact with the virtual input device 120.

[0074] S204: The control device 600 determines whether the posture and / or movement of the user 210a, 210b meets the ergonomic criteria by comparing the posture and / or movement of the user 210a, 210b with reference ergonomic data.

[0075] S205: The posture and / or movement of the user 210a, 210b fails to meet ergonomic standards. The control device 600 resets at least one setting 410a, 410b of the virtual input device.

[0076] S206: The control device 600 detects that the user no longer interacts with the virtual input device 120 and decides to end the session started in S201.

[0077] S207: The control device 600 commands the user interface device 110 to stop making the virtual input device 120 visible.

[0078] Figure 6 shows, in terms of several functional units, components of a control device 600 according to one embodiment. The processing circuitry 610 is provided using any combination of one or more suitable central processing units (CPUs), multiprocessors, microcontrollers, digital signal processors (DSPs), etc., capable of executing software instructions stored, for example, in a computer program product 810 (as in Figure 8) in the form of a storage medium 630. The processing circuitry 610 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0079] In particular, processing circuitry 610 is configured to cause control device 600 to perform a set of operations or steps, as disclosed above. For example, storage medium 630 may store the set of operations, and processing circuitry 610 may be configured to retrieve the set of operations from storage medium 630 to cause control device 600 to perform the set of operations. The set of operations may be provided as a set of executable instructions.

[0080] Thus, the processing circuit 610 is configured to perform the methods disclosed herein. The storage medium 630 may also comprise persistent storage, which may be, for example, any one or combination of magnetic memory, optical memory, solid-state memory, or even remotely located memory. The control device 600 may further comprise at least a communication interface 620 configured for communication with other entities, functions, nodes, and devices. Accordingly, the communication interface 620 may comprise one or more transmitters and receivers comprising analog and digital components. The processing circuit 610 controls the overall operation of the control device 600, for example, by sending data and control signals to the communication interface 620 and the storage medium 630, by receiving data and reports from the communication interface 620, and by retrieving data and instructions from the storage medium 630. Other components of the control device 600 and related functions are omitted so as not to obscure the concepts presented herein.

[0081] 7 schematically illustrates components of a control device 600 according to one embodiment with respect to several functional modules. The control device 600 of FIG. 7 includes several functional modules: an acquisition module 710 configured to perform step S102, a determination module 750 configured to perform step S110, and a reconfiguration module 760 configured to perform step S112. The control device 600 of FIG. 7 may further include several optional functional modules, such as any of a retrieval module 720 configured to perform step S104, an aggregation module 730 configured to perform step S106, and a construction module 740 configured to perform step S108. Broadly speaking, each functional module 710:760 may be implemented solely in hardware in one embodiment, or with the aid of software in another embodiment, i.e., the latter embodiment having computer program instructions stored on a storage medium 630 that, when run on a processing circuit, causes the control device 600 to perform the corresponding steps described above in connection with FIG. 7. It should also be mentioned that while the modules correspond to portions of a computer program, they need not be separate modules therein, and the way in which they are implemented in software depends on the programming language used. Preferably, one or more or all of the functional modules 710:760 may be implemented by the processing circuitry 610, possibly in cooperation with the communication interface 620 and / or the storage medium 630. The processing circuitry 610 is therefore configured to fetch instructions provided by the functional modules 710:760 from the storage medium 630 and execute these instructions, thereby performing any steps disclosed herein.

[0082] A first portion of the instructions performed by the control device 600 may be executed on a first device, and a second portion of the instructions performed by the control device 600 may be executed on a second device. The embodiments disclosed herein are not limited to any particular number of devices on which the instructions performed by the control device 600 may be executed. Thus, methods according to embodiments disclosed herein are suitable for being performed by a control device 600 residing in a cloud computing environment. Thus, although a single processing circuit 610 is shown in FIG. 6, the processing circuit 610 may be distributed among multiple devices or nodes. The same applies to the functional modules 710:760 in FIG. 7 and the computer program 820 in FIG. 8.

[0083] 8 shows an example of a computer program product 810 comprising a computer-readable storage medium 830. On this computer-readable storage medium 830, a computer program 820 may be stored that causes processing circuitry 610, and entities and devices operatively coupled to processing circuitry 610, such as communications interface 620 and storage medium 630, to perform methods according to embodiments described herein. Thus, computer program 820 and / or computer program product 810 may provide means for performing any step disclosed herein.

[0084] 8, computer program product 810 is shown as an optical disc, such as a CD (compact disc) or DVD (digital versatile disc) or Blu-Ray disc. Computer program product 810 may also be embodied as memory, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM), more particularly as a non-volatile storage medium of the device in an external memory, such as a USB (universal serial bus) memory or a flash memory, such as a compact flash memory. Thus, although computer program 820 is shown here schematically as a track on the illustrated optical disc, computer program 820 may be stored in any manner suitable for computer program product 810.

[0085] The inventive concept has been described above primarily with reference to a few embodiments. However, as will be readily appreciated by those skilled in the art, other embodiments besides those disclosed above are equally possible within the scope of the inventive concept as defined by the appended claims.

Claims

1. A control device (600) for ergonomic-based reconfiguration of a virtual input device (120), said control device (600) comprising a processing circuit (610), said processing circuit (610) providing said control device (600) with: acquiring posture and / or movement information of the users (210a, 210b) from sensors (112) while the users (210a, 210b) interact with the virtual input device (120) in a virtual environment; determining whether the posture and / or movements of the users (210a, 210b) meet ergonomic criteria by comparing the posture and / or movements of the users (210a, 210b) with reference ergonomic data; the reference ergonomic data represents postures and / or movements to be used by the users (210a, 210b), and when the postures and / or movements of the users (210a, 210b) deviate from the reference ergonomic data by more than a first threshold, determining that the postures and / or movements of the users (210a, 210b) cannot meet the ergonomic criteria; or determining that the posture and / or movement of the user (210a, 210b) cannot meet the ergonomic criteria when the reference ergonomic data represents postures and / or movements that should be avoided by the user (210a, 210b), and the posture and / or movement of the user (210a, 210b) deviates from the reference ergonomic data by less than a second threshold value; and resetting at least one setting (410a, 410b) of the virtual input device (120) as a function of the posture and / or movement of the user (210a, 210b) and the reference ergonomic data, in order to suggest to the user (210a, 210b) to change the posture and / or movement so as to come closer to satisfying the ergonomic criterion when the posture and / or movement of the user (210a, 210b) cannot satisfy the ergonomic criterion; is set to the postures and / or movements of the users (210a, 210b) are represented by user models (220a, 220b) that define a digital twin of the users (210a, 210b), and the digital twin is compared with the reference ergonomic data when determining whether the postures and / or movements of the users (210a, 210b) satisfy the ergonomic criteria; A control device (600) wherein the first and second thresholds depend on relative dimensions of the arms, hands, head and / or torso of the users (210a, 210b) derived from a digital representation of the users (210a, 210b).

2. 2. The control device (600) of claim 1, wherein the digital twin represents any one or any combination of the arms, hands, legs, feet, neck, head, and torso of the user (210a, 210b) with relative dimensions extracted from a digital representation of the user (210a, 210b).

3. 3. The control device (600) of claim 1 or 2, wherein the reference ergonomic data represents postures and / or movements to be used by the users (210a, 210b), and when the postures and / or movements of the users (210a, 210b) deviate from the reference ergonomic data by more than the first threshold, if it is determined that the postures and / or movements of the users (210a, 210b) cannot meet the ergonomic criteria, the control device (600) suggests that the users (210a, 210b) change their postures and / or movements to come closer to the reference ergonomic data.

4. 4. The control device (600) of claim 1, wherein when the reference ergonomic data represents postures and / or movements to be avoided by the user (210a, 210b), and when the posture and / or movements of the user (210a, 210b) deviate from the reference ergonomic data by less than the second threshold, if it is determined that the posture and / or movements of the user (210a, 210b) cannot meet the ergonomic criteria, the control device (600) suggests that the user (210a, 210b) change the posture and / or movements to move further away from the reference ergonomic data.

5. The processing circuit (610) instructs the control device (600) to:

5. The control device (600) of claim 1, further configured to retrieve historical data representing previous postures and / or movements of the user (210a, 210b), and further determining whether the postures and / or movements of the user (210a, 210b) satisfy the ergonomic criteria is a function of the historical data.

6. The processing circuit (610) instructs the control device (600) to: aggregating tracked poses and / or movements of the users (210a, 210b) over time; constructing a user model (220a, 220b) from the tracked poses and / or movements of the user (210a, 210b); 6. The control device (600) of any one of claims 1 to 5, further configured to:

7. A control device (600) as described in any one of claims 1 to 6, wherein the posture and / or movements of the users (210a, 210b) are tracked by at least one sensor (112) configured to record sensor data representing a digital representation of the users (210a, 210b), and the digital representation of the users (210a, 210b) is analyzed to track the posture and / or movements of the users (210a, 210b) relative to user input devices in the virtual environment.

8. 8. The control device (600) of claim 7, wherein the digital representation of the user (210a, 210b) relates to at least one of the hand position and / or movement of the user (210a, 210b), the arm position and / or movement of the user (210a, 210b), the head position and / or movement of the user (210a, 210b), the neck position and / or movement of the user (210a, 210b), and the torso position and / or movement of the user (210a, 210b).

9. 9. A control device (600) as described in any one of claims 1 to 8, wherein reconfiguring the at least one setting (410a, 410b) of the virtual input device (120) is constrained by at least one property of the physical environment in which the virtual input device (120) is deployed.

10. 10. The control device (600) of claim 9, wherein the constraints are derived by the control device (600) from information received by the control device (600) of the physical environment.

11. 11. The control device (600) of claim 1, wherein the at least one setting (410a, 410b) relates to a layout of buttons and / or keys on the virtual input device (120), and the at least one setting (410a, 410b) is reconfigured by changing the layout of the buttons and / or keys on the virtual input device (120).

12. 12. The control device (600) of claim 1, wherein the at least one setting (410a, 410b) relates to the size and / or shape of buttons and / or keys on the virtual input device (120), and the at least one setting (410a, 410b) is reconfigured by changing the size and / or shape of the buttons and / or keys on the virtual input device (120).

13. 13. The control device (600) of claim 1, wherein the at least one setting (410a, 410b) relates to the size and / or shape of the virtual input device (120), and the at least one setting (410a, 410b) is reconfigured by changing the size and / or shape of the virtual input device (120).

14. 14. The control device (600) of claim 1, wherein the at least one setting (410a, 410b) is related to the spatial location of the virtual input device (120), and the at least one setting (410a, 410b) is reset by changing the spatial location of the virtual input device (120).

15. 15. The control device (600) of claim 1, wherein the at least one setting (410a, 410b) of the virtual input device (120) is reset in stages from an initial setting, through at least one intermediate setting, to a final setting.

16. 16. The control device (600) of any one of claims 1 to 15, wherein the virtual input device (120) is one of a virtual keyboard, a virtual computer mouse, a virtual remote controller, and a virtual gaming controller.

17. The control device (600) of any one of claims 1 to 16, wherein the virtual environment is an extended reality (XR) virtual environment.

18. The control device (600) of any one of claims 1 to 17, wherein the virtual environment is an augmented reality (AR) virtual environment, a virtual reality (VR) virtual environment, or a mixed reality (MR) virtual environment.

19. The control device (600) of any one of claims 1 to 18, wherein the control device (600) is part of or integrated with a communication device.

20. 20. A system (100) comprising a control device (600) according to any one of claims 1 to 19 and a user interface device (110), the user interface device (110) comprising a projection module (114) for making a virtual input device (120) visible on a surface and a sensor (112) for detecting user interaction of a user (210a, 210b) with the virtual input device (120).

21. A method for ergonomic-based reconfiguration of a virtual input device (120), said method being performed by a control device (600), said method comprising: acquiring (S102) posture and / or movement information of a user (210a, 210b) from a sensor (112) while the user (210a, 210b) interacts with the virtual input device (120) in a virtual environment; determining (S110) whether the posture and / or movements of the users (210a, 210b) meet ergonomic criteria by comparing the posture and / or movements of the users (210a, 210b) with reference ergonomic data; the reference ergonomic data represents postures and / or movements to be used by the users (210a, 210b), and when the postures and / or movements of the users (210a, 210b) deviate from the reference ergonomic data by more than a first threshold, determining that the postures and / or movements of the users (210a, 210b) cannot meet the ergonomic criteria; or determining that the posture and / or movement of the user (210a, 210b) cannot meet the ergonomic criteria when the reference ergonomic data represents postures and / or movements that should be avoided by the user (210a, 210b), and the posture and / or movement of the user (210a, 210b) deviates from the reference ergonomic data by less than a second threshold value; and resetting (S112) at least one setting (410a, 410b) of the virtual input device (120) as a function of the posture and / or movement of the user (210a, 210b) and the reference ergonomic data, in order to suggest to the user (210a, 210b) to change the posture and / or movement so as to come closer to satisfying the ergonomic criterion when the posture and / or movement of the user (210a, 210b) cannot satisfy the ergonomic criterion; Including, the postures and / or movements of the users (210a, 210b) are represented by user models (220a, 220b) that define a digital twin of the users (210a, 210b), and the digital twin is compared with the reference ergonomic data when determining whether the postures and / or movements of the users (210a, 210b) satisfy the ergonomic criteria; The method, wherein the first and second thresholds depend on relative dimensions of the arms, hands, head and / or torso of the users (210a, 210b) derived from a digital representation of the users (210a, 210b).

22. A computer program (820) for ergonomic-based reconfiguration of a virtual input device (120), the computer program comprising computer code that, when run on a processing circuit (610) of a control device (600), causes the control device (600) to: acquiring (S102) posture and / or movement information of a user (210a, 210b) from a sensor (112) while the user (210a, 210b) interacts with the virtual input device (120) in a virtual environment; determining (S110) whether the posture and / or movements of the users (210a, 210b) meet ergonomic criteria by comparing the posture and / or movements of the users (210a, 210b) with reference ergonomic data; the reference ergonomic data represents postures and / or movements to be used by the users (210a, 210b), and when the postures and / or movements of the users (210a, 210b) deviate from the reference ergonomic data by more than a first threshold, determining that the postures and / or movements of the users (210a, 210b) cannot meet the ergonomic criteria; or determining that the posture and / or movement of the user (210a, 210b) cannot meet the ergonomic criteria when the reference ergonomic data represents postures and / or movements that should be avoided by the user (210a, 210b), and the posture and / or movement of the user (210a, 210b) deviates from the reference ergonomic data by less than a second threshold value; resetting (S112) at least one setting (410a, 410b) of the virtual input device (120) as a function of the posture and / or movement of the user (210a, 210b) and the reference ergonomic data, in order to suggest to the user (210a, 210b) to change the posture and / or movement so as to come closer to satisfying the ergonomic criterion when the posture and / or movement of the user (210a, 210b) cannot satisfy the ergonomic criterion; Let them do this, the postures and / or movements of the users (210a, 210b) are represented by user models (220a, 220b) that define a digital twin of the users (210a, 210b), and the digital twin is compared with the reference ergonomic data when determining whether the postures and / or movements of the users (210a, 210b) satisfy the ergonomic criteria; A computer program (820) wherein the first and second thresholds depend on relative dimensions of the arms, hands, head and / or torso of the users (210a, 210b) derived from a digital representation of the users (210a, 210b).

Images