Virtual Remote Telephysical Examination System

Abstract

A method, system, and apparatus for estimating a force acting on at least one joint of a user while the user is interacting with at least one virtual object within a virtual environment are described. Motion data associated with joint data of at least one joint of the user can be received from a sensor. The joint data can be used to determine force information. The force information can be used to determine a user strength associated with at least one joint.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 351,671, filed on June 13, 2022, which is hereby incorporated by reference in its entirety.

Background Art

[0002] Evaluating joint strength is an important step in rehabilitation monitoring and general musculoskeletal examinations. In in - person evaluations, joint strength has been evaluated using isometric tests that require physical interaction between a physician and a patient. However, a significant percentage of patients in need of care do not live near a medical center, especially in remote non - urban areas. Additionally, even with such infrastructure, proximity can be inhibited by external factors such as a pandemic. Such issues have highlighted the need for telemedicine and remote evaluation procedures to ensure continuous accessible care. In particular, there are multiple approaches for remote strength evaluation using advanced methods, such as motion capture, on - body sensors, and tactile devices at one end of the spectrum, and audio - visual feedback via video chat applications at the other end.

[0003] Sensor-based approaches involve placing on-body sensors on various joints and tracking target parameters such as speed, acceleration, and muscle activation. However, such approaches require additional technical expertise when dealing with the sensors and their exact placement. Additionally, they tend to inhibit natural motion and give the user a sense of discomfort. Apart from on-body sensors, common motion capture methods require multiple dedicated cameras calibrated across multiple viewpoints and thus cannot be easily set up at the patient's home for remote evaluation. Tactile-based methods also suffer from the same issues of expensive setups and calibration of multiple devices operating in tandem. Additionally, most of these systems focus on live synchronous versions of evaluations via video chat interactions and do not lead to asynchronous use cases. However, there is an implicit need to go beyond the current standards to provide additional feedback to physicians. The widespread availability of inexpensive RGB+depth (RGB-D) cameras provides a means to offer non-invasive tracking and an enhanced level of feedback without the hassle of calibration. Such cameras utilize the depth stream to provide inference and tracking of human joints without any markers for body segments.

[0004] Using an RGB-D camera, virtual reality systems and applications incorporate 3D human models such as personalized humanoid avatars representing users captured by the RGB-D camera. Virtual reality (VR) refers to a computer-generated environment that enables a user to experience their physical senses and perceptions. VR has numerous applications in countless industries ranging from entertainment and gaming to engineering and medicine. For example, a virtual environment can be a setting for interacting with a personalized avatar or a simulated surgery. A VR experience is rendered to be perceived by physical sensory modalities such as vision, hearing, touch, proprioception, and / or smell. Similarly, augmented reality (AR) refers to a hybrid environment that incorporates elements of the real world, the physical world, and elements of the virtual world. Like VR, AR has numerous applications across many different industries. Due to its complementary nature, AR is well-suited for applications such as gaming, engineering, medical science, tourism, and recreation. The use of 3D human models provides a better sense of immersion for the user to view details such as the dress a person is wearing and the emotions on their face. VR games that utilize these 3D human models can be used to enhance the user experience and assist in the remote evaluation of patients. Furthermore, the use of consumer depth cameras enables a more natural interaction experience with exercise game (exergame) systems.

SUMMARY OF THE INVENTION

[0005] It is understood that both the following general description and the following detailed description are exemplary and explanatory only and are not restrictive.

[0006] A method, system, and apparatus for rendering a personalized humanoid avatar within a virtual environment to assist in evaluating the strength of a user's joints are described. The virtual reality scene may include a personalized humanoid avatar within the virtual environment. The VR device may include one or more sensors that can determine one or more of the position, orientation, location, and / or motion of the user within the virtual environment.

[0007] In one embodiment, the method is to perform real-time camera-skeleton pose calibration based on receiving calibration data from a sensor, where the sensor comprises an RGB-D camera, and to output a user interface in a virtual environment to a display, where the user interface includes a game session control menu, and the game session control menu includes options for outputting a game for engaging the user to interact with at least one virtual object to move at least one specified part of the user's body, and to receive motion data from the sensor, where the motion data includes motion data of the user's movement during the game for engaging with at least one virtual object and the motion data includes joint data, and to output a personalized humanoid avatar of the user in the virtual environment to the display based on the real-time camera-skeleton pose calibration, and to cause the personalized humanoid avatar to execute at least one motion based on applying the motion data to the personalized humanoid avatar, and to track the angle of at least one joint over the duration from the start to the end of the user's movement while the user is engaging with at least one virtual object based on the joint data, and to determine a force estimation model for estimating the force acting on at least one joint while the user is engaging with at least one virtual object during the game based on the tracked angle, and to determine an inference of the user strength associated with at least one joint based on the force estimation model, and to transmit the force estimation model to a server, where the server stores the force estimation model in a database associated with the user.

[0008] Additional advantages will be described in part in the following description or may be learned by practice. The advantages are realized and achieved by the elements and combinations particularly pointed out in the appended claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive.

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, serve to explain the principles of the methods and systems described herein. For ease of readily identifying any particular element or operation, the leading digit or digits of a reference number may refer to the drawing number in which that element was first introduced.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Modes for Carrying Out the Invention

[0011] Before the method and system are disclosed and described, it should be understood that the method and system are not limited to a particular method, particular components, or particular implementations. It should also be understood that the terms used in this specification are for the purpose of describing particular embodiments only and are not intended to be limiting.

[0012] As used in this specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. In this specification, a range may be expressed as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from one particular value and / or to another particular value. Similarly, when values are expressed as approximations by use of the antecedent "about", it will be understood that the particular value forms another embodiment. It is further understood that each endpoint of a range is significant both in relation to the other endpoint and independently of the other endpoint.

[0013] "Optional" or "optionally" means that the subsequent described event or circumstance may or may not occur, and that the description includes both the case where the event or circumstance occurs and the case where it does not occur.

[0014] Throughout the description and claims of this specification, the word "comprise", and variations of the word such as "comprising" and "comprises", means "including, but not limited to", and is not intended to exclude, for example, other components, integers, or steps. "Exemplary" means "an example of" and is not intended to convey an indication of a preferred or ideal embodiment. "Such as" is used for illustrative purposes and not in a limiting sense.

[0015] Components that can be used to execute the disclosed methods and systems are disclosed. When these and other components are disclosed herein and combinations, subsets, interactions, groups, etc. of these components are disclosed, each specific mention of each of these various individual and collective combinations and permutations may not be explicitly disclosed, but each is understood to be specifically contemplated and described herein for all methods and systems. This applies to all aspects of the present application, including but not limited to steps in the disclosed methods. Thus, if there are various additional steps that can be performed, it is understood that each of these additional steps can be performed using any particular embodiment or combination of embodiments of the disclosed method.

[0016] The methods and systems of the present invention can be more readily understood by reference to the following detailed description of the preferred embodiments and the examples included therein, as well as the drawings and the description before and after them.

[0017] As will be understood by those skilled in the art, the methods and systems can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Further, the methods and systems may take the form of a computer program product on a computer-readable storage medium (e.g., non-transitory) having processor-executable instructions (e.g., computer software) embodied therein. More specifically, the methods and systems can take the form of web-implemented computer software. Any suitable computer-readable storage medium can be utilized, including a hard disk, CD-ROM, optical storage device, magnetic storage device, memregister, non-volatile random access memory (NVRAM), flash memory, or combinations thereof.

[0018] Embodiments of the method and system are described below with reference to block diagrams and flowchart diagrams of methods, systems, apparatuses, and computer program products. It will be understood that each block of the block diagrams and flowchart diagrams, as well as combinations of blocks in the block diagrams and flowchart diagrams, can be implemented by computer program instructions. These processor-executable instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that the instructions executed on the computer or other programmable data processing apparatus create means for implementing the functions specified in the block or blocks of the flowchart.

[0019] These processor-executable instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce a product including computer-readable instructions for implementing the functions specified in the block or blocks of the flowchart. The processor-executable instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus, creating a computer-implemented process such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the block or blocks of the flowchart.

[0020] Accordingly, the blocks of the block diagrams and flowchart diagrams support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowchart diagrams, as well as combinations of the blocks of the block diagrams and flowchart diagrams, can be implemented by a dedicated hardware-based computer system that performs the specified functions or steps, or a combination of dedicated hardware and computer instructions.

[0021] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. As used herein, the terms "user" or "subject" can refer to a person who uses an electronic device, or a device that uses an electronic device (e.g., an artificial intelligence electronic device).

[0022] A method and system for generating a personalized humanoid avatar of a user within a virtual environment are described to assist in the synchronous and asynchronous remote intensity evaluation of the user's joints in an interactive augmented reality setting. The VR device may comprise a camera, which can be any imaging device such as, for example, an RGB-D camera, a digital camera, and / or a digital video camera, or can communicate with it. Throughout this specification, VR or the VR device may be referred to. It should be understood that VR and AR are used interchangeably and may refer to the same situation or device. The camera may be associated with a field of view (e.g., a frame) representing the range of the observable world that the camera can image. The VR device can utilize the camera to capture one or more image data while the user performs one or more movements within the field of view, process the image data, and generate motion data of the user's movement, including joint data associated with the user's joints when the user performs different movements. The VR device may comprise a display or communicate with a display. For example, the display device may include a head-mounted device (HMD), a smartphone, a smart mirror, a monitor, a laptop, a tablet, a television, etc. The display can output a user interface that may include a game session control menu. The game session control menu may include options for outputting a game that engages the user to interact with at least one virtual object and engages the user to move a part of the user's body. The VR device can receive motion data generated by the camera while the user is interacting with at least one object during the game. The display can output a personalized humanoid avatar of the user within the virtual environment. The VR device can cause the personalized humanoid avatar to perform at least one motion based on the motion data. The VR device can track the angle of at least one joint while the user is interacting with at least one virtual object.The VR device can determine a force estimation model that estimates the force acting on at least one joint based on the angles of the joints of the tracked user. Using the force estimation model, the VR device can determine an inference of the intensity of the object associated with at least one joint.

[0023] Finally, the VR device can send the joint strength data to the server. The joint strength data can be associated with each user to improve the quality of the user's experience throughout the process and to track the joint strength estimate for each user. For example, the user's doctor can use the user's joint strength estimate to make any further assessments regarding the user's joints.

[0024] In one example, the camera may require calibration to compensate for the rendering perspective of the object interacting with the VR device. The calibration of the camera may include real-time camera skeleton pose and real-time floor calibration to estimate the floor plane of the environment detected in the calibration data.

[0025] The VR device may further include one or more cameras configured to capture an image of the object to generate a real-time 3D personalized humanoid avatar of the object.

[0026] A VR system may include two composite reality scenes that function like an exercise game when providing interactive and engaging elements to a remote evaluation procedure. The main focus may be on the upper body joints, as they are easier to track (unlike the lower body joints such as the ankles) and have a lower potential for noise and occlusion. The VR system may target three upper body joints, the elbow, wrist, and shoulder, for a total of four movements. Cameras may be used for motion tracking and 3D skeleton inference. The VR system may record the user's motion for the specified joints with respect to the range of motion via camera skeleton data and the time required to complete the motion by tracking hand gestures. The VR system can estimate the force values of the motion using an inverse dynamics solver, which can then be sent to virtual objects within the game environment to provide an in-game feedback mechanism to the physician performing the remote evaluation.

[0027] One or more virtual objects of various sizes, shapes, orientations, colors, etc. can be determined according to the VR application. For example, in a VR game application, virtual objects can be determined. Spatial data associated with one or more virtual objects can be determined. The spatial data associated with one or more virtual objects may include data associated with positions (e.g., x, y, z coordinates) in 3D space. For a given virtual object among one or more virtual objects, the position in 3D space may include the position defined by the center of mass of the virtual object and / or the position defined by one or more boundaries (e.g., contour or edge) of the virtual object. The spatial data associated with one or more virtual objects can be aligned with the spatial data associated with the center of the frame. Aligning can refer to determining the position of a given virtual object among one or more virtual objects relative to the position of the center of the frame. Aligning can also refer to the position of the virtual object relative to both the position of the center of the frame and the position of a personalized avatar within the composite reality scene. Aligning the virtual object with either the position of the center of the frame and / or the position of one or more physical objects within the virtual reality scene ensures that the virtual object is displayed at an appropriate scale within the virtual reality scene and does not overlap (e.g., "clip") with any of the one or more virtual objects within the virtual reality scene or the avatar. For example, the spatial data of the virtual object can be aligned with the spatial data of the center of the frame and the spatial data of a table (e.g., one of the one or more physical objects). Such alignment enables the virtual object to be displayed within the virtual reality scene, such that the virtual object appears to be placed on the table and does not overlap (e.g., "clip") with the table.

[0028] The movement of the VR device can cause changes in the virtual reality scene. For example, the VR device can pan, tilt, or rotate in the direction of the VR device. Such movement can affect the virtual reality scene and the personalized avatar, as well as any virtual objects rendered therein. For example, when the VR device is tilted downward, the viewpoint within the virtual environment can rotate downward, similar to a person moving their head downward. Similarly, when the VR device is tilted upward, the viewpoint within the virtual environment can rotate upward, similar to a person moving their head upward. In one example, when the VR device is rotated left or right, the viewpoint within the virtual environment can rotate left or right, similar to a person rotating their head left or right.

[0029] Each of the components described in this document can consist of one or more components, and its name can vary depending on the type of electronic device. Electronic devices according to various embodiments can include at least one of the components described herein. Some of the components may be omitted, and additional other components may be further included. Furthermore, some of the components of the electronic devices according to various embodiments of the present invention can be combined to form one entity and can perform the same functions as the corresponding components before combination.

[0030] FIG. 1 shows an exemplary system 100 that includes an electronic device (e.g., a smartphone or a laptop) configured to control one or more induction systems of one or more other electronic devices (e.g., a headset device or a sensor device) according to various embodiments. The system 100 may include an electronic device 101, a headset 102, one or more sensors 103, and one or more servers 106. The electronic device 101 may include a bus 110, a processor 120, a memory 130, an input / output interface 150, a display 160, and a communication interface 170. In one example, the electronic device 101 may omit at least one of the above-described components or may further include other components. The electronic device 101 may include, for example, a mobile phone, a smartphone, a tablet computer, a laptop, a desktop computer, a smartwatch, and the like.

[0031] The bus 110 may include circuitry for connecting the processor 120, the memory 130, the input / output interface 150, the display 160, and the communication interfaces 170 to each other and for transmitting communication (e.g., control messages and / or data) between the processor 120, the memory 130, the input / output interface 150, the display 160, and the communication interfaces 170.

[0032] The processor 120 can include one or more of a central processing unit (CPU), an application processor (AP), and a communication processor (CP). The processor 120 may control, for example, at least one of the processor 120, the memory 130, the input / output interface 150, the display 160, and the communication interface 170 of the electronic device 101, and / or may perform operations or data processing for communication. The processing (or control) operations of the processor 120 according to various embodiments will be described in detail with reference to the following drawings. For example, the processor 120 may be configured to cause the headset device 102 to output a virtual reality game such as the virtual reality program 147 stored in the memory 130 to the user.

[0033] The memory 130 may include volatile and / or non-volatile memory. The memory 130 can store, for example, instructions or data related to at least one different component of the electronic device 101. In one example, the memory 130 can store software and / or a program 140. The program 140 can include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or a virtual reality program (e.g., an "application") 147 configured to control one or more functions of the electronic device 101 and / or an external device (e.g., the headset 102 and / or one or more sensors 103). At least one part of the kernel 141, the middleware 143, or the API 145 may be referred to as an operating system (OS). The memory 130 can include a computer-readable recording medium on which a program for performing a method according to various embodiments is recorded by the processor 120.

[0034] The kernel 141 can be used to control or manage system resources (such as the bus 110, the processor 120, the memory 130, etc.) for executing operations or functions implemented in other programs (such as the middleware 143, the API 145, or the virtual reality program 147). Furthermore, the kernel 141 can provide an interface through which the middleware 143, the API 145, or the virtual reality program 147 can access individual components of the electronic device 101 to control or manage system resources.

[0035] The middleware 143 can, for example, perform a mediation role so that the API 145 or the virtual reality program 147 can communicate with the kernel 141 to exchange data. In addition, the middleware 143 can process one or more task requests received from the virtual reality program 147 according to priorities. For example, the middleware 143 can allocate the priority of using system resources (such as the bus 110, the processor 120, or the memory 130) of the electronic device 101 to at least one of the virtual reality programs 147. For example, the middleware 143 can process one or more task requests according to the priorities assigned to at least one of the application programs, and thus can perform scheduling or load balancing for the one or more task requests.

[0036] The API 145 can include, as an interface for controlling functions provided by the virtual reality program 147 within the kernel 141 or the middleware 143, at least one interface or function (such as an instruction) for file control, window control, video processing, or character control.

[0037] The virtual reality program 147 may include two composite reality scenes that function like an exercise game when providing an interactive engagement element to the remote evaluation procedure. The main focus may be on the joints of the upper body, which are easier to track (unlike the joints of the lower body such as the ankles) and have a lower potential for noise and occlusion. The VR system may target three upper body joints, the elbow, wrist, and shoulder, for a total of four movements. The sensor device 103 may be used for motion tracking and 3D skeleton inference. For example, the electronic device 101 may receive motion data from the sensor device 103 while the subject is interacting with the virtual reality program 147. The virtual reality program 147 may record the user's motion with respect to the defined joints in terms of the range of motion and the time required to complete the motion, by tracking hand gestures based on data received from the sensor device 103 (e.g., sensor skeleton data). The virtual reality program 147 can estimate the value of the motion force using an inverse dynamics solver, which can then be transmitted to virtual objects within the game environment to provide an in-game feedback mechanism to the physician performing the remote evaluation.

[0038] The input / output interface 150 can serve the role of an interface for transmitting commands or data input from the user or different external device(s) to different components of the electronic device 101. Further, the input / output interface 150 can output commands or data received from different components of the electronic device 101 to different external devices.

[0039] The display 160 may include various types of displays, such as a liquid crystal display (LCD) display, a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 can, for example, display various contents (such as text, images, videos, icons, symbols, etc.) to the user. The display 160 may include a touch screen. For example, the display 160 can receive touch, gesture, proximity, or hovering inputs using a stylus pen or a part of the user's body.

[0040] The communication interface 170 can establish communication, for example, between the electronic device 101 and an external device (such as the headset 102, the sensor device 103, or the server 106). For example, the communication interface 170 may communicate with an external device (such as the server 106) by being connected to the network 162 via wireless communication or wired communication. In one example, as a cellular communication protocol, the wireless communication can use at least one of Long-Term Evolution (LTE), LTE-Advanced (LTE-A), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), Global System for Mobile Communications (GSM), etc. Further, the wireless communication may include, for example, short-range communications 164, 165. The short-range communications 164, 165 may include at least one of, for example, Wireless Fidelity (WiFi), Bluetooth, Near Field Communication (NFC), Global Navigation Satellite System (GNSS), etc. GNSS can include at least one of, for example, Global Positioning System (GPS), Global Navigation Satellite System (Glonass), Beidou Navigation Satellite System (hereinafter, "Beidou"), Galileo, which is the European Global Satellite Navigation System, etc., depending on the usage area or bandwidth, etc. Hereinafter, "GPS" and "GNSS" may be used interchangeably in this document. The wired communication can include at least one of, for example, Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), Recommended Standard 232 (RS-232), Powerline Communication, Plain Old Telephone Service (POTS), etc. The network 162 may include at least one of, for example, a telecommunications network, a computer network (such as a LAN or a WAN), the Internet, and a telephone network.

[0041] The headset 102 may comprise a head-mounted display (HMD) device that may include an optical element that can selectively turn on or off the view of the external environment in front of a person's eyes. The headset 102 may be configured to execute a virtual reality program 147. Using various sensors and modules, the headset can perform real-time camera-skeleton pose and real-time floor calibration to estimate the floor plane of the detected environment in the initial data. In one example, the subject can adjust their position to interact with virtual objects within the virtual environment in order to calibrate the sensors and / or the headset 102. Based on the calibration, the headset 102 can output a personalized humanoid avatar of the subject. In one example, the headset 102 may comprise a display device such as a television, monitor, laptop, or tablet.

[0042] As an example, the movement of the headset 102 may cause a change in the virtual reality scene. For example, the headset 102 can pan, tilt, or rotate in the direction of the headset 102. Such movement may affect the virtual reality scene and the personalized avatar, as well as any virtual objects rendered therein. For example, when the headset 102 is tilted downward, the viewpoint within the virtual environment can rotate downward in the same way that a person shifts their head downward. Similarly, when the headset 102 is tilted upward, the viewpoint within the virtual environment can rotate upward in the same way that a person moves their head upward. In one example, when the headset 102 is rotated left or right, the viewpoint within the virtual environment can rotate left or right in the same way that a person rotates their head left or right.

[0043] In one example, the headset 102 can use various sensor devices and modules to detect objects or obstacles in front of the subject when the subject is executing a game via the headset 102. The headset 102 can be configured to warn the subject of any potential objects that can pose a safety risk to the subject while the subject is making movements during gameplay using the headset 102. For example, an object such as a pet or a person can enter within the distance of the subject's movement, causing a safety issue. The headset 102 can be configured to use an object detection module for detecting objects (such as pets, people, etc.) within the radius of the headset 102. When an object (such as a pet, a person, etc.) moves into the field of view of the sensors of the headset 102, an alarm can be triggered to warn the subject about the object. For example, the headset 102 can output a pop-up panel and detection results within the image stream. In one example, a predicted bounding box identifying the object for the subject within the image stream output to the subject by the headset 102 can be output.

[0044] The sensor device 103 can include one or more imaging devices or image capture devices, such as one or more RGB-D cameras (e.g., one or more Kinect cameras). The sensor device 103 may use a combination of sensors (e.g., one or more sensors) to identify the user and provide information associated with the identified user to the electronic device 101. In one example, the sensor device 103 can be configured to detect motion data associated with the user and provide the motion data to the electronic device 101. In one example, based on receiving calibration data from the sensor device 103, the electronic device 101 can perform calibration of the virtual environment.

[0045] In one example, the sensor device 103 can be configured to detect an object or obstacle in front of the subject when the subject is executing a game via the headset 102. The sensor device 103 can be configured to warn the subject of any potential object that can pose a safety risk to the subject while the subject is executing movements while playing a game using the headset 102. For example, an object such as a pet or a person can cause a safety problem while the subject is executing one or more movements (e.g., moving the lower limbs) while using the headset 102. For example, the object can be invisible to the sensors of the headset 102. Thus, it can be difficult for the headset 102 to detect any potential dangerous object that enters the range of motion of the subject while the subject is using the headset 102. The sensor device 103 can be configured to use an object detection module to detect objects (e.g., pets, people, etc.) within the radius of the headset 102. For example, when an object (e.g., a pet, a person, etc.) moves into the field of view of the sensor device 103, an alarm can be triggered to warn the subject of the object. For example, the headset 102 can output a pop-up panel and the detection result in the image stream. In one example, a predicted bounding box that identifies an object for the subject in the image stream output to the subject by the headset 102 can be output.

[0046] Server 106 may include a group of one or more servers. In one example, all or part of the operations performed by electronic device 101 may be performed on one or more different electronic devices (e.g., headset 102, sensor device 103, or server 106). For example, if electronic device 101 needs to perform a particular function or service automatically or on demand, instead of performing the function autonomously, electronic device 101 can alternatively or additionally request at least a portion of the related functions from a different electronic device (e.g., headset 102, sensor device 103, or server 106). The different electronic devices (e.g., headset 102, sensor device 103, or server 106) can perform the requested function or additional functions and can communicate the results to electronic device 101. Electronic device 101 may directly provide the requested function or service, or may additionally process and provide the received results. In one example, cloud computing, distributed computing, or client-server computing techniques can be used. For example, electronic device 101 can provide calibration data received from sensor device 103 to server 106, and server 106 can perform a calibration operation and return the results to electronic device 101.

[0047] Figure 2 shows an exemplary system 200. The system 200 can include various components that can communicate with some or other or all of the components. Figure 2 shows an exemplary system 200 in which an electronic device 101 is communicating with a headset 102, a sensor device 103, and a server 106. The electronic device 101, as well as the headset 102 and the sensor device 103, can be communicatively coupled through short-range wireless communication technologies 164, 165 (e.g., Bluetooth Low Energy or WiFi). The electronic device 101 can be communicatively coupled to the server 106 via a network 162. The electronic device 101 can determine location information. For example, the electronic device 101 may include a GPS sensor. The GPS sensor on the electronic device 101 can determine location information (e.g., GPS coordinates) and transmit the location information to the server 106.

[0048] The headset 102 can transmit data to the electronic device 101. The electronic device 101 can determine image data, geographical data, orientation data, etc. through various sensors. The electronic device 101 can further transmit the data to the server 106.

[0049] For example, system 200 may include an electronic device 101, a headset 102, a sensor device 103, and a server 106 according to various embodiments of the present disclosure. The electronic device 101 and the headset 102 may be communicatively coupled to the server 106 via a network 162. In one example, the electronic device 101 may include a display 210, a housing (or body) 220 to which the display 210 is coupled in a mounted state, and additional devices formed on the housing 220 to perform functions of the electronic device 101. In one example, the additional devices may include a first speaker 202, a second speaker 203, a microphone 205, sensors (e.g., a front camera module 207, a rear camera module, an illuminance sensor 209, etc.), a communication interface (e.g., a charging or data input / output port 211 and an audio input / output port 213), and buttons 215. As an example, when the electronic device 101 and the headset 102 are connected by a wired communication method, the electronic device 101 and the headset 102 may be connected based on at least some ports of the communication interface (e.g., the data input / output port 211).

[0050] In one example, the display 210 may include a flat display or a bent display (or a curved display) that can be folded or bent through a substrate as thin as paper or flexible without damage. The bent display may be coupled to the housing 220 so as to remain in a bent form. In one example, the mobile device 201 may be implemented as a display device that can be folded and unfolded very freely, such as a flexible display including a bent display. In one example, in a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, or an active matrix OLED (AMOLED) display, the display 210 can be made flexible to be folded or unfolded by replacing the glass substrate surrounding the liquid crystal with a plastic film.

[0051] Figure 3 shows exemplary movements. A Virtual Remote Telephysical Examination (VIRTePEX) system can consist of two composite reality scenes that function like an exercise game providing interactive and engaging elements to a remote assessment procedure. The main focus may be on the upper body joints as they are easier to track (unlike the lower body joints such as the ankles) and have a lower potential for noise and occlusion. As shown in Figure 3, the VIRTePEX system can target three upper body joints, the elbow, wrist, and shoulder, for a total of four movements. Cameras can be used for motion tracking and 3D skeleton inference. The VIRTePEX system can record a user's motion for the specified joints with respect to the range of motion via camera skeleton data and the time required to complete the motion by tracking hand gestures. The VIRTePEX system can use an inverse dynamics solver to estimate the force values of the motion and then transmit this to virtual objects within the game environment to provide an in-game feedback mechanism to the physician performing the remote assessment.

[0052] The VIRTePEX system may use a sensor device 103, such as a Kinect v2 depth camera, to track the movement of a target joint. The depth camera provides real-time 3D skeleton information of the target, which is useful for tracking and inferring parameters such as relative joint positions and angles. The force / torque on the joint can be estimated when the target executes a specified movement. Using the range and duration of the joint movement from a specified start position to an end position, the force required to execute the observed movement can be estimated (up to a constant). For example, this procedure may include using a depth camera to capture the motion of a human body and tracking different joints and related angles over a specified time interval. The depth camera may not directly provide the values of joint angles. However, the values of joint angles can be estimated using a simple dot product between the orientation vectors of limb segments. Then, using the joint angles, kinematic quantities such as angular velocity and acceleration required for estimating the force and torque provided by the inverse dynamics equations can be calculated.

[0053] Since all the movements considered are essentially rotations and the overall movement consisting of angular displacements at the joints has only one degree of freedom, the joints can be modeled as one-link revolute joints. The raw force estimates may not be comparable separately from the ground truth values, but they are consistent over movements corresponding to different intensity levels. Considering three upper joints of the elbow, shoulder, and wrist, as shown in Figure 3, and assuming all joints have flexion / extension movements and the shoulder joint has abduction movement. From the skeleton provided by a depth camera as shown in Figure 4, which includes the 3D position data of the tracked body joints, the joint angles can be calculated using the dot products generated between the 3D position vectors.

[0054] Figures 5A - 5B illustrate exemplary scenarios. Figure 5A shows an exemplary scene with a user that can be captured by the sensor device 103. Figure 5B shows an exemplary virtual reality scene (e.g., environment) with a personalized humanoid avatar of the user when the user is being captured by the sensor device 103. As shown in Figure 5B, the user can experience the virtual reality scene through the use of a headset device. Figure 5B shows a user participating in an exercise game (e.g., bowling) when the VIRTePEX system estimates the value of the user's motion force during the game play. For example, the exercise game can include a bowling game consisting of a virtual bowling alley scene where balls and bowling pins of variable mass are arranged side by side along a path. The variable pin weights can provide a means to evaluate the user in a standardized way based on the user's ability to knock down the pins from a predetermined set of virtual weights. The bowling scene can enable the system to target the upper body joints. The main purpose of the user can be to perform the movement at their comfortable level and observe whether the estimated force generated from the action can knock down the pins. During the user's execution of the activity, the joint angles and time from start to end can be tracked via a depth camera. At the end of the game session, the tracked joint data can be provided as input to an inverse dynamics solver to estimate the force that can be applied to the ball along the direction towards the pins. The sensor device 103 can also generate motion data associated with the user's movement that can be applied to the personalized humanoid avatar to simulate the user playing the game in a virtual environment.

[0055] Figure 6 shows an exemplary scenario in which a user participates in a game using a display device (e.g., a television). Figure 6 shows a virtual reality scene (e.g., an environment) being displayed from the display device (e.g., a television). The user can participate in an exercise game (e.g., bowling) within the comfort of the user's living space such as a living room or a bedroom. The VIRTePEX system can utilize the sensor device 103 to estimate the value of the force associated with the user's joints while the user is interacting with the objects displayed during the game. Similar to what is shown in Figure 5B, the user can participate in an exercise game including a bowling game consisting of a virtual bowling alley scene where a ball and variable mass bowling pins are arranged side by side along a path. The user can perform movements at their own comfortable level and observe whether the estimated force generated from the action can knock down the pins. While the user is performing the activity, the joint angles and time can each be tracked via a camera. At the end of the game session, the tracked joint data can be provided as an input to an inverse dynamics solver to estimate the force applied to the ball along the direction towards the pins. The sensor device 103 can capture the user's movements and generate motion data associated with the user's movements. The motion data can then be applied to a personalized humanoid avatar to simulate the user playing the game in a virtual environment.

[0056] FIG. 7 shows an exemplary scenario in which a user can participate in a virtual exercise game while being remotely evaluated by another person such as the user's physician. The user's physician can interact with the user remotely in real time while the user is participating in the virtual exercise game in order to provide an evaluation to the user in real time. In one example, the user can interact with another user who is at the opposite end of the bowling alley. Both users can perform the required joint movements, and the motion is imparted to each ball based on respective force estimates, with the goal of overcoming the other user's ball when crossing the midpoint and observing it in the resulting collision between the balls. In such an evaluation, the physician can determine a judgment regarding joint strength on a broad scale of 3 to 5 levels according to the physical resistance that the user can overcome.

[0057] The user's physician may also interact with the user in an asynchronous manner, and the user's participation in the exercise game may be recorded and uploaded to a server. The user's physician can retrieve the user's record in order to evaluate the user's participation in the virtual exercise game. Further, the record can provide a means for the physician to compare and benchmark joint strength data against other users. In one example, the user may be provided with two options, namely, to record their movements for future play or to play against the record of another user. During the recording mode, the user can perform the specified joint movements over a plurality of sessions, and during each session, the estimated force can be recorded. In the non-recording mode, the user first specifies the identifier of the opponent user. The task may include attempting to overcome an opposing ball that is a given force taken from the data recorded during the opponent's record. The asynchronous option enables the user to play against their previous level or a more personalized benchmark level set by the physician. The asynchronous option may further enable the user to train against their previous level or a personalized benchmark level until they can achieve the muscle strength required to exert the necessary force.

[0058] Figure 8 shows an exemplary process. Hand gesture control can be utilized to control the system interface. For example, hand gesture control provided by a depth camera may be used to track the intention of a subject to start and end an activity session. The user may signal the start and end of the tracking of an activity (e.g., elbow joint flexion) by opening and closing the hand, as shown in Figure 8. Hand gesture-based control can include the mapping used in the virtual scene and can correspond to real-world actions of holding and releasing a ball. Visual cues within the virtual scene can be provided to address any potential problems that may occur to a subject who does not know whether their gesture is registered. For example, the color of a virtual object (e.g., a bowling ball) may be toggled when the start of an activity is detected.

[0059] Figure 9 shows a flowchart of an exemplary method 900. Method 900 may be implemented in whole or in part by one or more of electronic device 101, headset 102, sensor device 103, server 106, or any other suitable device. In step 910, the display can output the user's avatar within the virtual environment based on sensor calibration. For example, electronic device 101 can generate a personalized humanoid avatar representing the user that can be output within the virtual environment displayed by headset 102. For example, the process may include creating skeletal joints and associated texture information. The display may include at least one of a head-mounted display, a television, a monitor, a laptop, or a tablet. In one example, the display may be further configured to output a game (e.g., a virtual exercise game) that engages the user to interact with virtual objects within the virtual environment. For example, interacting with virtual objects within the virtual environment may further include causing the user to perform one or more movements.

[0060] As an example, when the user's skeletal joints are fully detected, the calibration of the sensor can be executed to estimate a floor plane for calibrating the coordinates between the front surface of the sensor device 103 and the virtual environment. For example, in order to estimate the floor plane of the detected environment in the initial data, real-time camera skeletal pose and real-time floor calibration can be performed. For calibration purposes, it can be assumed that the user is standing or sitting in an upright position. This assumption can suggest that the estimated joints corresponding to the spine are distributed around a vertical tendency. Therefore, this vertical tendency can be used as the floor normal for floor calibration. Further, in order to ensure that the target user remains in a normal sitting or standing position during calibration, the system can also track the shoulder height and knee angle. In one example, the user can adjust their position to interact with the virtual object in order to calibrate the sensor and / or the headset / display 102.

[0061] As an example, orientation data can be determined. The orientation data can be associated with the headset 102. For example, the orientation data can include an indication of the 3D orientation of the headset 102. The orientation data can be determined based on the location of the center of the field of view of the headset 102. The orientation data can include an indication of the 3D orientation (such as yaw, pitch, roll, etc.) of the device. In one example, the orientation data can be determined by a sensor module included in the headset 102, such as a magnetic sensor, a gyro sensor, an accelerometer, or any combination thereof. In one example, the orientation can be determined based on the data received by the sensor device 103. In one example, the orientation data can be associated with a display device instead of the headset 102.

[0062] In step 920, motion data of the user's movement can be received. For example, the electronic device 101 may receive motion data from the sensor device 103 when the user interacts with various virtual objects in the virtual environment or during a virtual exercise game. The sensor device 103 may include an RGB-D camera for capturing an image of the user for motion tracking and 3D skeleton inference. Based on the captured image, the sensor device 103 can generate motion data. For example, the captured image can be associated with the joints of the upper body because they are easier to track (unlike the joints of the lower body such as the ankles) and have a lower possibility of noise and occlusion. As an example, the motion data may be captured for three upper body joints, the elbow, wrist, and shoulder, for a total of four movements. The range of motion of the specified joints can be tracked. For example, the motion data may include joint data associated with at least one joint of the user. For example, the joint data may be associated with the range of motion of the specified joint and the time required to complete the motion via the camera skeleton data. For example, the value of the joint angle can be estimated using the simple dot product between the orientation vectors of the limb segments. Then, using the joint angle, kinematic quantities such as angular velocity and acceleration required for the estimation of the forces and torques provided by the inverse dynamics equation can be calculated.

[0063] As an example, the electronic device 101 may apply the motion data to a personalized humanoid avatar. The electronic device 101 may cause the personalized humanoid avatar output by the headset 102 to execute movements according to the movements captured by the sensor device 103 in real time as the user moves. The motion data may be applied to the personalized humanoid avatar to simulate the user playing a game in the virtual environment.

[0064] In step 930, force information can be determined based on joint data. As an example, the angle of at least one joint can be tracked based on joint data while the user is interacting with a virtual object. The force information can be determined based on the tracked angle. The force information can include an estimation of the force acting on at least one joint while the user is interacting with the virtual object during the game. For example, the electronic device 101 can use an inverse dynamics solver to estimate the value of the force of the motion, and then this force value is transmitted to the virtual object within the game environment to provide an in-game feedback mechanism. For example, the feedback may be provided to a doctor who performs a remote evaluation of the user. The electronic device 101 can use the motion data including the tracked joint data as an input to the inverse dynamics solver to estimate the force and torque applied to the virtual object when the user interacts with the virtual object during an exercise game session. In one example, the estimated force may be applied to the virtual object.

[0065] As an example, second motion data associated with the movement of the user can be received from a second sensor. The second motion data can include second joint data associated with at least one joint of the second user. Second force information can be determined based on the second joint data. The second force information can include an estimation of the force acting on at least one joint of the second user while the second user is interacting with a second virtual object within the virtual environment. In one example, the second virtual object can be made to overcome the virtual object based on the force information and the second force information. In one example, the virtual object can be made to overcome the virtual object based on the force information and the second force information.

[0066] As an example, motion data and data associated with a user interacting with virtual objects within a virtual environment may be output to a communication network for remote access. For example, force information may further include a force perception association with the output of motion data and data associated with a user interacting with virtual objects within the virtual environment. As an example, data including force information may be transmitted to a server. The server can associate the data with the user and store the data associated with the user in a database.

[0067] In step 940, a user strength associated with at least one joint may be determined based on the force information. For example, the electronic device 101 can determine a user strength associated with at least one joint based on the force information. As an example, data associated with a user strength associated with at least one joint may be transmitted to the server as joint strength data. The joint strength data may be associated with the user to improve the quality of the user's experience throughout the process and to track the user's joint strength estimation. For example, the user's doctor can use the user's joint strength data to perform a further evaluation of the user's joints.

[0068] FIG. 10 shows a block diagram of a headset device 102 according to various exemplary embodiments. The headset device 102 may include one or more processors (e.g., an application processor (AP)) 1010, a communication module 1020, a subscriber identification module 1024, a memory 1030, a sensor module 1040, an input unit 1050, a display 1060, an interface 1070, an audio module 1080, a camera module 1091, a power management module 1095, a battery 1096, an indicator 1097, and a motor 1098. The camera module 1091 may include an aperture configured for changing the focus.

[0069] The processor 1010 can drive an operating system, an application program, etc., control a plurality of hardware or software components connected to the processor 1010, process various data including multimedia data, and execute operations (e.g., distance calculation). For example, the processor 1010 can be configured to generate a personalized humanoid avatar of a target and place the personalized humanoid avatar in a virtual reality scene, e.g., in the composite reality scene shown in FIGS. 5B and 6. The processor 1010 may be implemented, for example, as a system-on-chip (SoC). According to one exemplary embodiment, the processor 1010 may further include a graphics processing unit (GPU) and / or an image signal processor (ISP). The processor 1010 can include at least one part of the components of FIG. 10 described above (e.g., the cellular module 1021). The processor 1010 may load instructions or data (e.g., the composite reality sense program 147) received from at least one of different components (e.g., non-volatile memory) into volatile memory for processing, and store various data in non-volatile memory. The processor can receive inputs such as sensor readings and execute the augmented reality program 147 accordingly, for example, by adjusting the position of virtual objects in the augmented reality scene. For example, the processor 1010 can adjust the position and orientation of a personalized humanoid avatar in a virtual environment.

[0070] The communication module 1020 may include, for example, a cellular module 521, a Wi-Fi module 1023, a Bluetooth (BT) module 1025, a GNSS module 1027 (e.g., a GPS module, a Glonass module, a Beidou module, or a Galileo module), a Near Field Communication (NFC) module 1028, and a Radio Frequency (RF) module 1029. The communication module can receive data from the electronic device 101, the sensor device 103, and / or the server 106. The communication module can transmit data to the electronic device 101 and / or the server 106. In an exemplary configuration, the headset device 102 can transmit data determined by the sensor module 1040 to the electronic device 101 and / or the server 106. For example, the headset device 102 can transmit data collected by the sensor module 1040 to the electronic device 101 via the BT module 1025.

[0071] The cellular module 1021 can provide, for example, voice calls, video calls, text services, Internet services, etc. via a communication network. According to one exemplary embodiment, the cellular module 1021 can identify and authenticate the headset device 102 within the network 162 by using a subscriber identification module (e.g., a Subscriber Identification Module (SIM) card) 1024. According to one exemplary embodiment, the cellular module 1021 can execute at least some of the functions that may be provided by the processor 1010. According to one exemplary embodiment, the cellular module 1021 can include a communication processor (CP).

[0072] Each of the WiFi module 1023, the BT module 1025, the GNSS module 1027, or the NFC module 1028 may include, for example, a processor for processing data transmitted / received via the corresponding module. According to certain exemplary embodiments, at least a portion (e.g., two or more) of the cellular module 1021, the WiFi module 1023, the BT module 1025, the GPS module 1027, and the NFC module 1028 may be included in one integrated chip (IC) or IC package. The GPS module 1027 can communicate with the electronic device 101, the server 106, or some other location data service via the network 162 to determine location information, such as GPS coordinates.

[0073] The RF module 1029 can transmit / receive, for example, communication signals (e.g., radio frequency (RF) signals). The headset device 102 can transmit and receive data from the mobile device via the RF module 1029. Similarly, the headset device 102 can transmit and receive data from the server 106 via the RF module 1029. The RF module can transmit a request for location information to the server 106. The RF module 1029 can include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, a low noise amplifier (LNA), an antenna, etc. According to another exemplary embodiment, at least one of the cellular module 1021, the WiFi module 1023, the BT module 1025, the GPS module 1027, and the NFC module 1028 can transmit and receive RF signals via a separate RF module.

[0074] The subscriber identification module 1024 can include, for example, a card including the subscriber identification module and / or an embedded SIM, and can include unique identification information (e.g., integrated circuit card identifier (ICCID)) or subscriber information (e.g., international mobile subscriber identity (IMSI)).

[0075] Memory 1030 (e.g., memory 130) may include, for example, internal memory 1032 or external memory 1034. Internal memory 1032 can include, for example, at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.) and non-volatile memory (e.g., one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash memory, NOR flash memory, etc.), hard drive, or solid state drive (SSD), etc.).

[0076] External memory 1034 can further include flash drives such as CompactFlash (CF), Secure Digital (SD), Micro Secure Digital (Micro-SD), Mini Secure Digital (Mini-SD), Extreme Digital (xD), Memory Stick, etc. External memory 1034 can be operably and / or physically connected to the headset device 102 via various interfaces.

[0077] The sensor module 1040 can, for example, measure a physical quantity or detect the operating state of the headset device 102, and can convert the measured or detected information into an electrical signal. The sensor module 1040 can include, for example, at least one of a gesture sensor 1040A, a gyro sensor 1040B, a pressure sensor 1040C, a magnetic sensor 1040D, an acceleration sensor 1040E, a grip sensor 1040F, a proximity sensor 1040G, a color sensor 1040H (e.g., a red, green, blue (RGB) sensor), a bio sensor 1040I, a temperature / humidity sensor 1040J, an illuminance sensor 1040K, an ultraviolet (UV) sensor 1040M, an ultrasonic sensor 1040N, and an optical sensor 1040P. The proximity sensor 1040G can include LIDAR, radar, sonar, time of flight, infrared, or other proximity sensing technologies. The gesture sensor 1040A can determine gestures associated with the headset device 102. For example, when the headset device 102 moves within a mixed reality scene, the headset device 102 can move in a specific manner to perform a game action, for example. The gyro sensor 1040B can be configured to determine the operation of the headset device 102 in space. For example, when the headset device 102 is located on the user's head, the gyro sensor 1040B can determine that the user has rotated the user's head to a certain extent. Thus, the gyro sensor 1040B can communicate the degree of rotation to the processor 1010 to adjust the mixed reality scene by a specific number of degrees and, accordingly, maintain the position of a personalized avatar or virtual object, such as rendered within the mixed reality scene. The proximity sensor 1040G can be configured to use sonar, radar, LIDAR, or any other suitable means to determine the proximity between the headset device and one or more physical objects. The ultrasonic sensor 1040N can also be similarly configured to employ sonar, radar, LIDAR, time of flight, etc. to determine distance. The ultrasonic sensor can emit and receive acoustic signals and convert the acoustic signals into electrical signal data.Electrical signal data is communicated to the processor 1010 and can be used to determine any of image data, spatial data, etc. According to one exemplary embodiment, the optical sensor 1040P can detect light reflected by ambient light and / or an external object (e.g., a user's finger, etc.) and converted to a specific wavelength band by a light conversion member. Additionally or alternatively, the sensor module 1040 can include, for example, an E-nose sensor, an electromyogram (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an iris sensor, and / or a fingerprint sensor. The sensor module 1040 can further include a control circuit for controlling at least one or more sensors included therein. In a specific exemplary embodiment, the headset device 102 can further include a processor configured to control the sensor module 1004 separately or as a part of one of the processors 1010, and can control the sensor module 1040 while the processor 1010 is in a sleep state.

[0078] The input device 1050 can include, for example, a touch panel 1052, a (digital) pen sensor 1054, keys 1056, or an ultrasonic input device 1058. The touch panel 1052 can recognize touch input using, for example, at least one of electrostatic, pressure-sensitive, and ultrasonic types. Additionally, the touch panel 1052 can further include a control circuit. The touch panel 1052 can further include a tactile layer and can provide a tactile response to the user.

[0079] (Digital) pen sensor 1054 can be, for example, a part of the touch panel or can include an additional sheet for recognition. Keys 1056 can be, for example, physical buttons, optical keys, keypads, or touch keys. The ultrasonic input device 1058 can detect ultrasonic waves generated from the input means via a microphone (e.g., microphone 1088) and can confirm data corresponding to the detected ultrasonic waves.

[0080] The display 1060 (e.g., the display 1060) may include a panel 1062, a hologram unit 1064, or a projector 1066. The panel 1062 may include a structure identical or similar to the display 210 in FIG. 2. The panel 1062 may be implemented in a flexible, transparent, or wearable manner, for example. The panel 1062 may be configured as one module together with the touch panel 1052. According to one exemplary embodiment, the panel 1062 may include a pressure sensor (or force sensor) capable of measuring the intensity of the pressure on the user's touch. The pressure sensor may be implemented integrally with the touch panel 1052 or as one or more sensors separated from the touch panel 1052.

[0081] The hologram unit 1064 can display a three-dimensional image in the air using the interference of light. The projector 1066 can display an image by projecting a light beam onto a screen. The screen can be located, for example, inside or outside the headset device 102. According to one exemplary embodiment, the display 1060 may further include a control circuit for controlling the panel 1062, the hologram unit 1064, or the projector 1066.

[0082] The display 1060 can display a real-world scene and / or a mixed reality scene. The display 1060 can receive image data from the processor 1010 captured by the camera module 1091. The display 1060 can display the image data. The display 1060 can display one or more physical objects. The display 1060 may display one or more virtual objects such as virtual balls, virtual animals, virtual furniture, etc. The user may interact with one or more virtual objects, and the user may adjust their position within the virtual environment and reach for the virtual objects as needed.

[0083] Interface 1070 can include, for example, a High-Definition Multimedia Interface (HDMI) 1072, a Universal Serial Bus (USB) 1074, an optical communication interface 1076, or a D-subminiature (D-sub) 1078. Interface 1070 can be included, for example, in the communication interface 170 of FIG. 1. Additionally or alternatively, interface 1070 can include, for example, a Mobile High-Definition Link (MHL) interface, a Secure Digital (SD) / Multimedia Card (MMC) interface, or an Infrared Data Association (IrDA) standard interface.

[0084] Audio module 1080 can, for example, convert sound and electrical signals bidirectionally. At least some components of audio module 1080 may be included, for example, in the input / output interface 150 of FIG. 1. Audio module 1080 can convert sound information input or output via, for example, a speaker 1082, a receiver 1084, earphones 1086, a microphone 1088, etc.

[0085] Camera module 1091 is, for example, a device for capturing images and videos. According to one exemplary embodiment, it can include one or more image sensors (e.g., a front sensor or a rear sensor), a lens, an Image Signal Processor (ISP), or a flash (e.g., an LED or a xenon lamp). Camera module 1091 can include a front camera for capturing scenes. Camera module 1091 may also include a rear camera for capturing eye movement or changes in the line of sight.

[0086] The power management module 1095 can manage the power of, for example, the headset device 102. According to one exemplary embodiment, the power management module 1095 may include a power management integrated circuit (PMIC), a charger integrated circuit (IC), or a battery gauge. The PMIC can have wired and / or wireless charging types. Examples of wireless charging methods may include, for example, magnetic resonance method, magnetic induction method, electromagnetic method, etc., and further, additional circuits for wireless charging such as coil loops, resonant circuits, rectifiers, etc. may also be included. The battery gauge may measure, for example, the remaining amount of the battery 1096, the voltage, current, and temperature during charging, etc. The battery 1096 may include, for example, a rechargeable battery and / or a solar battery.

[0087] The indicator 1097 can display a specific state of the headset device 102 or one of its parts (e.g., the processor 1010), such as a booting state, a message state, a charging state, etc. The motor 1098 may convert an electrical signal into mechanical vibration and may generate vibration or a tactile effect. Although not shown, the headset device 102 may include a processing device (e.g., a GPU) for supporting mobile TV. The processing device for supporting mobile TV can process media data compliant with protocols such as, for example, Digital Multimedia Broadcasting (DMB), Digital Video Broadcasting (DVB), MediaFlo (trademark), etc.

[0088] For purposes of illustration, application programs and other executable program components are shown herein as individual blocks, but it is recognized that such programs and components can exist on different storage components at different times. Implementations of the described methods can be stored on or transmitted via some form of computer-readable medium. Any of the disclosed methods can be executed by computer-readable instructions embodied on a computer-readable medium. A computer-readable medium can be any available medium that can be accessed by a computer. By way of example and not limitation, a computer-readable medium can include "computer storage media" and "communication media". "Computer storage media" can include volatile and nonvolatile, removable and nonremovable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Exemplary computer storage media can include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer.

[0089] The methods and systems are described in connection with preferred embodiments and specific examples, but the embodiments herein are intended to be illustrative rather than limiting in all respects, and thus are not intended to be limited to the specific embodiments described.

[0090] Unless otherwise specified, no method described in this specification is ever intended to be construed as requiring that its steps be performed in a particular order. Thus, where a method claim does not actually recite the order in which its steps are to be followed, or where the specification or claims do not specifically state that the steps are to be limited to a particular order, no inference of order is ever intended, in any respect. This applies to any possible implicit basis for interpretation, including logical issues regarding the structure of steps or the flow of operations, the plain meaning derived from grammatical construction or punctuation, and the number or types of embodiments described in this specification.

[0091] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit of the invention. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice disclosed herein. The specification and examples are to be considered as illustrative only, and the true scope and spirit are intended to be indicated by the following claims.

Claims

1. A method comprising: outputting, based on calibration of a sensor, an avatar of a user in a virtual environment to a display; receiving, from the sensor, motion data of the user's movement, the motion data including joint data associated with at least one joint of the user; determining force information based on the joint data; determining a user strength associated with the at least one joint based on the force information.

2. The method according to claim 1, wherein the display is a head-mounted display (HMD).

3. The method according to claim 1, wherein the display is a display device comprising at least one of a television, a monitor, a laptop, or a tablet.

4. The method according to claim 1, wherein the sensor includes a camera.

5. The method according to claim 4, wherein the camera includes an RGB-D camera.

6. The method according to claim 1, wherein the display is further configured to output a game that engages the user to interact with a virtual object in the virtual environment.

7. The method according to claim 6, wherein interacting with the virtual object in the virtual environment further includes causing the user to perform one or more movements.

8. Determining the force information based on the joint data includes: tracking, based on the joint data, an angle of the at least one joint while the user is interacting with the virtual object; determining the force information based on the tracked angle.

9. The method according to claim 6, wherein the force information includes an estimation of a force acting on the at least one joint while the user is interacting with the virtual object during the game.

10. receiving, from a second sensor, second motion data of a user's movement, the second motion data including second joint data associated with at least one joint of a second user. Determining second force information based on the second joint data, the second force information including an estimation of a force acting on at least one joint of the second user while the second user is interacting with a second virtual object in the virtual environment; The method according to claim 9, further comprising causing the second virtual object to overcome the virtual object based on the force information and the second force information.

11. Receiving, from a second sensor, second motion data of a user's movement, the second motion data including second joint data associated with at least one joint of a second user; Determining second force information based on the second joint data, the second force information including an estimation of a force acting on at least one joint of the second user while the second user is interacting with a second virtual object in the virtual environment; The method according to claim 9, further comprising causing the virtual object to overcome the second virtual object based on the force information and the second force information.

12. The method according to claim 6, further comprising outputting the motion data and data associated with the user interacting with the virtual object in the virtual environment to a communication network for remote access.

13. The method according to claim 12, wherein the force information includes a force perception association with the output of the motion data and the data associated with the user interacting with the virtual object in the virtual environment.

14. The method according to claim 1, further comprising transmitting data including the force information to a server, the server storing the data in a database associated with the user.

15. The method according to claim 1, wherein the calibration of the sensor includes performing real-time camera-skeleton pose calibration.

16. An apparatus comprising: One or more processors; A memory storing processor-executable instructions that, when executed by the one or more processors, cause the apparatus to: Output, to a display, an avatar of a user in a virtual environment based on calibration of a sensor; Receiving motion data of the user's movement from the sensor, wherein the motion data includes joint data associated with at least one joint of the user; Determining force information based on the joint data; Determining user strength associated with the at least one joint based on the force information, an apparatus for causing the above to be performed.

17. The apparatus according to claim 16, wherein the display is a head-mounted display (HMD).

18. The apparatus according to claim 16, wherein the display is a display device including at least one of a television, a monitor, a laptop, or a tablet computer.

19. The apparatus according to claim 16, wherein the sensor includes a camera.

20. The apparatus according to claim 19, wherein the camera includes an RGB-D camera.

21. The apparatus according to claim 16, wherein the display is further configured to output a game that engages the user to interact with a virtual object within the virtual environment.

22. The apparatus according to claim 21, wherein interacting with the virtual object within the virtual environment further includes causing the user to perform one or more movements.

23. Determining the force information based on the joint data includes: Tracking an angle of the at least one joint while the user is interacting with the virtual object based on the joint data; Determining the force information based on the tracked angle, the apparatus according to claim 21.

24. The apparatus according to claim 21, wherein the force information includes an estimation of a force acting on the at least one joint while the user is engaged with the virtual object during the game.

25. The memory stores processor-executable instructions that, when executed by the one or more processors, further cause the apparatus to: Receive second motion data of the user's movement from a second sensor, wherein the second motion data includes second joint data associated with at least one joint of a second user; Determining second force information based on the second joint data, the second force information including an estimation of a force acting on at least one joint of the second user while the second user is interacting with a second virtual object in the virtual environment; Causing the virtual object to overcome the second virtual object based on the force information and the second force information, the apparatus according to claim 24. **Claim 26** The memory stores processor-executable instructions that, when executed by the one or more processors, further cause the apparatus to Receive second motion data of a user's movement from a second sensor, the second motion data including second joint data associated with at least one joint of a second user; Determine second force information based on the second joint data, the second force information including an estimation of a force acting on at least one joint of the second user while the second user is interacting with a second virtual object in the virtual environment; Causing the virtual object to overcome the virtual object based on the force information and the second force information, the apparatus according to claim 24. **Claim 27** The memory stores processor-executable instructions that, when executed by the one or more processors, further cause the apparatus to output the motion data and data related to the user interacting with the virtual object in the virtual environment to a communication network for remote access, the apparatus according to claim 21. **Claim 28** The force information includes a force perception association with the output of the motion data and the data associated with the user interacting with the virtual object in the virtual environment, the apparatus according to claim 27. **Claim 29** The memory stores processor-executable instructions that, when executed by the one or more processors, further cause the apparatus to transmit data including the force information to a server, the server storing the data in a database associated with the user, the apparatus according to claim 16. **Claim 30** The apparatus according to claim 16, wherein the calibration of the sensor includes performing real-time camera-skeleton pose calibration.

Images