Tactile human medical examination

Abstract

An electronic device and method for medical examination of a human body using haptics is provided. The electronic device controls a first head-mounted display to present a 3D model of an anatomical portion of a body of a human subject. The presented 3D model includes a region corresponding to a defective portion in the anatomical portion. The electronic device transmits a touch input to a wearable sensor in contact with the anatomical portion. Such input corresponds to a human touch on the region of the presented 3D model. The electronic device receives, via the wearable sensor, a biosignal associated with the defective portion based on the touch input. The biosignal includes a physiological signal and a somatosensory information associated with the defective portion. In response to the human touch, the electronic device controls a wearable haptic device to generate haptic feedback based on the received set of biosignals.

Description

[0001] Cross-references / citations of related applications

[0002] This application is an international patent application claiming priority to U.S. Patent Application No. 17 / 094,564, filed November 10, 2020, with the United States Patent and Trademark Office. Each of the above-mentioned referenced applications is incorporated herein by reference in its entirety. Technical Field

[0003] Various embodiments of this disclosure relate to telemedicine and virtual reality technologies. More specifically, various embodiments of this disclosure relate to electronic devices and methods for human medical examinations using tactile sensation. Background Technology

[0004] Advances in medical technology have paved the way for a wide range of health and medical services, such as telemedicine and remote surgery. Consequently, some traditional medical practices, such as physical examinations of patients, appear to be moving away from the path of modern medicine. Less experienced practitioners may have limited access to examine or learn about symptoms and use touch to make diagnoses for different types of patients.

[0005] By comparing the described system with some aspects of this disclosure, as illustrated in the remainder of this application and with reference to the accompanying drawings, the limitations and disadvantages of conventional and traditional methods will become apparent to those skilled in the art. Summary of the Invention

[0006] An electronic device and method for human medical examination using tactile sensation is provided, which is substantially as shown in at least one figure and / or described in conjunction with at least one figure, and is set forth more fully in the claims.

[0007] These and other features and advantages of this disclosure can be understood by reading the following detailed description and the accompanying drawings, in which similar reference numerals always denote similar parts. Attached Figure Description

[0008] Figure 1 This is a diagram of an exemplary network environment for human medical examination using touch, according to embodiments of the present disclosure.

[0009] Figure 2 This is a block diagram of an electronic device for human medical examination using tactile sensation, according to embodiments of the present disclosure.

[0010] Figure 3 This is a diagram illustrating an exemplary scene of a 3D scan of a human object using a camera apparatus (rig) according to an embodiment of the present disclosure.

[0011] Figure 4This is a diagram illustrating an exemplary operation for a human medical examination using touch, according to an embodiment of the present disclosure.

[0012] Figure 5 This is an illustration of an exemplary scenario in which a human subject uses a second head-mounted display according to an embodiment of the present disclosure.

[0013] Figure 6 This is a diagram illustrating exemplary operations for training a machine learning model according to embodiments of the present disclosure.

[0014] Figure 7 This is a diagram illustrating an exemplary scenario for training a user using a 3D model from a 3D model database, according to an embodiment of the present disclosure.

[0015] Figure 8 This is a flowchart illustrating an exemplary method for human medical examination using touch according to embodiments of the present disclosure. Detailed Implementation

[0016] The embodiments described below can be found in the disclosed electronic devices and methods for human medical examination using tactile sensation. An exemplary aspect of this disclosure provides an electronic device that can be configured to control a first head-mounted display (such as a virtual reality headset) to present a first three-dimensional (3D) model of at least a first anatomical portion (such as an arm) of the body of a human object (which may be a patient). The presented first 3D model may include a first region corresponding to a defective portion in the first anatomical portion of the body. For example, the defective portion may correspond to a fracture in the arm of the human object's body.

[0017] At any time, a practicing physician, while wearing a head-mounted display, can touch a first area of ​​a first 3D model presented to examine a defective portion in a first anatomical part of a human subject's body. In response to this touch, electronic device 102 can transmit the touch input to a wearable sensor in contact with at least the first anatomical part of the body. The wearable sensor can receive the touch input and can send a set of biosignals associated with the defective portion to the electronic device. The wearable sensor can extract the set of biosignals based on the acquisition of physiological data points (such as heart rate) and somatic sensory information from the defective portion of the body.

[0018] The electronic device can receive a set of transmitted biosignals via wearable sensors. This set of biosignals may include physiological signals associated with the defect (such as heart rate) and somatic sensory information associated with the defect (such as sensations associated with touch, pressure, cold, or heat). The electronic device can control wearable haptic devices (such as wearable gloves worn by a practicing physician) to generate haptic feedback based on the received set of biosignals. The haptic feedback can be generated as a response to human touch on a first area of ​​a first 3D model.

[0019] This electronic device enables practicing physicians to perform physical examinations on any defective parts of a human subject in a remote consultation setting. In some cases, the device can also enable practicing physicians or interns to learn aspects of physical examination and medical consultation by interacting with 3D models of several test subjects or patients, whose medical records, biosignals from different parts of the body, and 3D scan data can be stored on a server.

[0020] While head-mounted displays allow practitioners to view and interact with 3D models of anatomical parts of human subjects, wearable haptic devices can enable practitioners to feel the touch and other somatic sensations associated with defective parts. For example, such sensations can simulate the cold or warm feeling, touch, or pressure that a practitioner typically experiences when touching a part of the body with his / her hand and applying pressure to that part. Compared to traditional video-based consultations, the electronic devices disclosed herein can enhance the remote consultation experience for both the practitioner and the patient and can help the practitioner make a better diagnosis and / or prognosis for any disease or medical condition of the patient.

[0021] To further enhance the telemedicine experience, a haptic device that contacts the first anatomical part of the patient's body can generate haptic feedback at the defective area. This haptic feedback helps the patient feel touch or sensation in response to a touch applied by a physician to a first area of ​​a presented 3D model. The electronic device can implement machine learning models to generate recommendations for the physician. Such recommendations can be based on a set of received biosignals, medical condition information associated with the human subject, and anthropometry characteristics related to the human subject's body. For example, such recommendations may include, but are not limited to, treatment procedures for the defective area, the current condition of the defective area, the prognosis for the defective area, and diagnoses related to the defective area of ​​the human subject.

[0022] Figure 1 This is a diagram of an exemplary network environment for human medical examination using tactile sensing, according to embodiments of the present disclosure. Reference Figure 1A block diagram of a network environment 100 is shown. The network environment 100 may include an electronic device 102, a first head-mounted display 104, a wearable haptic device 106, a wearable sensor 108, and a haptic device 110. The electronic device 102, the first head-mounted display 104, the wearable haptic device 106, the wearable sensor 108, and the haptic device 110 can communicate with each other via a communication network 112. The network environment 100 is shown only as an example environment for remote medical examinations (or consultations) and should not be construed as a limitation of this disclosure.

[0023] In the network environment 100, a user 114, such as a practicing physician or intern, is shown wearing a first head-mounted display 104 and a wearable haptic device 106. Furthermore, a human object 116 (which may be a patient) is shown having a wearable sensor 108 in contact with at least a first anatomical portion 118 of the human object 116's body. In one embodiment, at least the first anatomical portion 118 of the object may be in contact with the haptic device 110.

[0024] Electronic device 102 may include suitable logic, circuitry, and interfaces configured to control a first head-mounted display 104 to present a first three-dimensional (3D) model 120 of at least a first anatomical portion 118 of the body of a human object 116. Electronic device 102 may also be configured to receive a set of biosignals associated with defective portions of the first anatomical portion 118 of the body based on touch input. Examples of electronic device 102 may include, but are not limited to, computing devices, smartphones, cellular phones, mobile phones, gaming devices, mainframes, servers, computer workstations, and / or consumer electronics (CE) devices.

[0025] In one embodiment, the electronic device 102 may be implemented as a head-mounted device that can be worn by a user 114. In such an embodiment, the first head-mounted display 104 may be integrated with the head-mounted device. Example embodiments of the head-mounted device may include, but are not limited to, smart glasses, virtual reality (VR) head-mounted devices, augmented reality (AR) head-mounted devices, or mixed reality (MR) head-mounted devices.

[0026] The first head-mounted display 104 may include suitable logic, circuitry, and interfaces for a first 3D model 120 that can be configured to present at least a first anatomical portion 118 of the body of a human object 116. The first head-mounted display 104 may be worn on the head by a user 114 (such as a practicing physician) or as part of a helmet. The first head-mounted display 104 may include display optics such that when the user 114 wears the first head-mounted display 104, the display optics are positioned in front of one or both eyes of the user 114. In embodiments, the first head-mounted display 104 may include an inertial measurement unit for the user 114's VR experience. Examples of the first head-mounted display 104 may include, but are not limited to, virtual reality head-mounted devices, optical head-mounted displays, augmented reality head-mounted devices, mixed reality head-mounted devices, virtual reality glasses, and virtual reality eyepieces.

[0027] The wearable haptic device 106 may include suitable logic, circuitry, and interfaces configured to generate haptic feedback based on a set of biosignals associated with a defective portion in a first anatomical region 118 of the body of the human object 116. As an example, the wearable haptic device 106 may be worn by a user 114 (such as a practicing physician) on one or both hands. Haptic feedback may be generated in response to a human touch by the user 114 on a first region 122 of the first 3D model 120. In embodiments, the wearable haptic device 106 may include sensors, such as haptic sensors, that allow measurement of the force of the human touch by the user 114 on the first region 122 of the first 3D model 120. Examples of wearable haptic devices 106 may include, but are not limited to, haptic gloves, wired gloves with haptic actuators, gaming gloves with haptic actuators, wearable fingertip haptic devices (such as haptic sleeves or touch sleeves), graspable haptic devices (which can generate kinesthetic sensations such as the sensation of movement, position, and force in the wearer's skin, muscles, tendons, and joints), or wearable devices (which generate haptic sensations such as pressure, friction, or temperature in the wearer's skin), joysticks, mice, finger pads, robotic handles, grippers, and humanoid robotic hands with haptic actuators.

[0028] Wearable sensor 108 may include appropriate logic, circuitry, and interfaces that can be configured to receive touch input from electronic device 102. The received touch input may correspond to a human touch by user 114 (such as a practicing physician) on a first region 122 of the first 3D model 120. Wearable sensor 108 may be configured to measure one or more parameters associated with human subject 116 to generate a set of biosignals. Based on the touch input, wearable sensor 108 may transmit this set of biosignals to electronic device 102. This set of biosignals may include physiological signals and somatosensory information associated with defective portions (e.g., fractured bone) in a first anatomical portion 118 of the human subject 116's body. This set of biosignals may include, for example, an electroencephalogram (EEG) of human subject 116, an electrocardiogram (ECG) of human subject 116, an electromyogram (EMG) of human subject 116, and a skin conductance response (GSR) of human subject 116.

[0029] In one embodiment, the wearable sensor 108 may contact at least a first anatomical portion 118 of the body of a human object 116. In another embodiment, the wearable sensor 108 may be wrapped, wrapped, or strapped around the first anatomical portion 118 of the body. The wearable sensor 108 may acquire multimodal data through sensors such as, but not limited to, photoplethysmography (PPG) sensors, temperature sensors, blood pressure sensors, ambient oxygen partial pressure (ppO2) sensors, or sensors that collect somatic sensory information associated with the first anatomical portion 118 of the body. Example embodiments of the wearable sensor 108 may include, but are not limited to, a belt-type wearable sensor, a vest with embedded biosensors, a belt with embedded biosensors, a wristband with embedded biosensors, an instrumented wearable belt, wearable clothing with embedded biosensors, or a modified wearable manufactured article with biosensors.

[0030] The haptic device 110 may include suitable logic, circuitry, and interfaces configured to reproduce the sensation of human touch at any specific location on the first anatomical portion 118. This sensation can be reproduced as tactile by the haptic device 110 and can be based on tactile input (i.e., human touch from user 114) on a first region 122 of the first 3D model 120. The haptic device 110 may be configured to be worn by a human object 116. For example, the haptic device 110 may contact at least the first anatomical portion 118 of the body of the human object 116. Examples of the haptic device 110 may include, but are not limited to, wearable clothing with haptic actuators, wearable devices with haptic actuators, or any device in the form of wearable straps or medical tapes / cloths with haptic actuators. In embodiments, the haptic device 110 may be integrated or embedded in a wearable sensor 108.

[0031] Communication network 112 may include a communication medium through which electronic device 102, first head-mounted display 104, wearable haptic device 106, wearable sensor 108, and haptic device 110 can communicate with each other. Examples of communication network 112 may include, but are not limited to, the Internet, cloud networks, Wi-Fi networks, personal area networks (PANs), local area networks (LANs), or metropolitan area networks (MANs). Various devices in network environment 100 may be configured to connect to communication network 112 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Zigbee, EDGE, IEEE 802.11, Li-Fi, 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0032] In operation, electronic device 102 can initiate a session on first head-mounted display 104. The session can be a remote consultation session between user 114, such as a practicing physician, and human subject 116. The session may include a medical examination of at least a first anatomical portion 118 of the human subject 116's body (e.g., a physical examination in a computer simulation environment).

[0033] Electronic device 102 can generate a first 3D model 120 of at least the first anatomical portion 118 of the body of a human object 116 based on 3D scan data of at least the first anatomical portion 118 of the human body. The 3D scan data can be acquired in near real-time from a camera setup or stored in a medical database. In some embodiments, the first 3D model 120 can be further generated based on anthropometric features (such as skin color, skin texture, weight, height, or age) associated with the body of the human object 116. In these or other embodiments, the first 3D model 120 can be generated based on appearance attributes associated with defective portions in the first anatomical portion 118 of the body of the human object 116. Appearance attributes can be used to determine modifications to the first 3D model 120 to indicate defective portions.

[0034] After the session is initialized, the electronic device 102 can control the first head-mounted display 104 to present a first 3D model 120 of at least a first anatomical portion 118 of the body of the human object 116. As an example, the first 3D model 120 may be a computer-generated model and may be presented in a computer simulation environment such as a VR or AR environment. The first 3D model 120 may include a first region 122 that may correspond to a defective portion in the first anatomical portion 118 of the body. Examples of defective portions may include, but are not limited to, tissue deformation or defects, abraded tissue or skin, fractured bone, tumor tissue or organ, deformed bone, or swelling or mass.

[0035] The first head-mounted display 104 may be associated with a user 114, such as a practicing physician who can use the first head-mounted display 104 to view and examine at least a first anatomical portion 118 of the body of a human object 116. The first anatomical portion 118 of the body may include, for example, a hand, arm, leg, abdomen, torso, head, bones, or internal organs, skin, muscles, tendons, or combinations thereof. For example, in Figure 4 The system provides control over the first head-mounted display 104 to render the details of the first 3D model 120.

[0036] At any time, a user 114, such as a practicing physician, can touch a first region 122 of the presented first 3D model 120 while wearing the first head-mounted display 104 to examine defective portions in a first anatomical portion 118 of the human object 116's body. In response to this touch, electronics 102 can transmit touch input to a wearable sensor 108, which can contact at least the first anatomical portion 118 of the human object 116's body. The transmitted touch input may correspond to a human touch (e.g., performed by user 114) on the first region 122 of the presented first 3D model 120. For example, the touch input can activate the wearable sensor 108 to measure one or more parameters associated with the human object 116, such as, but not limited to, body temperature associated with defective portions, somatic sensations (including tactile sensations such as touch or kinesthesia), and blood pressure of the human object 116. For example, in Figure 4 The details of transmitting touch input to wearable sensor 108 are provided.

[0037] Electronic device 102 can receive a set of biosignals from wearable sensor 108 via communication network 112. This set of biosignals can be transmitted based on touch input and can be associated with defective portions in a first anatomical portion 118 of the human object 116's body. This set of biosignals may include physiological signals and somatosensory information associated with defective portions in the first anatomical portion 118 of the body. For example, in Figure 4Details related to receiving touch input from wearable sensor 108 are provided in the document.

[0038] Electronic device 102 can control wearable haptic device 106 to generate haptic feedback based on a received set of biosignals. The haptic feedback can be generated as a response to a human touch (such as that of user 114) on a first region 122 of the first 3D model 120. For example, in Figure 4 The document further provides details on the generation of haptic feedback.

[0039] While the first head-mounted display 104 enables a user 114, who may be a practicing physician, to view and interact with a first 3D model 120 of at least a first anatomical portion 118 of a human object 116, the wearable haptic device 106 enables the user 114 to feel touch and other somatic sensations associated with the defective portion. For example, such sensations can simulate the cold or warm sensations, touch, or pressure typically experienced when a practicing physician uses his / her hand to touch a part of the body and applies pressure to that part. Compared to traditional video-based consultations, the electronic device 102 of this disclosure can enhance the remote consultation experience for both the practicing physician and the patient, and can help the practicing physician perform better diagnosis and / or prognosis for any disease or medical condition of the patient.

[0040] Figure 2 This is a block diagram of an electronic device for human medical examination using tactile sensation, according to embodiments of the present disclosure. (In conjunction with...) Figure 1 Component explanation Figure 2 . refer to Figure 2 , showed Figure 1 The electronic device 102 is framed at 200. The electronic device 102 may include circuitry 202 and memory 204. Memory 204 may include a machine learning model 204A. The electronic device 102 may also include an input / output (I / O) device 206 and a network interface 208. In one embodiment, the electronic device 102 may include a first head-mounted display 104 as a display unit of the electronic device 102.

[0041] Circuit 202 may include appropriate logic, circuitry, and interfaces that can be configured to execute program instructions associated with different operations to be performed by electronic device 102. Circuit 202 may include one or more dedicated processing units that can be implemented as an integrated processor or processor cluster that collectively performs the functions of one or more dedicated processing units. Circuit 202 may be implemented based on a variety of processor technologies known in the art. Examples of implementations of circuit 202 may be x86-based processors, graphics processing units (GPUs), reduced instruction set computing (RISC) processors, application-specific integrated circuit (ASIC) processors, complex instruction set computing (CISC) processors, microcontrollers, central processing units (CPUs), coprocessors, or combinations thereof.

[0042] Memory 204 may include appropriate logic, circuitry, and interfaces, and may be configured to store program instructions to be executed by circuitry 202. Memory 204 may be configured to store a first 3D model 120 and a machine learning model 204A of human object 116. Memory 204 may also be configured to store a first set of features, including anthropometric features related to the body, medical information associated with human object 116, and a second set of features related to defects in the body of human object 116. Examples of implementations of memory 204 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drive (HDD), solid-state drive (SSD), CPU cache, and / or secure digital card (SD card).

[0043] Machine learning model 204A can be a recommendation model, which can be trained to recognize the relationship between input features and medical recommendations. Input features may include, for example, features obtained from a set of biosignals, medical condition information, and / or medical records of human subject 116. Machine learning model 204A can be defined by its hyperparameters, such as the number of weights, cost function, input size, number of layers defined by the topology, etc. The hyperparameters of machine learning model 204A can be tuned, and the weights can be updated to move towards the global minimum of the cost function of machine learning model 204A. After several epochs of training on features in the training dataset, machine learning model 204A can be trained to output a result given input features. For example, this result can be used to select a recommendation from a database that can provide recommendations for different medical conditions or symptoms. After the wearable haptic device 106 generates haptic feedback, the recommendation can be provided to human subject 116 as medical advice or consultation.

[0044] Machine learning model 204A may include electronic data, which may be implemented as a software component of an application executable, for example, on electronic device 102 (or the first head-mounted display 104). Machine learning model 204A may rely on libraries, external scripts, or other logic / instructions for execution by a processing device such as electronic device 102. Machine learning model 204A may include code and routines configured to enable a computing device such as electronic device 102 to perform one or more operations to recommend based on one or more inputs and outputs. Alternatively or additionally, machine learning model 204A may be implemented using hardware including a processor, a microprocessor (e.g., to perform or control the execution of one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, machine learning model 204A may be implemented using a combination of hardware and software.

[0045] Examples of machine learning model 204A may include, but are not limited to, linear regression, logistic regression, decision tree, support vector machine (SVM), Naive Bayes, random forest, and K-nearest neighbor (KNN) algorithms. In one embodiment, machine learning model 204A may be implemented as a neural network. Examples of neural networks may include, but are not limited to, deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), CNN-RNN, region-based CNN (R-CNN), fast R-CNN, faster R-CNN, artificial neural networks (ANN), YOLO (You Only Look Once) network, RNN based on long short-term memory (LSTM), CNN+ANN, LSTM+ANN, RNN based on gated recurrent units (GRU), fully connected neural networks, RNN based on connectionality time classification (CTC), deep Bayesian neural networks, regenerative adversarial networks (GAN), and / or combinations of such networks. In some embodiments, machine learning model 204A may use data flow graphs to implement numerical computation techniques. In some embodiments, the machine learning model 204A may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0046] I / O device 206 may include appropriate logic, circuitry, and interfaces, and may be configured to receive input from user 114, who may be a practicing physician, and to provide output based on the received input. I / O device 206, which may include various input and output devices, may be configured to communicate with circuitry 202. Examples of I / O device 206 may include, but are not limited to, touchscreens, keyboards, mice, joysticks, microphones, display devices, and speakers.

[0047] Network interface 208 may include appropriate logic, circuitry, and interfaces that can be configured to facilitate communication between electronic device 102 and first head-mounted display 104, wearable haptic device 106, wearable sensor 108, and haptic device 110 via communication network 112. Network interface 208 can be implemented using various known technologies to support wired or wireless communication between electronic device 102 and communication network 112. Network interface 208 may include, but is not limited to, antennas, radio frequency (RF) transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, encoder-decoder (CODEC) chipsets, subscriber identity module (SIM) cards, or local buffer circuitry. Network interface 208 can be configured to communicate wirelessly with networks such as the Internet, intranets, or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and metropolitan area networks (MANs). Wireless communication can be configured to use one or more of a variety of communication standards, protocols, and technologies, such as 5G NR, GSM, EDGE, W-CDMA, LTE, CDMA, TDMA, Bluetooth, Wi-Fi (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Li-Fi, Wi-MAX, protocols for email and instant messaging, and Short Message Service (SMS).

[0048] like Figure 1 The functions or operations performed by electronic device 102 as described herein can be performed by circuit 202. For example, in Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 The operation performed by circuit 202 is described in detail.

[0049] Figure 3 This is an illustration of an exemplary scene showing a 3D scan of a human object using a camera apparatus according to an embodiment of this disclosure. Combined with... Figure 1 and 2 To describe the elements Figure 3 . refer to Figure 3Scene 300 is illustrated. In scene 300, a camera apparatus that may include one or more imaging sensors, such as a first imaging sensor 302A, a second imaging sensor 302B, and a third imaging sensor 302C, is shown. In scene 300, a human object 116 is shown having a T-pose at a scan point associated with the camera apparatus. This disclosure is applicable to other poses of the human object 116 for 3D scanning.

[0050] In one embodiment, one or more imaging sensors may be depth sensors (such as time-of-flight sensors) configured to scan the human object 116 to capture detailed physical features of the human object 116, such as skin color, skin texture, and hair on the body of the human object 116. According to an embodiment, circuitry 202 may be configured to control a camera apparatus (including one or more imaging sensors) to scan at least a first anatomical portion 118 of the body of the human object 116. The camera apparatus may perform a 3D scan of at least the first anatomical portion of the human object 116 to capture detailed physical features. One or more imaging sensors may be connected to a single target machine to perform the scanning of the human object 116. In some embodiments, one or more imaging sensors may be connected to multiple target machines to perform the scanning of the human object 116.

[0051] like Figure 3 As shown, a first imaging sensor 302A may be positioned such that it scans at least the upper portion of the body of the human object 116. A second imaging sensor 302B may be positioned such that it scans at least the middle portion of the body of the human object 116. A third imaging sensor 302C may be positioned such that it scans at least the lower portion of the body of the human object 116. In scenario 300, a platform on which the human object 116 can stand for 3D scanning is shown. The platform may be rotatable, allowing the camera apparatus to perform 3D scanning of the human object 116 from multiple viewpoints. In an embodiment, circuitry 202 (or the camera apparatus) may register viewpoint-specific scan data from all such viewpoints into 3D scan data.

[0052] One or more imaging sensors may be configured to scan the body of human object 116, such as in a hospital, in the home of human object 116, or in a specific location (e.g., a scanning studio) for scanning the body of human object 116.

[0053] In some embodiments, imaging data may be acquired from at least one surgical aid, such as a laparoscope, X-ray, or ultrasound. Such imaging data may include, for example, images and depth information associated with one or more internal organs of the human object 116. Circuit 202 may generate a 3D representation of one or more internal organs of the human object 116 (with or without defective portions) based on such imaging data. In these or other embodiments, circuit 202 may fuse 3D scan data from a camera device with the generated 3D representation.

[0054] In an embodiment, circuitry 202 may be configured to receive depth information associated with a defective portion from one or more imaging sensors. For example, circuitry 202 may receive depth information (which may also include texture, reflectivity, or albedo information) relating to the amount of swelling in the defective portion due to a fracture of bone in the human object 116. Based on the received depth information, circuitry 202 may generate a first region 122 of a first 3D model 120 corresponding to the defective portion in a first anatomical part 118 of the body. The first region 122 may be rendered as part of the first 3D model 120. For example, the first region 122 may be rendered to model a defective portion, such as a fractured bone in a first anatomical part 118 of the body of the human object 116 (such as an arm).

[0055] Circuit 202 can be configured to receive 3D scan data of at least a first anatomical portion 118 of the body based on a 3D scan of one or more imaging sensors (equipped with a camera). The 3D scan data may include depth information associated with a defective portion in the first anatomical portion 118 of the body. According to an embodiment, the 3D scan data may include a 3D representation of at least one internal organ of the human object 116. The 3D representation of at least one internal organ may be generated based on a scan performed, for example, by a laparoscopic camera. The defective portion may be described in the 3D representation as a swelling or mass in internal tissue or at least one internal organ (e.g., lung, kidney, or stomach). Circuit 202 can be configured to control a first head-mounted display 104 to present a first 3D model 120 of at least the first anatomical portion 118 of the body based on the received 3D scan data. For example, in Figure 4 Further details related to the rendering operation of the first 3D model 120 are provided.

[0056] Figure 4 This is a diagram illustrating exemplary operation of a human medical examination using tactile sensation according to an embodiment of the present disclosure. (In conjunction with...) Figure 1 , Figure 2 and Figure 3 Component explanation Figure 4 . refer to Figure 4Figure 400 is shown to depict exemplary operations from 402 to 422. The exemplary operations shown in 400 may begin at 402 and may be performed by any computing system, device, or apparatus, such as by... Figure 2 The electronic device 102 performs the operation.

[0057] At 402, a scan of human object 116 can be performed. According to an embodiment, circuitry 202 can be configured to control camera equipment (e.g., such as...). Figure 3 (As shown) to scan at least a first anatomical portion 118 of the body of a human subject 116, and based on this scan, receive 3D scan data 402A of at least the first anatomical portion of the human subject's body from a camera device. A scan of the human subject 116 may have to be performed before a telemedicine session or telemedicine examination can be initiated. In some embodiments, the scan may be performed more than once before one or more telemedicine sessions and / or during the duration of one or more telemedicine sessions. For example, if a person has a fractured kneecap, then a scan is performed before the start of the first telemedicine session, and another 3D scan is performed before the start of a subsequent consultation session to examine the fracture healing and determine future treatment procedures.

[0058] At 404, the first 3D model 120 can be presented on the first head-mounted display 104. According to an embodiment, circuitry 202 can be configured to control the first head-mounted display 104 to present the first 3D model 120 of at least a first anatomical portion 118 of the body of a human object 116. Prior to presentation, the first 3D model 120 can be generated based on 3D scan data of the first anatomical portion 118 of the body (such as 3D scan data 402A), depth information (such as depth information 404A), a first set of features 404B, and a second set of features 404C.

[0059] In one or more embodiments, depth information 404A may be associated with the defect portion and may be received from one or more imaging sensors (e.g., Figure 3 (As described in the text). For example, depth information 404A may be associated with the amount of swelling in the arm of human object 116 caused by a fracture of a bone in the first anatomical portion 118 of the body. 3D scan data 402A may be received based on a scan performed by a camera apparatus (which includes one or more imaging sensors). 3D scan data 402A may include details such as body shape, skin texture, body reflectivity information, skin shading, etc., associated with at least the first anatomical portion 118 of the body of human object 116.

[0060] According to an embodiment, circuit 202 may be configured to receive a first set of features 404B. The first set of features 404B may include anthropometric features related to the body of human subject 116 and medical condition information associated with human subject 116. Anthropometric features may involve quantitative measurements associated with the body of human subject 116. Examples of anthropometric features associated with human subject 116 may include, but are not limited to, human subject 116's height, human subject 116's weight, human subject 116's body mass index (BMI), dimensional measurements of different body parts (e.g., chest measurements, waist measurements, hip measurements, or limb measurements), and skinfold thickness. Similarly, medical condition information associated with human subject 116 may include, for example, a medical history, surgical, traumatic, or injury history, previous health conditions (e.g., varicose veins), important health statistics (such as blood pressure (BP) or iron levels in the blood), test report information (e.g., blood test reports), and other test-related information (such as pathology reports). Test report information may include, for example, X-ray reports, magnetic resonance imaging (MRI) reports, etc., associated with at least a first anatomical portion 118 of the body. The first 3D model 120 can be generated based on a first set of features 404C to accurately depict the physical features of the human object 116 in the first 3D model 120. After the first 3D model 120 is generated, it can be displayed on the first head-mounted display 104.

[0061] According to an embodiment, circuit 202 may be configured to receive a second set of features 404C. Such features may be associated with a defective portion in a first anatomical part 118 of the body of the human object 116 and may include appearance attributes associated with the defective portion. In some embodiments, the second set of features 404C may include physical deformation associated with at least the first anatomical part 118 of the body or deformation associated with one or more internal organs of the body. As an example, the second set of features 404C may indicate swelling (i.e., deformation) in the defective portion due to a fractured bone and redness of the skin around the swelling. As another example, the second set of features 404C may indicate a lump or cyst in an internal organ of the human object 116, such as on the liver. The defective portion in the first 3D model 120 may be generated based on the second set of features 404C.

[0062] According to an embodiment, circuit 202 can be configured to update the generated first 3D model 120 by mesh deformation of at least a first region 122 of the generated first 3D model 120. Mesh deformation can be applied based on a received second set of features 404C to model the appearance of defective portions in the first region 122 of the first 3D model 120. First head-mounted display 104 can present the updated first 3D model 120 based on control signals from circuit 202.

[0063] At 406, a first region 122 can be selected. According to an embodiment, circuitry 202 can be configured to select a first region 122 in a first 3D model 120 representing the first anatomical portion 118, as shown in 406A. The first region 122 may correspond to a defective portion in the first anatomical portion 118 of the body. The first region 122 may be selected by a user 114 (such as a practicing physician) for medical examination of a human subject 116. By way of example and not limitation, a defective portion may be one or more of the following: tissue deformity or defect, abraded tissue or skin, fractured bone, tumor tissue or organ, deformed bone, or swelling or mass.

[0064] At 408, touch input can be transmitted. Circuit 202 can be configured to transmit touch input to a wearable sensor 108 that is in contact with a first anatomical portion 118 of the body. In some embodiments, the wearable sensor 108 may not be in contact with the first anatomical portion 118 of the body. As shown in 408A, touch input may include selection of a first region 122 of a first 3D model 120. Touch input can be transmitted to the wearable sensor 108 to initiate the acquisition of biosignals such as physiological signals (e.g., heart rate, blood pressure, sweat volume (galvanic skin response)) and somatic sensory information associated with the human object 116. In some instances, biosignals may also include body temperature information associated with the body of the human object 116. The wearable sensor 108 can acquire temperature sensing information with or without direct contact with the body of the human object 116.

[0065] According to an embodiment, circuitry 202 may be configured to transmit touch input to a tactile device 110 that is in contact with at least a first anatomical portion 118 of the body of a human object 116. The tactile device 110 may receive the transmitted touch input via a communication network 112. The tactile device 110 may generate tactile sensations on the first anatomical portion 118 of the body based on the received touch input. Tactile sensations may include one or more senses, such as touch / feeling on the skin, a feeling of warmth or cold in a touch, or pressure applied to the skin by touch. Through these tactile sensations, when user 114 touches a first area 122 of the first 3D model 120, the tactile device 110 may allow the human object 116 to perceive and feel the touch on a defective portion of the first anatomical portion 118. Therefore, the tactile device 110 may provide a tactile and sensory experience to the human object 116.

[0066] According to another embodiment, touch input can be transmitted to a robotic arm instead of the haptic device 110. For example, touch input can be transmitted in the form of a pressure signal via a wearable haptic device 106 (in response to a touch applied by user 114 on a first region 122). The robotic arm can be manipulated in space to mimic a touch applied by user 114 to a defective portion of a first anatomical part 118 of the body. The amount of pressure applied by user 114 can be experienced by a human object 116 as a tactile sensation on a defective portion of the first anatomical part 118 of the body.

[0067] According to another embodiment, instead of having a wearable sensor 108, one or more sensors may be present around the human subject and may not be in contact with the body of the human subject 116. In such an implementation, the acquisition of biosignals, such as physiological signals, body temperature information, and somatic sensory information associated with the human subject 116, can be accomplished by such sensors. For example, the body temperature of the human subject 116 can be recorded using a thermal scanner. Alternatively, electronic nose technology (such as an artificial nose) can be used to analyze the respiratory profile of the human subject 116 to detect volatile organic compounds that may contribute to the detection of certain diseases such as lung cancer and Alzheimer's disease. Alternatively, venous visualization technology (VVT) can be used to analyze the location of peripheral veins in the human subject 116.

[0068] At 410, a set of biosignals can be received. According to an embodiment, circuitry 202 can be configured to receive a set of biosignals from wearable sensor 108 based on touch input. This set of biosignals can be associated with a defective portion in a first anatomical part 118 of the body. For example, the set of biosignals can be associated with swelling on the arm of human subject 116. The set of biosignals can include physiological signals and somatosensory information associated with the defective portion. For example, physiological signals can include electrocardiogram (ECG) signals, respiratory rate, skin conductance, muscle currents, electromyography (EMG), or electroencephalography (EEG) of human subject 116. Physiological signals can be measured by wearable sensor 108, which can contact the first anatomical part 118 of the body of human subject 116. According to one embodiment, somatosensory information can include tactile information associated with the defective portion, kinesthetic information associated with the defective portion, and tactile perception information associated with the defective portion. For example, somatosensory information can include information such as touch on the skin, pressure applied to different locations on the skin due to touch, and skin temperature at the time of touch. Tactile information may include tactile input from the skin of human object 116. Kinesthetic information may include information related to pain in muscles, tendons, and joints. Sensations from internal organs of human object 116 may include, for example, a feeling of fullness in the stomach or bladder. Tactile perception information may involve somatic perception of patterns (e.g., edges, curvature, and texture) on the skin surface, as well as proprioception of hand position and shape.

[0069] At 412, haptic feedback can be generated. According to an embodiment, circuitry 202 can be configured to control wearable haptic device 106 to generate haptic feedback based on a received set of biosignals. Haptic feedback can be generated as a response to human touch, such as touch input from user 114. In one embodiment, haptic feedback can include kinematic and haptic feedback. Kinematic feedback can correspond to sensations in at least one of the joint angles of muscles, joints, tendons, arms, hands, wrists, fingers, etc. Haptic feedback can correspond to sensations in the fingers or on the skin surface, such as vibrations, touch, and texture of the skin. In another embodiment, wearable haptic device 106 can use haptic feedback to simulate warmth or coldness associated with a defective portion on a part of the user's skin in contact with wearable haptic device 106. For example, if user 114 is wearing a haptic glove on their right hand and touches a swollen area on a 3D model of a human arm, the haptic feedback can stimulate sensations typically experienced when physically touching a swelling during a physical examination. Such sensations can include tactile and / or kinematic sensations, as well as warmth associated with the swelling.

[0070] At 414, input features can be generated for machine learning model 204A. According to an embodiment, circuit 202 can be configured to generate input features for machine learning model 204A based on a received set of biosignals, medical condition information associated with human subject 116, and anthropometric features related to the body of human subject 116. Input features may include at least one of, for example, human subject 116's body temperature, blood pressure, medical conditions associated with human subject 116 such as obesity and thyroid disorders, human subject 116's BMI, skin texture around defective portions, X-rays of the first anatomical portion 118, etc.

[0071] Machine learning model 204A can be trained on a training dataset that may include biosignals, medical condition information, and anthropometric features of various test subjects. For example, in Figure 6 The document further provides details on training the machine learning model 204A.

[0072] At 416, machine learning model 204A can output a recommendation. According to an embodiment, circuitry 202 can input generated input features into machine learning model 204A, and machine learning model 204A can output a recommendation based on the input features. In one embodiment, the input to machine learning model 204A may include data points from a received set of biosignals, medical condition information, and cascaded anthropometric features (i.e., input features). In another embodiment, machine learning model 204A may be a hybrid model and may receive data points from the set of received biosignals, medical condition information, and anthropometric features at different stages of machine learning model 204A. Each of these stages can output an inference, which can later be used to output a recommendation. In an embodiment, a first head-mounted display 104 may present the output recommendation to assist user 114 (which may be a practicing physician) in providing a remote consultation to human subject 116. Examples of the output recommendation may include, but are not limited to, the treatment process for the defective part, the current status of the defective part, the prognosis of the defective part, a diagnosis associated with the defective part, or a combination thereof.

[0073] In an exemplary scenario, recommendations may include a set of tests that may be necessary to treat human subject 116, such as MRI, X-ray, or blood tests. Recommendations may also include classifying one or more parameters (in the form of biosignals) measured by wearable sensor 108 into normal, abnormal, or high-risk statistics. For example, if blood pressure measured by wearable sensor 108 is (140, 80) mmHg, the output recommendation could classify blood pressure as an abnormal statistic. This classification can help user 114 prescribe appropriate medications or examinations based on the recommendations. Furthermore, recommendations may include suggested treatment procedures for human subject 116 based on one or more measured parameters. For example, recommendations may include details of certain medications that are suitable prescriptions for human subject 116. Recommendations may indicate the duration of treatment for human subject 116, the current stage of the disease (e.g., stage II leukemia), or the incubation period in cases where the human subject is diagnosed with a viral or bacterial infection.

[0074] According to an embodiment, circuitry 202 can be configured to extract recommendations from machine learning model 204A and control the first head-mounted display 104 to present the recommendations. The recommendations can be presented by the first head-mounted display 104 in the form of a report to be provided to user 114. The report based on the recommendations can be analyzed by user 114 for use in the treatment of human subject 116.

[0075] At 418, it can be determined whether the recommendation was selected by user 114. In one embodiment, circuit 202 can be configured to determine whether to select the recommendation based on user input from user 114. For example, through user input, user 114 (who may be a practicing physician) can approve the recommendation as suitable for human subject 116. If the recommendation was selected by user 114, control can proceed to 420. Otherwise, control can proceed to 422.

[0076] At 420, the selected recommendations can be shared on a display device (such as an extended reality (XR) headset) associated with human object 116. In this embodiment, circuitry 202 can share the selected recommendations on the display device associated with human object 116.

[0077] At 422, a rescan can be performed. In one embodiment, circuitry 202 can skip the recommendation (output at 416) and control can be passed to 402. Alternatively, control can be passed to 404, and a 3D model of the anatomical portion of another human subject can be displayed on the first head-mounted display 104. When control is passed to 402 or 404, user 114 (who may be a practicing physician) can seamlessly switch between multiple remote consultation (and / or training) sessions with one or more human subjects.

[0078] Although 400 is shown as a discrete operation, such as 402, 404, 406, 408, 410, 412, 414, 416, 418, 420 and 422, in some embodiments, depending on the particular implementation, such discrete operations may be further divided into additional operations, combined into fewer operations or eliminated without departing from the essence of the disclosed embodiments.

[0079] Figure 5 This is an illustration showing an exemplary scenario of a human subject using a second head-mounted display according to embodiments of the present disclosure. Combined with... Figure 1 , Figure 2 , Figure 3 and Figure 4 Element description Figure 5 . refer to Figure 5 Scene 500 is shown. Scene 500 may include a human object 116 as a wearer of the second head-mounted display 502. Scene 500 may also include a first 3D model 120 and a second 3D model 504 of at least the hand of a user 114 when wearing the wearable haptic device 106.

[0080] According to an embodiment, circuitry 202 can be configured to control a second head-mounted display 502 to present a first 3D model 120 and a second 3D model 504. The second head-mounted display 502 can be controlled based on touch input on a first region 122 of the first 3D model 120. The second 3D model 504 may include at least the hand of a user 114 and can be presented to simulate the movement of that hand when a human touch is applied to the first region 122 of the first 3D model 120.

[0081] In one exemplary scenario, the human object 116 may be located at a location different from that of the user 114 who is examining the first 3D model 120 associated with the human object 116. While the user 114 may be undergoing a medical examination, the second head-mounted display 502 can enable the human object 116 to view, in real-time or near real-time, what the user 114 can see through the first head-mounted display 104, as well as the user 114's hand movements (i.e., through the second 3D model 504). Additionally, wearable sensors 108, wearable haptic devices 106, and haptic devices 110 can be used to simulate the experience of a touch- and feel-based physical medical examination in a virtual telemedicine setup for both the user 114 and the human object 116.

[0082] Figure 6 This is a diagram illustrating exemplary operations for training a machine learning model according to embodiments of the present disclosure. (In conjunction with...) Figure 1 , Figure 2 , Figure 3 , Figure 4and Figure 5 Component explanation Figure 6 . refer to Figure 6 The diagram illustrates 600, which describes exemplary operations from 602 to 604. The exemplary operations shown in 600 may begin at 602 and can be performed by any computing system, apparatus, or device, such as those provided by [unclear text - likely a computer or device]. Figure 2 The electronic device 102 performs the operation.

[0083] At 602, training data can be input into machine learning model 204A. The training data can be multimodal data and can be used to train machine learning model 204A. The training data may include, for example, a set of biosignals 602A, medical condition information 602B, and anthropometric features 602C from various test subjects. For example, the training data can be associated with multiple human test subjects who have undergone previous medical examinations.

[0084] Several input features can be generated for the machine learning model 204A based on training data (which can be obtained from a medical record database). The training data may include various data points associated with health disorders, diseases, injuries, laboratory tests, medical images, and other relevant information. For example, the training data may include data points associated with human subjects having more than one disease or medical condition, such as diabetes and hypertension. Alternatively or additionally, the training data may include data points associated with human subjects suffering from different types of health conditions and disorders, such as lifestyle-based diseases (e.g., obesity) and genetic disorders (e.g., Down syndrome). The training data may also include data points associated with human subjects having different ages, sexes, and anthropometric characteristics, such as each subject's BMI, height, or weight.

[0085] At position 604, a machine learning model 204A can be trained on the training data. Before training, a set of hyperparameters can be selected based on, for example, user input from a software developer. For instance, specific weights can be selected for each data point in the input features generated from the training data.

[0086] During training, several input features can be sequentially passed as input to a machine learning model 204A. The machine learning model 204A can output several recommendations (such as recommendation 606A) based on such inputs. Each such recommendation can be compared to a ground truth recommendation (such as ground truth 606B). Parameters of the weights of the machine learning model 204A can be updated based on an optimization algorithm (e.g., stochastic gradient descent) to minimize a loss function 606C of the machine learning model 204A. The value of the loss function 606C for a given pair of input features and output recommendations can determine how much the output recommendation deviates from the ground truth recommendation. In an embodiment, training error can be calculated based on the loss function 606C. The loss function 606C can be used to determine the error in the predictions of the machine learning model 204A. For example, if recommendation 606A is incorrect (when compared to ground truth 606A), the loss function 606C can indicate a loss value (i.e., above a threshold, such as 50%).

[0087] Once trained, the machine learning model 204A can select higher weights for data points in the input features, which can contribute more to the output recommendation than other data points in the input features. As an example, if the input features are generated to identify a fracture in a test subject's left knee, the skin color and swelling around the left knee can be given higher weights than the test subject's body temperature.

[0088] In an exemplary scenario, input features may include symptoms associated with a human subject, such as “high fever, cough, and cold.” Machine learning model 204A may output a recommendation such as “develop fever medication and antibiotics.” If the underlying fact 606A also includes recommendations for treatments such as “develop fever medication and antibiotic prescriptions,” then the output of machine learning model 204A can be considered accurate and ready for deployment. In some cases, the recommendations output by machine learning model 204A may be validated by user 114, such as a practicing physician. For example, the recommendation output may be categorized by user 114 as a “successful recommendation,” a “partially successful recommendation,” or a “failed recommendation” based on the accuracy of the recommendation.

[0089] Although 600 is shown as a discrete operation, such as 602, 604 and 606, in some embodiments, depending on the particular implementation, such discrete operations may be further divided into additional operations, combined into fewer operations or eliminated without departing from the essence of the disclosed embodiments.

[0090] Figure 7 This is an illustration of an exemplary scene for training a user using a 3D model from a 3D model database, according to embodiments of the present disclosure. Combined with... Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 To describe the elements Figure 7 . refer to Figure 7 Scene 700 is shown. In scene 700, a 3D model database 702, a patient history database 704, a user 706, a head-mounted device 708, a wearable haptic device 710, a 3D model 712, and a first region corresponding to the internal organ 712A in the 3D model 712 are shown.

[0091] Electronic device 102 can be configured to control head-mounted device 708 to present 3D model 712 for training purposes of user 706, who may be a medical student or intern. As an example, 3D model 712 can be presented as a VR object. Electronic device 102 can receive 3D model 712 from 3D model database 702 (associated with one or more hospitals). Furthermore, electronic device 102 can access patient history database 704, which may include a first set of characteristics associated with the test subject (such as...). Figure 4 (as described in) and the second set of features (such as) Figure 4 (As stated in the text).

[0092] For training purposes, electronic device 102 can present defective parts (e.g., scars or fractures) based on information accessed from patient history database 704. At any time, user 706 can touch any location on the 3D model 712. In response, electronic device 102 can receive tactile input and can extract a set of biosignals from patient history database 704. Such biosignals may be previously acquired or can be synthetically generated based on a mathematical model of tactile sensation. Electronic device 102 can control wearable haptic device 710 (worn by user 706) to generate tactile feedback based on this set of biosignals. For user 706, the tactile feedback can stimulate sensations typically experienced when touching defective parts (e.g., fractured knees) during a physical examination.

[0093] In one embodiment, the 3D model 712 may also include a first region corresponding to a patient's internal organ 712A (e.g., the lung). The patient history database 704 may include medical records of multiple patients with various types of health conditions and ailments, such as combinations of multiple defects. For example, the 3D model 712 associated with a human object may be generated based on a fracture in the human object's arm and cancer in a specific organ of the human object (such as internal organ 712A). The electronic device 102 may activate options (such as user interface (UI) elements or touch gestures on a head-mounted device 708) to allow a user to select and view different regions of the 3D model 712 and different data points associated with the 3D model 712 (including those in the patient history database 704).

[0094] Figure 8 This is a flowchart illustrating an exemplary method for human medical examination using touch according to embodiments of the present disclosure. Combined with... Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 Component description Figure 8 . refer to Figure 8 The example method of 800 is shown. This method can be performed by any computing system, for example, by... Figure 1 The electronic device 102 performs the operation. An exemplary method of 800 can begin at 802 and proceed to 804.

[0095] At 804, the first head-mounted display 104 can be controlled to present a first 3D model 120 of at least a first anatomical portion 118 of the body of the human object 116. According to an embodiment, circuitry 202 can be configured to control the first head-mounted display 104 to present the first 3D model 120 of at least the first anatomical portion 118 of the body of the human object 116. The presented first 3D model 120 may include a first region 122 corresponding to a defective portion in the first anatomical portion 116 of the body of the human object 116.

[0096] At point 806, touch input can be transmitted to a wearable sensor 108 that is in contact with at least a first anatomical portion 118 of the body. According to an embodiment, circuitry 202 can be configured to transmit touch input to the wearable sensor 108 that is in contact with at least a first anatomical portion 118 of the body of the human object 116. The transmitted touch input can correspond to a human touch on a first region 122 of the presented first 3D model 120.

[0097] At point 808, a set of biosignals associated with the defect can be received via wearable sensor 108 based on touch input. According to an embodiment, circuitry 202 can be configured to receive a set of biosignals associated with the defect via wearable sensor 108. This set of biosignals may include physiological signals and somatic sensory information associated with the defect, such as… Figure 4 As described in [the text].

[0098] At 810, the wearable haptic device 106 can be controlled to generate haptic feedback based on a received set of biosignals. According to an embodiment, circuitry 202 can be configured to control the wearable haptic device 106 to generate haptic feedback based on a received set of biosignals. The haptic feedback can be generated as a response to a human touch.

[0099] Although 800 illustrates discrete operations such as 804, 806, 808, and 810, this disclosure is not limited thereto. Therefore, in some embodiments, depending on the particular implementation, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated without departing from the essence of the disclosed embodiments.

[0100] Various embodiments of this disclosure may provide a non-transitory computer-readable medium and / or storage medium having stored thereon computer-executable instructions executable by a machine and / or computer (e.g., electronic device 102). The instructions may cause the machine and / or computer (e.g., electronic device 102) to perform operations including controlling a first head-mounted display 104 to present a first three-dimensional (3D) model 120 of at least a first anatomical portion 118 of the body of a human object 116. The presented first 3D model 120 may include a first region 122 corresponding to a defective portion in the first anatomical portion 118 of the body. Operations may also include transmitting touch input to a wearable sensor 108 in contact with at least the first anatomical portion 118 of the body. The transmitted touch input may correspond to a human touch on the first region 122 of the presented first 3D model 120. Operations may also include receiving a set of biosignals associated with the defective portion via the wearable sensor 108 based on the touch input. This set of biosignals may include physiological signals and somatic sensory information associated with the defective portion. Furthermore, operation may include controlling the wearable haptic device 106 to generate haptic feedback based on a set of received biosignals. The haptic feedback may be generated as a response to a human touch.

[0101] Exemplary aspects of this disclosure may include electronic devices (such as electronic device 102) that may include circuitry (such as circuitry 202) communicatively coupled to a first head-mounted display (such as first head-mounted display 104) and a wearable haptic device (such as wearable haptic device 106). Circuitry 202 may be configured to control the first head-mounted display 104 to present a first three-dimensional (3D) model (such as first 3D model 120) of at least a first anatomical portion (such as first anatomical portion 118) of the body of a human object (such as human object 116). The presented first 3D model 120 may include a first region (e.g., first region 122) corresponding to a defective portion in the first anatomical portion 118 of the body. Circuitry 202 may also be configured to transmit touch input to a wearable sensor (such as wearable sensor 108) in contact with at least the first anatomical portion 118 of the body. The transmitted touch input may correspond to a human touch on the first region 122 of the presented first 3D model 120. Circuit 202 may be further configured to receive a set of biosignals associated with the defect via wearable sensor 108 based on touch input. This set of biosignals may include physiological signals and somatosensory information associated with the defect. Circuit 202 may also be configured to control wearable haptic device 106 to generate haptic feedback based on the received set of biosignals. The haptic feedback may be generated as a response to human touch.

[0102] According to an embodiment, the first region 122 of the first 3D model 120 may correspond to a defective portion, which may be one of the following: tissue deformation or defect, abraded tissue or skin, fractured bone, tumor tissue or organ, deformed bone, or swelling or mass.

[0103] According to an embodiment, circuit 202 may be further configured to receive depth information associated with the defective portion from one or more imaging sensors, such as a first imaging sensor 302A, a second imaging sensor 302B, and a third imaging sensor 302C. Circuit 202 may also generate a first region 122 corresponding to the defective portion in the first anatomical portion 118 of the body based on the received depth information.

[0104] According to an embodiment, circuit 202 may also be configured to control a camera setup including one or more imaging sensors (such as a first imaging sensor 302A, a second imaging sensor 302B, and a third imaging sensor 302C) to scan at least a first anatomical portion 118 of the body of a human object 116. Circuit 202 may also receive 3D scan data of at least the first anatomical portion 118 of the body of the human object 116 based on the scan. Circuit 202 may also control a first head-mounted display 104 to present a first 3D model 120 of at least the first anatomical portion 118 of the body based on the received 3D scan data.

[0105] According to an embodiment, the 3D scan data may include a 3D representation of at least one internal organ of a human object 116.

[0106] According to an embodiment, circuit 202 may also be configured to receive a first set of features (such as first set of features 404C), which includes anthropometric features related to the body of human object 116 and medical condition information associated with human object 116. Circuit 202 may be further configured to generate a first 3D model 120 based on the received first set of features 404C. The generated first 3D model 120 may be displayed on the first head-mounted display 104.

[0107] According to an embodiment, circuit 202 may also be configured to receive a second set of features (such as second set of features 404C) associated with a defective portion in a first anatomical portion 118 of the body of the human object 116. The received second set of features 404C may include appearance attributes associated with the defective portion.

[0108] According to an embodiment, circuit 202 may also be configured to update the generated first 3D model 120 by mesh deformation of at least a first region 122 of the generated first 3D model 120. Mesh deformation may be applied based on the received second set of features 404C. The updated first 3D model 120 may be displayed on the first head-mounted display 104.

[0109] According to an embodiment, the second set of features 404C may include at least one of a physical deformation associated with at least a first anatomical portion 118 of the body or a deformation associated with one or more internal organs of the body.

[0110] According to one embodiment, somatic sensory information may include tactile information associated with the defective part, kinesthetic information associated with the defective part, and tactile perception information associated with the defective part.

[0111] According to an embodiment, circuit 202 can also be configured to generate input features for a machine learning model (such as machine learning model 204A) based on a received set of biosignals, medical condition information associated with human subject 116, and anthropometric features related to the body of human subject 116. Circuit 202 can also input the generated input features into machine learning model 204A. Machine learning model 204A outputs recommendations based on the input features.

[0112] According to an embodiment, the recommendations may include at least one of the following: a treatment process for the defective part, the current condition of the defective part, a prediction of the defective part, and a diagnosis related to the defective part.

[0113] According to an embodiment, circuit 202 can be further configured to extract recommendations from machine learning model 204A. Circuit 202 can also control the first head-mounted display 104 to present the recommendations.

[0114] According to an embodiment, circuit 202 may also be configured to transmit touch input to a tactile device (such as tactile device 110) that is in contact with at least a first anatomical portion 118 of the body of a human object 116. Tactile device 110 may receive the transmitted touch input and generate tactile sensation on the first anatomical portion 118 of the body based on the received touch input.

[0115] According to an embodiment, circuitry 202 may also be configured to control a second head-mounted display (such as second head-mounted display 502) worn on the head of a human object 116 to present the first 3D model 120 and a second 3D model including a human hand. The second 3D model may be presented to simulate the movement of a human hand when the hand applies a human touch to a first region 122 of the presented 3D first model 120. The second head-mounted display 502 may be controlled based on the received touch input.

[0116] This disclosure can be implemented in hardware or a combination of hardware and software. It can be implemented centrally in at least one computer system or distributedly, wherein different components may be distributed across several interconnected computer systems. A computer system or other apparatus suitable for performing the methods described herein may be appropriate. The combination of hardware and software can be a general-purpose computer system having a computer program that, when loaded and executed, can be controlled to cause it to perform the methods described herein. This disclosure can be implemented in hardware that includes a portion of an integrated circuit that also performs other functions.

[0117] This disclosure can also be embedded in a computer program product that includes all features enabling the implementation of the methods described herein and, when loaded into a computer system, is capable of executing those methods. In this context, a computer program represents any expression of a set of instructions in any language, code, or notation, which is intended to cause a system with information processing capabilities to directly perform a particular function, or to perform a particular function after one or both of: a) being translated into another language, code, or notation; or b) being reproduced in a different material form.

[0118] Although this disclosure has been described with reference to certain embodiments, those skilled in the art will understand that various changes can be made and equivalents can be substituted without departing from the scope of this disclosure. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this invention without departing from the scope of the invention. Therefore, this disclosure is not limited to the specific embodiments disclosed, but rather will include all embodiments falling within the scope of the appended claims.

Claims

1. An electronic device comprising: circuitry communicatively coupled to a first head-mounted display and a wearable haptic device, wherein the circuitry is configured to: control the first head-mounted display to present a first three-dimensional (3D) model of at least a first anatomical portion of a body of a human subject, wherein the presented first 3D model includes a first region corresponding to a defective portion in the first anatomical portion of the body; transmit a touch input to a wearable sensor in contact with at least the first anatomical portion of the body, wherein the transmitted touch input corresponds to a human touch on the first region of the presented first 3D model; receive, via the wearable sensor, a set of biosignals associated with the defective portion based on the touch input, the set of biosignals including physiological signals and somatosensory information associated with the defective portion; control the wearable haptic device to generate haptic feedback based on the received set of biosignals, the haptic feedback generated as a response to the human touch; receive a first set of features including anthropometric features related to the body and medical condition information associated with the human subject; generate input features for a machine learning model based on the received set of biosignals, the medical condition information associated with the human subject, and the anthropometric features related to the body; and input the generated input features to the machine learning model, wherein the machine learning model outputs a recommendation based on the input features.

2. The electronic device of claim 1, wherein the first region of the first 3D model corresponds to the defective portion, the defective portion being one of: a tissue deformation or defect, a bruised tissue or skin, a fractured bone, a tumor tissue or organ, a deformed bone, or a swelling or lump. 3.The electronic device of claim 1, wherein the circuitry is further configured to: receive, from one or more imaging sensors, depth information associated with the defective portion; and generate, based on the received depth information, the first region corresponding to the defective portion in the first anatomical portion of the body. 4.The electronic device of claim 1, wherein the circuitry is further configured to: control a camera rig including one or more imaging sensors to scan at least the first anatomical portion of the body of the human subject; receive, based on the scan, 3D scan data of at least the first anatomical portion of the body of the human subject; and control the first head-mounted display to present the first 3D model of at least the first anatomical portion of the body based on the received 3D scan data. 5.The electronic device of claim 4, wherein, the 3D scan data includes a 3D representation of at least one internal organ of the human subject. 6.The electronic device of claim 1, wherein the circuitry is further configured to: generate the first 3D model based on the received first set of features, wherein the generated first 3D model is presented on the first head-mounted display. 7.The electronic device of claim 1, wherein the circuitry is further configured to receive a second set of features related to the defective portion in the first anatomical portion of the body of the human subject, and wherein the received second set of features includes appearance attributes associated with the defective portion.

8. The electronic device of claim 7, wherein the circuitry is further configured to update the generated first 3D model by mesh deformation of at least the first region of the generated first 3D model, wherein, applying mesh deformation based on the received second set of features, and presenting the updated first 3D model on the first head-mounted display. 9.The electronic device of claim 7, wherein the second set of features includes at least one of a physical deformity associated with at least the first anatomical portion of the body or a deformity associated with one or more internal organs of the body. 10.The electronic device of claim 1, wherein the somatosensory information includes tactile information associated with the defect portion, kinesthetic information associated with the defect portion, and tactile perception information associated with the defect portion. 11.The electronic device of claim 1, wherein the recommendation includes at least one of a treatment procedure for the defect portion, a current condition of the defect portion, a prediction for the defect portion, and a diagnosis related to the defect portion. 12.The electronic device of claim 1, wherein, the circuitry is further configured to: extract the recommendation from the machine learning model; and control the first head-mounted display to present the recommendation. 13.The electronic device of claim 1, wherein, the circuitry is further configured to transmit the touch input to a haptic device in contact with at least the first anatomical portion of the body of the human subject, wherein the haptic device: receives the transmitted touch input; and generates a haptic sensation on the first anatomical portion of the body based on the received touch input. 14.The electronic device of claim 1, wherein, the circuitry is further configured to: control a second head-mounted display worn on a head of the human subject to present the first 3D model and a second 3D model including a human hand, wherein present the second 3D model to simulate movement of the human hand when the human hand exerts human touch on a first region of the presented first 3D model, and control the second head-mounted display based on the received touch input.

15. A method comprising: controlling a first head-mounted display to present a first three-dimensional (3D) model of at least a first anatomical portion of a body of a human subject, wherein the presented first 3D model includes a first region corresponding to a defect portion in the first anatomical portion of the body; transmitting a touch input to a wearable sensor in contact with at least the first anatomical portion of the body, wherein the transmitted touch input corresponds to human touch on the first region of the presented first 3D model; receiving, via the wearable sensor, a set of biosignals associated with the defect portion based on the touch input, the set of biosignals including physiological signals and somatosensory information associated with the defect portion; controlling a wearable haptic device to generate haptic feedback based on the received set of biosignals, the haptic feedback being generated as a response to the human touch; receiving a first set of features including anthropometric features related to the body and medical condition information associated with the human subject; generating input features for a machine learning model based on the received set of biosignals, the medical condition information associated with the human subject, and the anthropometric features related to the body; and ​ inputting the generated input features to the machine learning model, wherein the machine learning model outputs a recommendation based on the input features.

16. The method of claim 15, further comprising: extracting the recommendation from the machine learning model; and controlling the first head-mounted display to present the recommendation.

17. The method of claim 15, further comprising: transmitting the touch input to a haptic device in contact with at least the first anatomical portion of the body of the human subject; receiving, by the haptic device, the transmitted touch input; and generating, by the haptic device, a haptic sensation on the first anatomical portion of the body based on the received touch input.

18. A non-transitory computer-readable medium having stored thereon computer- executable instructions that, when executed by an electronic device, cause the electronic device to perform operations comprising: controlling a first head-mounted display to present a first three-dimensional (3D) model of at least a first anatomical portion of a body of a human subject, wherein the presented first 3D model includes a first region corresponding to a defective portion in the first anatomical portion of the body; transmitting a touch input to a wearable sensor in contact with at least the first anatomical portion of the body, wherein the transmitted touch input corresponds to a human touch on the first region of the presented first 3D model; based on the touch input, receiving, via the wearable sensor, a set of biosignals associated with the defective portion, the set of biosignals including physiological signals and somatosensory information associated with the defective portion; controlling a wearable haptic device to generate haptic feedback based on the received set of biosignals, the haptic feedback being generated as a response to the human touch; receiving a first set of features including anthropometric features related to the body and medical condition information associated with the human subject; generating input features for a machine learning model based on the received set of biosignals, medical condition information associated with the human subject, and anthropometric features related to the body; and inputting the generated input features to the machine learning model, wherein the machine learning model outputs a recommendation based on the input features.

Images