Audiovisual interaction with an implanted device

Abstract

A method for audiovisual interaction of a user with an implanted device to resolve a voice command of the user is disclosed. The method includes one or more processors determining a health-related question of the user corresponding to the voice command of the user. The method further includes identifying an implanted device of the user related to the health-related question. The method further includes collecting sensor data of the implanted device, wherein the sensor data includes one or more conditions of the implanted device and an operating environment of the implanted device in the body of the user. The method further includes generating an audiovisual response corresponding to the health-related question of the user.

Description

BACKGROUND

[0001] The present invention relates generally to the field of virtual assistants, and more specifically to audiovisual interaction with a body-implanted device.

[0002] In recent years, the development of digital assistants and the growth of Internet of Things (IoT) capable devices have created a race to introduce new voice interfaces (e.g., for smart speakers, virtual assistant hardware / software, etc.) and functionality. Cognitive analytics combines the use of cognitive computing and analytics. Cognitive computing combines artificial intelligence and machine learning algorithms in an approach that attempts to replicate human brain behavior. Analytics is the scientific process of transforming data into insights to make better decisions. Cognitive analytics applies intelligent technology to make unstructured data sources available across the range of the analytics process for decision and improvement functionality.

[0003] Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are augmented, or supplemented, by computer-generated sensory information, sometimes across multiple sensory modalities. AR can be defined as a system that implements all three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects. The sensory information that is overlaid can be constructive or destructive. The main value of augmented reality is the way in which the components of the digital world are merged into the perception of the real world, not as a simple display of data, but as an integrated, immersive sensory experience.

[0004] An implant is a medical device manufactured for the replacement, support, and / or augmentation of a biological structure. An active implantable medical device is any active medical device intended to be totally or partly introduced, surgically or medically, into the human body or by medical intervention into the natural orifice and intended to remain after the procedure. Medical devices benefit patients by helping health care providers diagnose and treat patients and helping patients overcome disease or illness, improve their quality of life. SUMMARY

[0005] Aspects of the present invention disclose a method, computer program product, and system for audiovisual interaction of a user with an implanted device to resolve a voice command of the user. The method includes one or more processors determining a health-related question of the user corresponding to the voice command of the user. The method further includes one or more processors identifying an implanted device of the user related to the health-related question. The method further includes one or more processors collecting sensor data of the implanted device, wherein the sensor data includes one or more conditions of the implanted device and an operating environment of the implanted device within the user. The method further includes one or more processors generating an audiovisual response corresponding to the health-related question of the user based at least in part on the sensor data. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1 is a functional block diagram of a data processing environment according to embodiments of the application.

[0007] Figure 2 is a flowchart depicting operational steps of a program for audiovisual interaction of a user with an implanted device to resolve voice commands of the user according to embodiments of the application.

[0008] Figure 3 is a block diagram of components of Figure 1 according to embodiments of the application. DETAILED DESCRIPTION

[0009] Embodiments of the application allow for audiovisual interaction of a user with an implanted device to resolve voice commands of the user. Embodiments of the application determine a condition of one or more implanted devices of the user. Embodiments of the application enable the user to provide voice interaction with the one or more implanted devices. Additional embodiments of the application generate recommended activities for the user based on a durability limit of the user and / or parameters of the one or more implanted devices. Further embodiments of the application generate audiovisual responses to voice commands of the user based on data feeds of the one or more implanted devices.

[0010] Some embodiments of the application recognize that various implanted devices are susceptible to external electronic devices and implant failure. For example, implant failure refers to any medical implant failing to meet the requirements involved in installation by the manufacturer or health care provider. Implant failure can have many causes, such as mechanical degradation in the form of wear / fatigue, failure associated with exposure to high voltage electricity or high intensity microwaves, or electrochemical degradation in the form of corrosion. However, different implants fail at different rates. Embodiments of the application help prevent failure of implanted devices by identifying a condition of the implanted device in real-time and generating recommended activities within a durability limit of the implanted device.

[0011] Various embodiments of the application can operate to improve implanted devices by utilizing data of sensors to determine whether the implanted device is operating within defined performance parameters. Additionally, embodiments of the application can identify the root cause of poor performance. Furthermore, embodiments of the application enable voice interaction with implanted devices utilizing Internet of Things (IoT) connectivity.

[0012] Implementation of embodiments of the application can take a variety of forms and exemplary implementation details are discussed subsequently with reference to the accompanying drawings.

[0013] The present application will now be described in detail with reference to the drawings. Figure 1 is a functional block diagram of a distributed data processing environment, generally designated 100, according to one embodiment of the application. Figure 1Only an illustration of one implementation is provided, and this description is not intended to be limiting. Numerous modifications to the described implementations can be made by those skilled in the art without departing from the scope of the present application as recited by the claims.

[0014] The present application can include various accessible data sources, such as database 144, implanted device 130, or client device 120 1-N that can include personal data, content, or information that a user wishes not to be processed. Personal data includes personally identifying information or sensitive personal information as well as user information such as tracking or geolocation information. Processing refers to any automated or non-automated operation or set of operations, such as collection, recording, organization, structuring, storage, adaptation, alteration, retrieval, consultation, use, disclosure or otherwise making available (through transmission, distribution or otherwise) to others, combination, restriction, erasure, or destruction performed on personal data. Interaction program 200 enables authorized and secure processing of personal data. Interaction program 200 provides informed consent, notifies of collection of personal data, allows a user to opt in to processing of personal data or opt out of processing of personal data. Consent can take several forms. Opt-in consent can force a user to take a positive action before processing of personal data. Alternatively, opt-out consent can force a user to take a positive action to prevent processing of personal data before processing of personal data. Interaction program 200 provides information about the nature of the personal data and the processing (e.g., type, scope, purpose, duration, etc.). Interaction program 200 provides a user with a copy of stored personal data. Interaction program 200 allows for correction of incorrect or incomplete personal data, or making it complete. Interaction program 200 allows for immediate deletion of personal data.

[0015] Distributed data processing environment 100 includes server 140, implanted device 130, and client devices 120 1-N all interconnected through network 110. Network 110 can be, for example, a telecommunications network, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN) such as the Internet, or a combination of these, and can include wired, wireless, or fiber optic connections. Network 110 can include one or more wired and / or wireless networks capable of carrying data, voice, and / or video signals including multimedia signals that include voice, data, and video information. In general, network 110 can be any combination of connections and protocols that will 1-N support communications between server 140, implanted device 130, and client devices 120

[0016] Client devices 120 1-N (i.e., client device 120 n) can be one or more of a laptop computer, a tablet computer, a smart phone, a smart watch, a smart speaker, a virtual assistant, an augmented reality (AR) glasses, an Internet of Things (IoT) capable device, or any programmable electronic device capable of communicating via a network 110 with various components and devices within the distributed data processing environment 100. Generally, client devices 120 1-N represent one or more programmable electronic devices or combinations of programmable electronic devices that are capable of executing machine-readable program instructions and communicating with other computing devices (not shown) within the distributed data processing environment 100 via a network such as network 110. According to embodiments of the present application, client devices 120 1-N may include respective instances of components described and illustrated further in detail below. Figure 3

[0017] Client devices 120 1-N may include respective instances of user interfaces 122 1-N , applications 124 1-N , and sensors 126 1-N , each of which correspond to and perform equivalent functions in respective instances of client devices. In various embodiments of the present application, a user interface is a program that provides an interface between a user of a device and a plurality of application programs resident on a client device. A user interface such as user interface 1221 involves both the information presented to the user by the program, such as graphics, text and sound, and the control sequences the user uses to control the program. There are various types of user interfaces. In one embodiment, user interface 1221 is a graphical user interface. A graphical user interface (GUI) is a type of user interface that allows users to interact with electronic devices (such as computer keyboards and mice) through graphical icons and visual indicators (e.g., helper symbols), rather than having to type commands or navigate text menus. In computing, the introduction of the GUI was in response to the perceived steep learning curve of command-line interfaces, which required typing commands on a keyboard. Actions in a GUI are typically performed through direct manipulation of graphical elements. In another embodiment, user interface 1221 is a script or an application programming interface (API). In one embodiment, user interface 1221 is a voice user interface. A voice-user interface (VUI) enables spoken interaction between humans and computers, using speech recognition to understand spoken commands and answer questions, and typically using text-to-speech to play back replies.

[0018] ​Application 1241 is a computer program designed to run on client device 1201. Applications are often used to provide similar services to those accessed on personal computers (e.g., web browsers, playing music, email programs, or other media, etc.) In one embodiment, application 1241 is a mobile application software. For example, mobile application software or "apps" are computer programs designed to run on smart phones, tablet computers, and other mobile devices. In another embodiment, application 1241 is a web user interface (WUI) and can display text, documents, web browser windows, user options, application interfaces, and operating instructions, and includes information (e.g., graphics, text, and sound) that the program presents to the user and control sequences that the user uses to control the program. In another embodiment, application 1241 is a client-side application of interactive program 200. For example, application 1241 is an augmented reality application that combines virtual reality with the real world in the form of live video images that are digitally enhanced with computer-generated graphics.

[0019] Sensor 1261 is a device, module, machine, or subsystem that detects events or changes in its environment and sends the information to other electronic devices, typically a computer processor. In general, sensor 1261 represents a variety of sensors of client device 1201 that collect and provide various data (e.g., proximity, images, motion, electromagnetic fields, radio frequency, light, etc.). In one embodiment, client device 1201 transmits data of sensor 1261 to server 140 via network 110. For example, sensor 1261 can be a camera that client device 1201 uses to capture images of a user's environment that are transmitted to a remote server (e.g., server 140).

[0020] In general, implanted device 130 represents one or more programmable electronic devices or combinations of programmable electronic devices that are capable of executing machine-readable program instructions and communicating with other computing devices (not shown) within distributed data processing environment 100 via a network such as network 110. In accordance with embodiments of the present application, implanted device 130 can include components that are further described and depicted in detail below. Figure 3 Further detailed depiction and description of components.

[0021] Implant device 130 is a medical device manufactured to replace missing biological structure, support a damaged biological structure, augment an existing biological structure, and / or local drug / therapy delivery. Implant device 130 includes a sensor array 132. In one embodiment, implant device 130 utilizes sensor array 132 to monitor the functionality of implant device 130 and / or the operating environment of implant device 130. For example, sensor array 132 is one or more sensors of a medical device implanted into a user (e.g., a joint replacement). In this example, implant device 130 is paired with a portable virtual assistant (e.g., client device 120i) via a WLAN (e.g., network 110) to provide monitoring of the health of the user and the condition of implant device 130. Additionally, sensor array 132 can provide sensor feed data to the portable virtual assistant and a remote server (e.g., server 140).

[0022] In various embodiments of the present application, server 140 can be a desktop computer, a computer server, or any other computer system known in the art. Generally, server 140 represents any electronic device or combination of electronic devices capable of executing computer readable program instructions. According to embodiments of the present application, server 140 can include components as further described and depicted in detail below. Figure 3

[0023] Server 140 can be a standalone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting and processing data. In one embodiment, server 140 can represent a server computing system utilizing multiple computers as a server system, such as in a cloud computing environment. In another embodiment, server 140 can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smartphone, or any programmable electronic device capable of communicating with client device 120 1-N , implant device 130, and other computing devices (not shown) within distributed data processing environment 100 via network 110. In another embodiment, server 140 represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single seamless resource pool when accessed within distributed data processing environment 100.

[0024] Server 140 includes a storage device 142, a database 144, and an interaction program 200. Storage device 142 can be implemented with any type of storage device, such as a hard disk drive, a solid state drive, a tape drive, a flash drive, a memory card, or any other type of storage device capable of storing computer readable program instructions that can be executed by client device 120 1-N ​, a permanent storage device 305, hard drive, or flash memory that the implanted device 130 and servers 140, such as a database server, access and utilize for storage of data. In one embodiment, the storage device 142 can represent multiple storage devices within the server 140. In various embodiments of the present application, the storage device 142 stores multiple types of data that can include a database 144. The database 144 can represent one or more organized collections of data stored and accessed from the server 140. For example, the database 144 includes a corpus of knowledge corresponding to users, sensor feed data, medical records, implanted device parameters, and the like. In one embodiment, the data processing environment 100 can include additional servers (not shown) that host additional information accessible via the network 110.

[0025] Generally, the interaction program 200 communicates with wearable and IoT devices, such as virtual assistants, that utilize sensor data from body-implanted devices to determine a user's daily workload and health status and recommend alternatives to reduce the user's stress. In one embodiment, the interaction program 200 utilizes data from the sensor array 132 to determine the condition of the implanted device 130 and the health status of the user. In another embodiment, the interaction program 200 utilizes the client device 120 1-N to enable the user to communicate with one or more instances of the implanted device 130. In another embodiment, the interaction program 200 utilizes the client device 120 1-N to provide the user with a visual display of the current condition of the implanted device 130. Also, the interaction program 200 is capable of utilizing a live feed of the operating environment of the client device 120 1-N to identify and recommend activities with respect to performance parameters of the implanted device 130 and / or the health status of the user via AR overlays.

[0026] Figure 2 is a flowchart illustrating operational steps of the interaction program 200 for audiovisual interaction of a user with an implanted device to resolve voice commands of the user, according to an embodiment of the present application. In one embodiment, the interaction program 200 initiates in response to the user pairing the implanted device 130 with an instance of the interaction program 200 and the client device 120 1-N via the network 110. For example, the interaction program 200 initiates in response to the user registering (e.g., opting in) a portable virtual assistant (e.g., the client device 120 1-N ) with the interaction program 200 via a WLAN (e.g., the network 110). In another embodiment, the interaction program 200 is a background application that continuously monitors the client device 120 1-N . For example, the interaction program 200 is a background application that continuously monitors the client device 120 1-N) a client-side application (e.g., application 1241) that initiates and monitors the portable virtual assistant to find voice commands.

[0027] At step 202, the interaction program 200 identifies a voice command of the user. Various embodiments of the present application enable the user to perform voice interactions with one or more implanted devices (e.g., implanted device 130) of the user to determine a condition of the one or more implanted devices. For example, a portable AI voice assistant (e.g., client device 1201) is paired with an implanted device of the user and / or IoT-enabled sensors of the implanted device to allow the user to perform voice interactions with the implanted device (i.e., based on voice commands, the user can perform two-way voice interactions with the one or more implanted devices). Additionally, the user is allowed to ask specific questions related to the one or more implanted devices or specify a problem that the user is experiencing.

[0028] In one embodiment, the interaction program 200 identifies a voice command from the client device 1201 of the user. For example, the interaction program 200 utilizes natural language processing (NLP) techniques (e.g., speech recognition, speech segmentation, tokenization, etc.) to identify a subject matter of a voice command of the user to the portable virtual assistant (e.g., client device 1201). In this example, the interaction program 200 can utilize lexical semantics on a textual representation of audio of the user that includes the voice command received by the portable virtual assistant to identify the subject matter. In an alternative example, the interaction program 200 connects the portable virtual assistant (e.g., client device 1201) to the ingestible / injectable electronic device (e.g., implanted device 130) to perform two-way voice interactions with the ingestible / injectable electronic device.

[0029] In another embodiment, the interaction program 200 generates a knowledge corpus corresponding to the user in the storage device 142. For example, the interaction program 200 collects historical data from various sources (e.g., sensors, medical records, etc.) to generate a knowledge corpus (e.g., database 144) that includes identified subject matters corresponding to voice commands, sensor feed data, health conditions, implanted device parameters, etc. of the user. In this example, the interaction program 200 connects (e.g., hosts on a cloud-based server) the portable virtual assistant (e.g., client device 1201) of the user to the knowledge corpus.

[0030] In decision step 204, the interaction program 200 determines whether the voice command is related to the user's health issue. In one embodiment, the interaction program 200 utilizes data corresponding to the user in the database 144 to determine whether the user's voice command is related to the implanted device 130 and / or the user's condition. For example, the interaction program 200 utilizes a knowledge corpus (e.g., the database 144) to determine whether the topic of the user's voice command is related to one or more implanted medical devices (e.g., the implanted device 130) of the user. Further, the interaction program 200 utilizes a knowledge corpus (e.g., the database 144) to determine whether the topic of the user's voice command is related to a health issue (e.g., a health condition, medication delivery, pain, etc.) of the user. In this example, the interaction program 200 utilizes NLP techniques (e.g., term extraction, lexical semantics, topic segmentation and identification, etc.) to determine a relationship between the topic of the voice command and the user's issue.

[0031] In another embodiment, if the interaction program 200 determines that the user's voice command is not related to the implanted device 130 and / or the user's condition (decision step 204, "No" branch), the interaction program 200 continues to identify the voice command from the user's client device 1201. In one scenario, if the interaction program 200 determines that the topic of the user's voice command is not related to a question about the user's body, health, or the status of one or more implanted medical devices (e.g., the implanted device 130), the interaction program 200 continues to identify the topic of the user's voice command to the portable virtual assistant (e.g., the client device 1201).

[0032] In another embodiment, if the interaction program 200 determines that the user's voice command is related to the implanted device 130 and / or the user's condition (decision step 204, "Yes" branch), the interaction program 200 identifies an instance of the implanted device 130 related to the user's voice command. In one scenario, if the interaction program 200 determines that the topic of the user's voice command is related to a question about the user's body, health, or the status of one or more implanted medical devices (e.g., the implanted device 130), the interaction program 200 utilizes a knowledge corpus (e.g., the database 144) corresponding to the user to identify an implanted medical device of the user related to the question.

[0033] In step 206, the interaction program 200 identifies an implanted device corresponding to the health issue. In various embodiments of the present application, the implantable device can include various types of sensors (e.g., temperature sensors, movement sensors, pressure sensors, etc.). The selection of the sensor can be based on the functionality of the implantable device (e.g., orthopedic, cardiovascular, etc.). For example, the interaction program 200 identifies a wearable and implantable device (e.g., the implanted device 130, the client device 120 1-NThe sensor feeds of the implanted devices and the medical reports of the user are associated to generate a corpus of knowledge about health-related parameters and functions (e.g., implanted device malfunctions, correlations of body parameters to performance of implanted devices, etc.).

[0034] In one embodiment, the interaction program 200 utilizes the database 144 to identify instances of implanted devices 130 that are relevant to the voice commands of the user. For example, the interaction program 200 assigns a unique identifier (e.g., a radio frequency identifier (RFID), a series of values, etc.) to one or more implanted medical devices (e.g., implanted device 130) of the user and associates the location within the user with the unique identifier. In this example, the interaction program 200 utilizes the subject matter of the voice commands of the user (e.g., pain, function, health issues, implanted status, etc.) and the corpus of knowledge (e.g., database 144) to identify implanted medical devices (e.g., implanted device 120) that correspond to the subject matter of the voice commands.

[0035] In decision step 208, the interaction program 200 determines whether the implanted device is affected by an external device. In various embodiments of the present application, each implanted device generates sensor feeds that correspond to the mobility and activity of the user and the performance of the implanted device. In addition, the interaction program 200 can utilize the sensor feeds of the normal functioning of each implanted device to specify values of corresponding performance parameters of the sensors.

[0036] In one embodiment, the interaction program 200 determines whether the client device 1202 affects the performance of the implanted device 130. For example, the interaction program 200 monitors one or more sensors (e.g., sensor array 132) of an implanted medical device (e.g., implanted device 130) to identify deviations from expected sensor values of performance parameters that correspond to the sensor feeds of the implanted medical device. In this example, the interaction program 200 determines whether the deviations are a result of the implanted medical device receiving sensor feeds from a device (e.g., an IoT device, client device 1202, etc.) within the operating environment (e.g., the body of the user, the immediate surrounding environment of the body of the user, etc.) of the implanted medical device that affect the performance of the implanted medical device. In one scenario, the interaction program 200 utilizes the normal performance parameters of the electrical sensors (e.g., sensor array 132) of a pacemaker (e.g., implanted device 130) to detect that an external device (e.g., client device 1202) is emitting electromagnetic waves that affect the performance of the pacemaker. In addition, the interaction program 200 can identify one or more implanted medical devices of the user that are affected by the pacemaker.

[0037] In another embodiment, if the interaction program 200 determines that the client device 1202 is affecting the performance of the implanted device 130 (decision step 208, "Yes" branch), the interaction program 200 generates an AR object corresponding to the implanted device 130 and the client device 1202. In one scenario, the interaction program 200 determines that the deviation of the expected sensor values of the performance parameters corresponding to the sensor feed of the implanted medical device (e.g., implanted device 130) is caused by a device (e.g., IoT device, client device 1202, etc.) within the operating environment of the implanted medical device (e.g., user's body, the immediate surrounding environment of the user's body, etc.). As a result, the interaction program 200 generates a three-dimensional (3D) object representing the implanted medical device (e.g., implanted device 130) in a virtual reality (VR) environment that displays an indicator in the direction of the device affecting the implanted medical device via wearable VR glasses (e.g., client device 1203).

[0038] In another embodiment, if the interaction program 200 determines that the client device 1202 is not affecting the performance of the implanted device 130 (decision step 208, "No" branch), the interaction program 200 generates a response to the voice command from the user's client device 1201. In one scenario, if the interaction program 200 determines that there is no deviation of the expected sensor values of the performance parameters corresponding to the sensor feed of the implanted medical device (e.g., implanted device 130), the interaction program 200 generates an audiovisual response to the user's voice command, as discussed below in step 210.

[0039] In step 210, the interaction program 200 generates a response corresponding to the user's voice command. In one embodiment, the interaction program 200 generates an audible response corresponding to the user's voice command of the client device 1201. For example, the interaction program 200 utilizes natural language generation (NLG) to convert information from a knowledge corpus (e.g., database 144) corresponding to the implanted medical device (e.g., implanted device 130) identified in step 206 into a readable human language. In this example, the interaction program 200 utilizes text-to-speech technology to create an audio version of the collected user data suitable for communicating the cause of the identified user or implanted medical device's condition. Additionally, the interaction program 200 uses a portable virtual assistant (e.g., client device 1201) to transmit an audible response corresponding to the user's voice command to the user.

[0040] In another example, the interaction program 200 can identify a current state of the user's implanted medical device (e.g., the implanted device 130) based on the collected user data (e.g., biometric data, sensor data, etc.). In this example, the interaction program 200 utilizes the durability limits of the implanted medical device of the knowledge corpus (e.g., the database 144) and sensor feed data (e.g., performance parameters) of the implanted device to determine whether the implanted device has exceeded a rated workload, is performing within rated constraints, or has stopped functioning normally (i.e., determine the current state based on deviations from normal performance parameters based on sensors such as pressure, temperature, etc.). Additionally, the interaction program 200 can utilize data (e.g., images, pressure readings, etc.) of one or more sensors (e.g., the sensor array 132) of the implanted medical device to determine a condition of the user's body at the location of the implanted medical device (e.g., determine that the person is feeling pain due to inflammation around a knee replacement). In another example, the interaction program 200 can determine a future state of the implanted medical device (e.g., the implanted device 130) by using performance parameters to estimate whether a current workload of the implanted medical device over a period of time will exceed the user's durability limits.

[0041] In step 212, the interaction program 200 generates an augmented reality model corresponding to the response. In one embodiment, the interaction program 200 utilizes data of the sensor array 132 to generate an AR model of the implanted device 130 and the operating environment. Also, the interaction program 200 utilizes the client device 1203 to display the AR model to the user. For example, the interaction program 200 can utilize images of the implantable medical device (e.g., the implanted device 130) to generate a 3D AR object corresponding to the implantable medical device. In this example, the interaction program 200 utilizes images from the sensor feed (e.g., the sensor array 132) of the implanted medical device to generate a visual representation of the health of the operating environment of the user (e.g., the area in which the implant is located in the body). Additionally, the interaction program 200 can utilize the sensor feed from the implanted medical device to identify movement and performance information of the implanted medical device (i.e., allowing for real-time updates of the current status and actions of the implanted device 130). Furthermore, the interaction program 200 can utilize a pair of AR glasses (e.g., the client device 1203) to display the 3D AR object and the operating environment to the user in response to a voice command of the user, which facilitates communicating the generated response information to the user.

[0042] In another embodiment, the interaction program 200 utilizes data from sensors 1263 of the client device 1203 to generate a simulated model of the AR model of the implanted device 130. For example, the interaction program 200 utilizes image data from a user of an AR glasses (e.g., client device 1203) and an IoT device (e.g., client device 1202) to simulate performance of a 3D AR object corresponding to an implanted medical device (e.g., implanted device 130). In one scenario, if the interaction program 200 determines that the implanted medical device is affected by an external device that emits electromagnetic waves, the interaction program 200 can utilize image data from the AR glasses to display an indication of the direction of the external device (e.g., client device 1202).

[0043] At step 214, the interaction program 200 identifies recommended activities for the user corresponding to the response. In one embodiment, the interaction program 200 utilizes data from sensors 1263 of the client device 1203 to determine activities to be performed by the user based on the implanted device 130. For example, the interaction program 200 utilizes a camera feed and object recognition of the AR glasses (e.g., client device 1203) to identify activities that adversely affect the implanted medical device (e.g., implanted device 130) based on the current condition of the implanted device or the health status of the user. In this example, the interaction program 200 utilizes the overlay to identify (e.g., highlight, bind, etc.) objects in the area surrounding the user that can cause the user to exceed the endurance limit of the implanted medical device. Additionally, the interaction program 200 utilizes the AR glasses to display the recommended activities and prohibited activities in real-time.

[0044] In one scenario, the interaction program 200 identifies a skateboard, stairs, and an elevator (e.g., objects) in the area surrounding the user. Additionally, the interaction program 200 determines that the user has an implanted knee (e.g., implanted device 130) with swelling (e.g., the current health condition of the user). As a result, the interaction program 200 sends visual recommendations to the user via the AR glasses (e.g., client device 1203) by inputting images indicating approval or prohibition of activities related to the objects into the overlay corresponding to each object in the feed of the AR glasses. Thus, the interaction program 200 approves the use of the elevator and prohibits the use of the stairs and the skateboard by the user based on the capabilities of the implanted knee (e.g., the implant is rated not to withstand the pressure caused by the skateboard) and the user’s condition (e.g., the knee swelling).

[0045] Figure 3 A block diagram of components of the client device 120, the implanted device 130, and the server 140 according to illustrative embodiments of the present application is described. It should be understood that 1-N , the implanted device 130, and the server 140. It should be understood that Figure 3 The above-described code can be further stored in the form of software program instructions in an information storage medium and is implemented in a processor. The software program instructions can be provided to the processor of a computer using a storage medium. The storage medium is readable by the processor of the computer and stores a data file in which the software program instructions are stored. The software program instructions can be provided to the processor of the computer using a storage medium. The storage medium is readable by the processor of the computer and stores a data file in which the software program instructions are stored.

[0046] Figure 3 includes a processor 301, a cache 303, a memory 302, a persistent storage device 305, a communication unit 307, an input / output (I / O) interface 306, and a communication fabric 304. The communication fabric 304 provides communication between the cache 303, the memory 302, the persistent storage device 305, the communication unit 307, and the input / output (I / O) interface 306. The communication fabric 304 can be implemented with any architecture designed for passing data and / or control information between processors such as microprocessors, communication and network processors, and any other hardware components within a system. For example, the communication fabric 304 can be implemented with one or more buses or cross-over switches.

[0047] The memory 302 and the persistent storage device 305 are computer readable storage media. In this embodiment, the memory 302 includes random access memory (RAM). Generally, the memory 302 can include any suitable volatile or non-volatile computer readable storage media. The cache 303 is a fast memory that enhances the performance of the processor 301 by holding recently accessed data and holding data near recently accessed data.

[0048] Program instructions and data (e.g., software and data 310) used to practice embodiments of the present application can be stored in the persistent storage device 305 and the memory 302 for execution by one or more respective processors 301 via the cache 303. In an embodiment, the persistent storage device 305 includes a magnetic hard disk drive. As an alternative or in addition to a magnetic hard disk drive, the persistent storage device 305 can include a solid state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer readable storage media that can store program instructions or digital information.

[0049] The media used by the persistent storage device 305 can also be removable. For example, a removable hard drive can be used for the persistent storage device 305. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer readable storage media, also a part of persistent storage device 305. The software and data 310 can be stored on the persistent storage device 305 for access and / or execution by one or more respective processors 301 via the cache 303. With respect to the client device 120 1-N , the software and data 310 include the user interface 122 1-N , the applications 124 1-N , the sensors 126 1-NData about the implant device 130, the software and data 310 include data for the user sensor array 132. Data about the server 140, the software and data 310 include data for the storage device 142 and the interaction program 200.

[0050] In these examples, the communication units 307 provide communications with other data processing systems or devices. In these examples, the communication units 307 include one or more network interface cards. The communication units 307 can provide communications through the use of either or both physical and wireless communications links. The program instructions and data used to practice embodiments of the present application, for example, the software and data 310, can be downloaded to the persistent storage 305 through the communication units 307.

[0051] The I / O interface 306 allows for input and output of data with other devices that can be connected to each computer system. For example, the I / O interface 306 can provide a connection to an external device 308 such as a keyboard, a keypad, a touch screen, and / or some other suitable input device. The external device 308 can also include a portable computer- readable storage medium, such as, for example, a thumb drive, an external hard drive, an

[0052] The display 309 provides a mechanism to display data to a user, and can be, for example, a computer monitor.

[0053] The programs described herein are identified based upon the application for which they are implemented in a particular embodiment of the application. However, it should be understood that any particular program identified herein can be implemented with program instructions as will be apparent to those skilled in the art, and that such program instructions can be executed by a processor to implement the application as set forth in any aspect of the application. It is to be understood that the foregoing description is exemplary of the application only and is intended to provide an overview for understanding the nature and character of the application as it is claimed. It is to be understood that the application is not to be limited to the specific forms of the application herein described and that numerous modifications can be made to the application as described without departing from the spirit and scope of the application as set forth in any aspect of the application.

[0054] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0055] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or

[0056] The computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions into the respective computing / processing device for storage in a computer readable storage medium within the respective computing / processing device.

[0057] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and a procedural programming language such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0058] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0059] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0060] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0061] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0062] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for audiovisual interaction with an implanted device, the method comprising: The user's health-related problems are determined by one or more processors based at least in part on the user's historical medical data and the user's voice commands; An implantable device for a user, identified by one or more processors in relation to the user's health-related problems determined at least in part based on the user's historical medical data and the user's voice commands; One or more processors collect sensor data from the implanted device, wherein the sensor data includes one or more conditions of the implanted device and the operating environment of the implanted device in the user's body; as well as An audiovisual response corresponding to the user's health-related problem is generated by one or more processors, based at least in part on sensor data including the one or more conditions of the implanted device and the operating environment of the implanted device in the user's body.

2. The method according to claim 1, further comprising: A three-dimensional augmented reality (AR) object corresponding to the implanted device is generated by one or more processors, at least in part, based on the collected sensor data; as well as A visual representation of the condition of the operating environment of the implanted device in the user's body is generated by one or more processors, at least in part, based on collected sensor data including the condition of the implanted device and the operating environment of the implanted device in the user's body.

3. The method according to claim 1, further comprising: One or more objects within the sensor feed of the AR device are identified by one or more processors; One or more processors determine, at least in part, whether an activity associated with the one or more objects adversely affects the user, based on the one or more conditions of the implanted device and the operating environment of the implanted device. as well as One or more processors generate a coverage map corresponding to each of the one or more objects, wherein the coverage map indicates a recommendation for approval of one or more activities associated with the one or more objects.

4. The method according to claim 1, further comprising: One or more processors determine whether a second device affects the performance of the implantable device, wherein the second device is selected from the group consisting of a second implantable device and external devices.

5. The method of claim 1, wherein the implanted device for identifying the user associated with the health-related problem further comprises: One or more processors make one or more topics of the voice command relate to one or more conditions of the user’s health-related problems, which are determined at least in part based on the user’s historical medical data and the user’s voice command; as well as One or more implanted devices are identified by one or more processors and have performance functions related to the one or more topics of the voice command.

6. The method according to claim 1, further comprising generating the audiovisual response corresponding to the user's health-related problem: One or more processors determine the state of the implanted device based on sensor data collected from the implanted device in relation to the user's voice commands; One or more processors determine the condition of the user's body at the location of the implanted device; as well as An audible response is generated by one or more processors, including information corresponding to the user's voice command, wherein the information is based at least in part on the state of the implanted device and the condition of the user's body at the location of the implanted device.

7. The method according to claim 6, further comprising: One or more processors transmit a three-dimensional augmented reality object corresponding to the implanted device and the audible response to the user via one or more computing devices, wherein at least one of the one or more computing devices includes a device with augmented reality capabilities.

8. A computer program product for audiovisual interaction with an implanted device, the computer program product comprising: One or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions comprising: Program instructions for determining the user's health-related problems based at least in part on the user's historical medical data and the user's voice commands; Program instructions for identifying the user's implantable device in relation to the user's health-related problems, which are determined based at least in part on the user's historical medical data and the user's voice commands; Program instructions for collecting sensor data from the implanted device, wherein the sensor data includes one or more conditions of the implanted device and the operating environment of the implanted device in the user's body; and Program instructions for generating an audiovisual response corresponding to the user's health-related problem, based at least in part on sensor data including the one or more conditions of the implanted device and the operating environment of the implanted device in the user's body.

9. The computer program product of claim 8, further comprising program instructions stored on the one or more computer-readable storage media, for: At least in part, based on the collected sensor data, a three-dimensional augmented reality (AR) object corresponding to the implanted device is generated; and A visual representation of the condition of the operating environment of the implanted device in the user's body is generated, at least in part, based on collected sensor data including the condition of the implanted device and the operating environment of the implanted device in the user's body.

10. The computer program product of claim 8, further comprising program instructions stored on the one or more computer-readable storage media, for: Identify one or more objects within the sensor feed of the AR device; The determination of whether activities associated with the one or more objects adversely affect the user is based at least in part on the one or more conditions of the implanted device and the operating environment of the implanted device. as well as Generate a coverage map corresponding to each of the one or more objects, wherein the coverage map indicates a recommendation for approval of one or more activities associated with the one or more objects.

11. The computer program product of claim 8, further comprising program instructions stored on the one or more computer-readable storage media, for: Determine whether the second device affects the performance of the implantable device, wherein the second device is selected from the group consisting of a second implantable device and an external device.

12. The computer program product of claim 8, wherein the program instructions for identifying the user in relation to the health-related problem in the implanted device further include program instructions for: The voice command relates one or more topics to one or more conditions of the user's health-related problems, determined at least in part based on the user's historical medical data and the user's voice command; and Identify one or more implanted devices that have performance functions related to the one or more topics of the voice command.

13. The computer program product of claim 8, wherein the program instructions for generating the audiovisual response corresponding to the user's health-related problem further include program instructions for: The status of the implanted device is determined based on the sensor data collected from the implanted device in relation to the user's voice command; Determine the condition of the user's body at the location of the implanted device; as well as Generate an audible response including information corresponding to the user's voice command, wherein the information is based at least in part on the state of the implanted device and the condition of the user's body at the location of the implanted device.

14. The computer program product of claim 13, further comprising program instructions stored on the one or more computer-readable storage media, for: The three-dimensional augmented reality object corresponding to the implanted device and the audible response are transmitted to the user via one or more computing devices, wherein at least one of the one or more computing devices includes a device with augmented reality capabilities.

15. A computer system for audiovisual interaction with an implanted device, the computer system comprising: One or more computer processors; One or more computer-readable storage media; as well as Program instructions stored on the computer-readable storage medium for execution by at least one of the one or more processors, the program instructions comprising: Program instructions for determining the user's health-related problems based at least in part on the user's historical medical data and the user's voice commands; Program instructions for identifying the user's implantable device in relation to the user's health-related problems, which are determined based at least in part on the user's historical medical data and the user's voice commands; Program instructions for collecting sensor data from the implanted device, wherein the sensor data includes one or more conditions of the implanted device and the operating environment of the implanted device in the user's body; and Program instructions for generating an audiovisual response corresponding to the user's health-related problem, based at least in part on sensor data including the one or more conditions of the implanted device and the operating environment of the implanted device in the user's body.

16. The computer system of claim 15, further comprising program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more processors, for: At least in part, based on the collected sensor data, a three-dimensional augmented reality (AR) object corresponding to the implanted device is generated; and A visual representation of the condition of the operating environment of the implanted device in the user's body is generated, at least in part, based on collected sensor data including the condition of the implanted device and the operating environment of the implanted device in the user's body.

17. The computer system of claim 15, further comprising program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more processors, for: Identify one or more objects within the sensor feed of the AR device; The determination of whether activities associated with the one or more objects adversely affect the user is based at least in part on the one or more conditions of the implanted device and the operating environment of the implanted device. as well as Generate a coverage map corresponding to each of the one or more objects, wherein the coverage map indicates a recommendation for approval of one or more activities associated with the one or more objects.

18. The computer system of claim 15, further comprising program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more processors, for: Determine whether the second device affects the performance of the implantable device, wherein the second device is selected from the group consisting of a second implantable device and an external device.

19. The computer system of claim 15, wherein the implanted device for identifying the user associated with the health-related problem further includes program instructions for: The voice command relates one or more topics to one or more conditions of the user's health-related problems, determined at least in part based on the user's historical medical data and the user's voice command; and Identify one or more implanted devices that have performance functions related to the one or more topics of the voice command.

20. The computer system of claim 15, wherein generating the audiovisual response corresponding to the user's health-related question further includes program instructions for: The status of the implanted device is determined based on the collected sensor data of the implanted device related to the user's voice command; Determine the condition of the user's body at the location of the implanted device; as well as Generate an audible response including information corresponding to the user's voice command, wherein the information is based at least in part on the state of the implanted device and the condition of the user's body at the location of the implanted device.

Images