Phone-Based Telehealth Devices

Abstract

The telehealth device includes a phone having a microphone, an auscultation component for acquiring sound, and a solid medium for acoustically coupling the auscultation component to the microphone. The auscultation component is part of a stethoscope, and the solid medium is the air tube of the stethoscope. The telehealth device also includes an otoscope operable to be positioned in front of a camera on the phone. A clip holds the stethoscope and the otoscope and is secured to the phone. A software module installed on the phone enables the telehealth device to engage a user in a two-way audio and / or video consultation with a physician at a remote device in real time.

Description

[Technical field]

[0001] Priority claim This patent application is related to and claims priority to the following three co-pending U.S. patent applications:

[0002] A) Title of the invention: "Synchronous Tele-Health Diagnostic System Integrated With Smart Devices", Application No.: 63072961, Application Date: 01-SEP-2020, B) Title of the invention: "Synchronous Tele-Health Diagnostic System Integrated With Smart Devices", Application No.: 63118751, Filing Date: 27-NOV-2020, and C) "TELEPHONE BASED TELE-HEALTH APPARATUS", Application No. 17 / 248,027, Filing Date: 06-JAN-2021; These are incorporated herein in their entirety to the extent they do not contradict the descriptions herein. [Background technology]

[0003] background Technical Field FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate generally to healthcare devices, and more specifically to phone-based telehealth devices.

[0004] Related Technology Telehealth devices allow doctors and other healthcare professionals to provide services remotely to those who need medical assistance. A phone refers to a device that allows two or more people to make voice and / or video calls. While phones were wire-based in the early evolution of technology (called Plain Old Telephone Systems, POTS), phones are now available in the form of mobile phones / smartphones that communicate wirelessly over communication standards such as 5G / 4G / 3G, WiFi, etc. Summary of the Invention [Problem to be solved by the invention]

[0005] With the proliferation of telephones, there is a continuing need to provide telehealth devices that leverage the capabilities of telephones.

[0006] BRIEF DESCRIPTION OF THE DRAWINGS Exemplary embodiments of the present disclosure are described with reference to the accompanying drawings, which are briefly described below. [Brief description of the drawings]

[0007] [Figure 1] 1A-1C are different views of a telephone-based telehealth device in one embodiment of the present disclosure. [Diagram 2] 1A-1C are different views of a telephone-based telehealth device in one embodiment of the present disclosure. [Diagram 3] 1A-1C are different views of a telephone-based telehealth device in one embodiment of the present disclosure. [Figure 4] 1A-1C are different views of a telephone-based telehealth device in one embodiment of the present disclosure. [Figure 5A] 1A-1C are different views of a telephone-based telehealth device in one embodiment of the present disclosure. [Figure 5B] 1A-1C are different views of a telephone-based telehealth device in one embodiment of the present disclosure. [Figure 6A] FIG. 13 is an exploded view showing the otoscope portion of a telephone-based telehealth device in an alternative embodiment of the present disclosure. [Figure 6B] 13A-13C show different views of an otoscope in an assembled form in an alternative embodiment. [Figure 6C] 13A-13C show different views of an otoscope in an assembled form in an alternative embodiment. [Figure 7] FIG. 13 illustrates a fiber holder ring used in an otoscope of a telephone-based telehealth device in an alternative embodiment of the present disclosure. [Figure 8]FIG. 1 is a block diagram of some application modules running on a phone-based telehealth device in one embodiment of the disclosure. [Figure 9A] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9B] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9C] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9D] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9E] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9F] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9G] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9H] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9I] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9J] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9K] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 9L] 1A-1C are diagrams of exemplary screens displayed on a phone-based telehealth device that enable user interaction with the functionality provided by the device. [Figure 10] FIG. 1 is a block diagram of a telephone device implemented in accordance with certain aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] In the drawings, like reference numbers generally indicate identical, functionally similar, and / or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference number.

[0009] Detailed Description 1. Overview According to one aspect of the disclosure, a stethoscope and an otoscope are attached to a phone. The phone may be, for example, a readily available off-the-shelf device such as a mobile phone. The stethoscope is connected to the microphone port of the phone and the otoscope is attached to the phone to couple light to a camera in the phone. The stethoscope may be a passive mechanical and acoustic attachment (to the phone) and the otoscope may be a passive mechanical and optical attachment (to the phone). The term "passive" with respect to a component means that there are no components in the component that require electrical power to operate in accordance with the features described herein.

[0010] Together with a corresponding application module executable on the phone, the integrated telehealth device (phone + stethoscope + otoscope + application module) enables two-way audio and / or video consultation between the user and a remote medical specialist or physician. Synchronized real-time diagnostic data obtained by the device via the stethoscope and otoscope (operated by the user as instructed by the remote physician) is made available to the physician, who can then prescribe the necessary treatments and medications. Thus, the telehealth device can provide a telehealth solution that can be equivalent to a primary care physician visit.

[0011] Some aspects of the present disclosure are described below with reference to illustrative examples. However, those skilled in the art will recognize that the present disclosure can be implemented without one or more of the specific details, or with other methods, components, materials, etc. In other examples, well-known structures, materials, or operations are not shown in detail to avoid obscuring the features of the present disclosure. Furthermore, the described features / aspects can be implemented in various combinations, but only some combinations are described herein for brevity.

[0012] 2. Phone-Based Telehealth Devices 1-4, 5A and 5B are different views of a phone-based telehealth device (device 100) in one embodiment of the present disclosure. In this embodiment, a smartphone (110) is used as the phone. As is well known in the related art, a smartphone is a mobile device that combines cellular and computing capabilities in one unit. Smartphones typically include a mobile operating system that allows applications to be loaded onto the smartphone, thereby enabling features such as web browsing, audio and video players, cameras, etc., in addition to traditional phone features such as voice calling. However, in other embodiments, other devices (that enable voice calling), such as tablet PCs (personal computers), iPads, internet-connected Chromebooks, computers, IoT (Internet of Things)-enabled devices, as well as wired phones, IP (Internet Protocol) phones, etc., can be used instead. FIG. 1 is shown including a smartphone 110, a clip 120, a stethoscope 130, an otoscope XY adjuster 140, and an otoscope 150.

[0013] Smartphone 110 represents an off-the-shelf mobile phone with an operating system, supports the addition (installation) of application modules, and supports circuit-switched and / or packet-switched voice and / or video calling. Smartphone 110 comprises one or more microphones, and one or more cameras. A microphone port 180 for the microphone is shown in FIG. 5A. Thus, the audio signal received at microphone port 180 is converted into an electronic signal suitable for transmission to the other end, where another phone, or a device such as a computer, tablet, etc. reproduces the audio signal from the electronic signal. A camera port 160 for a camera on the rear side of smartphone 110 is shown in FIG. 1. Camera port 160 is used with an otoscope, as described below. Phone 110 also has a front camera that is used during video calling, thus allowing the remote physician to "see" the patient / user.

[0014] Clip 120 is a mechanical assembly or harness designed to grip the outer edge and back of smartphone 110 using, for example, spring action or tightening screws. Clip 120 is shown more clearly in Figures 1 and 3. Stethoscope 130 and otoscope 150 are attached to clip 120 (as described below) and thereby secured to smartphone 110. Clip 120 is sized so as not to interfere with device controls on smartphone 110. Clip 120 may also be designed to meet the device (smartphone 110) manufacturer's mounting standards, if any.

[0015] The stethoscope 130 is composed of an auscultation part (or chest piece) 130A, an air tube 130B and a coupling part 130C. The auscultation part 130A is made of a flexible diaphragm, an acoustic chamber and a ring or nut for elastically fastening the flexible / elastic diaphragm to the acoustic chamber. FIG. 4 shows the acoustic chamber 130A-1 more clearly. FIG. 5 shows the flexible diaphragm 130A-2 and the ring or nut 130A-3 more clearly. The acoustic chamber 130A-1 may be made of, for example, metal or hard non-porous plastic. The auscultation part 130 is used to obtain sounds, such as heart and lung sounds, or sounds generally inside the patient's body. The auscultation part 130 may need to be placed close to the sound source.

[0016] The air tube 130B transmits the sound waves from the auscultation component 130A to the microphone port / opening of the smartphone 100 and may be, for example, a hollow rubber tube (generally a solid medium). One end of the air tube 130B is connected to the auscultation component 130A and the other end is connected to the coupling component 130C. The coupling component 130C couples the auscultation sound waves to the microphone outlet / port of the smartphone 110 while fixing the air tube by a compression fit. One or more suction cups or soft rubber couplers are used for cushioning and optimal air coupling to the microphone hole / holes. This coupling allows the user voice to be captured without significant attenuation and the user can continue normal audio and / or video / visual communication (through the microphone) without interruption or significant degradation.

[0017] The auscultation component 130 is secured to the clip 120 by double-sided tape or adhesive or hooks (when not in use). Alternatively, the auscultation component 130 can be secured to the clip 120 using screws, hooks, temporary containment mechanisms, or can be permanently molded as part of the clip 120, in which case the chest piece cannot be removed, but offers the advantage of compactness. Sounds received through the stethoscope 130 and the microphone port of the smartphone 110 are processed by an application module in the smartphone 110 for analysis by a physician (doctor) and / or transmitted (e.g., in real time) to a remote device (e.g., another phone, etc.). The application module can also simultaneously play the sounds on a local speaker / earphone of the smartphone 110 or record them locally in the smartphone 110. Note that since the microphone of the phone 110 is used to receive cardiac auscultation, the user must use earphones without a microphone (or with the microphone disabled). Alternatively, the telehealth application can be designed to have the software controlling the (earphone + external microphone) headset use the microphone port 180 on the phone rather than the external microphone provided on the headset.

[0018] Although the air tube 130B in the figures is shown as being rather short, the length of the air tube 130B can be much longer in practice. In one embodiment, the length of the air tube 130B is up to 300 centimeters with no minimum length restriction. Such a length allows the user to simultaneously see the doctor on the display of the phone 110, hear the doctor's voice, converse with the doctor, and allow the doctor to see the user via the front camera of the phone 110 while simultaneously operating the stethoscope 130 to capture auscultatory sounds of the heart and lungs. Such a facility allows for easy use by the user, especially when the user is an elderly person, a differently-abled person, or a sick patient. This is because the user does not need to operate two or more devices at the same time, resulting in asynchronous and confusing data streams for the doctor and patient at the same time, thereby reducing the quality of the diagnosis. Also, the user's anxiety associated with losing contact with the remote doctor is absent in this embodiment. However, in another embodiment, the air tube 130B is omitted and the chestpiece 130A is directly coupled to the coupling piece 130C.

[0019] The otoscope 150 is composed of a speculum 150A and a lens holder 150B. The speculum 150A and the lens holder 150 are attached to the otoscope XY adjuster 140 and then pivoted via a pivot mechanism 121 on the clip 120. The lens holder 150B houses a magnifying lens, the focal length of which is adjustable by a screw to allow light from a viewing source to be focused on the camera port 160. The otoscope 150 can be pivoted via the pivot mechanism 121 using a radial coordinate that is placed in front of the camera port 160. An alternative embodiment of the pivot mechanism 121 uses a Cartesian XY translation frame. FIG. 3 shows the otoscope 150 placed in front of the camera port. The otoscope 150 allows the user / patient of the smartphone 110 to obtain video or still images of the user / patient, such as the ear, nose and throat, by placing the speculum 150A on the part to be viewed. Video and still images (i.e., images) obtained using otoscope 150 and camera port 160 are processed and / or transmitted (e.g., in real time) by an application module in smartphone 110 to a remote device (e.g., another phone, computer, or smart communication device, etc.) for analysis by a physician. The application module can also simultaneously render the video and / or still images on a display screen of smartphone 110 or record them locally within smartphone 110.

[0020] In an alternative embodiment of the present disclosure, the otoscope function is expanded to also use the smartphone 110's flash LED (light emitting diode) in combination with a light guide to illuminate an object to be viewed and analyzed. FIG. 6A illustrates an exploded view of an otoscope in an alternative embodiment. In use, the otoscope 600 shown therein is positioned such that light collected by the otoscope hits a camera port of the smartphone 110, such as camera port 160 (FIG. 1). Similar to the otoscope 130, the otoscope 600 may also be mounted to pivot on a clip, such as clip 120, as shown in FIGS. 1-4, 5A and 5B. Alternatively, the otoscope 600 may be attached to the smartphone 110 using a double-sided adhesive tape (removable) or an elastic band mechanism.

[0021] Referring to FIG. 6A, the otoscope 600 is shown to be made of the following components: speculum 610, fiber optic holder ring 620, lens 630, tunnel 640, and cylinder 660. The speculum 610 functions similarly to the speculum 150A of FIG. 1 to collect light from an object being viewed (ear, nose, etc.). Additionally, the speculum 160 has a fiber guide path thereon through which an optical fiber can transmit light from a light source, such as the flash LED of the smartphone 110, to the object being viewed. An exemplary guide path 615 within the speculum is shown in FIG. 6A.

[0022] The fiber holder ring 620 includes holes through which the individual optical fibers pass, and is therefore fixed in place when the components of FIG. 6A are assembled. The holes are aligned with the guide paths in the speculum 610. The number of fibers, the diameter of the fibers, and therefore the holes in the ring 620 may be determined based on the size of the flash LED. The fiber holder ring 620 is shown in more detail separately in FIG. 7, where only one hole 711 is shown. An entire fiber bundle made of individual fibers (such as 710) is collected (bundled together as indicated by marker 720) and optically coupled to the flash LED by compression, whereby the geometric plane containing the faces of the bundled fibers (e.g., 720 in FIG. 7) coincides with the geometric plane containing the LED flash exterior surface without voids, and the center of the fiber bundle approximately coincides with the center of the flash to allow optimal coupling of light emitted from the flash into the bundled fibers.

[0023] Cylinder 660 is attached to ring 620 at one end and the other end is attached to an XY adjuster (not shown), such as adjuster 140 of Figure 1. Alternatively, an XY adjuster is not used and cylinder 660 is attached directly to the camera port using a suitable mechanism (e.g., double-sided sticky tape, etc.) so that the camera port is directly beneath the bottom (690) of cylinder 660. Lens 630 is used for magnification and is mounted within and concentric with cylinder 660.

[0024] Tunnel 640 receives the fiber bundle from ring 620 and routes the bundle to the camera's flash LED (indicated by marker 720 in FIG. 7). Tunnel 640 may be designed using flexible optical fiber / fiber or light pipe. The smartphone 110 camera flash lens (not shown) is at the bottom of tunnel 640. Slot 650 is an adjustment mechanism that adjusts the center of the light carrying fiber bundle to coincide with the center of the flash while providing one-to-one optical coupling between the flash and the fiber. This mechanism allows for adjustments for differences in flash location on different smartphones. The entire otoscope assembly can be pivoted about a pivot point (similar to pivot point 121 in FIG. 1) or can be mounted on a Cartesian XY stage to allow the user to approximately align the center of the rear camera with the center of the lens tower (i.e., cylinder 660). Tabs 640A and 640B (FIG. 6C) can be moved in and out of the assembly to allow for center alignment of the flash and the fiber. The light carrying fiber bundle (eg, 720 shown in FIG. 7) is tied and routed through a hole inside tab 640B to allow alignment of the fiber bundle with the plane of the flash lens.

[0025] 6B and 6C show two different views of the otoscope 600 in an assembled form. Note that the otoscope (150 or 600) includes an optical attachment filter (attached to the otoscope / optical attachment) to convert IR to visible wavelengths. Thus, in FIG. 1, the filter is housed in lens holder 150B along with the lens. Similarly, in FIG. 6A, the filter is housed in cylinder 660 proximal to lens 630. The reason for adding the filter is that IR (infrared light) may not be detectable by smartphone cameras such as those of phone 110, and IR from the human body is rich in information. The optical filter converts IR to a useful visible light spectrum. Ambient visible light is blocked using a black soft rubber blocker bonded to the bottom surface 690 to prevent ambient light from entering the otoscope and to avoid scratching the smartphone camera lens, body, and flash.

[0026] It will be appreciated that the device 100 being a single device (single part) may be conducive to easy use by the patient. It is noted here that most of the existing devices or systems available for point-of-care diagnosis seem to have the problem that they require additional hardware to be implemented in the device and that the redundant communication infrastructure and / or asynchronous streams of two devices make the diagnosis confusing and inaccurate. Alternatively, the conventional systems seem to require a dedicated and redundant communication infrastructure to function and provide the point-of-care diagnosis. Such a requirement represents a usability issue, where a caregiver or patient needs at least two different devices to relay the point-of-care diagnostic information to a doctor or healthcare provider, albeit asynchronously. If the patient is uncooperative or restrained, using two devices is problematic and causes diagnostic inaccuracies. For example, if a parent or caregiver of an elderly, heterogeneous healthy or sick patient makes a telehealth request to a doctor, from a usability perspective, they have to operate two devices simultaneously, causing asynchronous and confusing data streams for the doctor and the patient, thereby reducing the quality of the diagnosis. In the absence of available point-of-care diagnostic equipment, physicians must make diagnoses and prescribe medications based solely on video streams and / or patient interviews. These shortcomings make current devices or systems highly ineffective in providing quality healthcare to patients.

[0027] Meanwhile, the apparatus 100 allows the user to see and talk to the physician at the remote device even while handling the stethoscope 130. And, when using an otoscope, as opposed to sending audio through headphones that the user may find difficult to use simultaneously with the otoscope, the phone 110 maintains audio and video contact with the physician (video contact via the front camera) intact by automatically activating the phone's speaker when the user is using the otoscope (e.g., for an ear examination).

[0028] The smartphone 110 includes (or can be loaded with) application modules that integrate and synchronously (in real time) transmit diagnostic information obtained via the stethoscope 130 and the otoscope 150 / 600 to a remote device. The smartphone 110 also includes application modules for determining the user's pulse rate, temperature, blood pressure and pulse oxygen content from images / video obtained from finger pressing on the camera port 160. In combination with corresponding software on the remote device / terminal, the apparatus 100 provides two-way voice and / or video consultation capabilities for remote diagnosis, storage, drug prescription (or treatment advice) and billing. The operation of some application modules of the smartphone 110 will now be described.

[0029] 3. Application Module FIG. 8 is a block diagram of some application blocks / modules that may execute on the smartphone 110. The specific blocks are shown for illustrative purposes only, and there may be many more application blocks that may execute on the smartphone 110. Also, the operations of some blocks may be combined into one block. The term "software module" as used herein may refer to a single block or a combination of multiple blocks. Furthermore, a mobile operating system (OS) may be present on the phone 110.

[0030] It should be noted here that one or more of the blocks in FIG. 8 may depend on other application modules (not shown, but included in the telehealth application - referring to the combination of all modules as a whole) for decision making, forwarding data to the next module, etc. Thus, in the following description, the telehealth application is assumed to include an application module control block (AMCB) that generates control signals for the blocks in FIG. 8 (if necessary). One or more of such control signals may be generated based on user interaction or input to the telehealth application. As an example, when the user presses button 941 on the screen shown in FIG. 9H, the telehealth application "knows" that the user is attempting to obtain cardiac auscultation sounds. Thus, the AMCB sends a signal to the microphone interface block 805 to forward the audio data received on path 801 to the cardiac auscultation sounds block 810, but not to the pulmonary auscultation sounds block 815.

[0031] In one embodiment, one or more of the blocks in FIG. 8 may be designed according to WebRTC (Web Real-Time Communications) technology to enable real-time or synchronous exchange of various diagnostic and other information between the phone 110 and a remote device. As is well known in the related art, WebRTC refers to a free, open-source project that provides real-time communications (RTC) via a simple application programming interface (API) to web browsers and mobile applications. Thus, for example, the microphone interface block 805 and the camera interface block 830 may be implemented using the getUserMedia API (provided by WebRTC), and the transmit block 890 may be implemented using the RTCPeerConnection API (also provided by WebRTC).

[0032] 8, the microphone interface block 805, cardiac auscultation sounds block 810, pulmonary auscultation sounds block 815, and audio playback block 820 operate on digitized audio data. The microphone interface block 805 receives digitized audio data from the microphone port of the phone 110 and also through the audio hardware (including an analog-to-digital converter (ADC)) of the phone 110 when the stethoscope 130 is used to obtain audio information. The microphone interface block 805 may be implemented using an application programmer interface (API) function call provided by the operating system in the phone 110. The microphone interface block 805 forwards the received audio data to the corresponding one of blocks 810 and 820 depending on whether the audio data represents cardiac auscultation sounds or pulmonary auscultation sounds. The microphone interface block may make such a determination based on input from the application module control block (AMCB), as described above. 1, auscultation sounds of the heart or lungs are obtained by a user placing auscultation component 130A on either the chest or lung area of ​​the user. The user may be guided via phone 110 by a physician at the remote end to properly place auscultation component 130A.

[0033] The cardiac auscultation sounds block 810 applies digital filtering to remove noise from the received heart sound data to obtain filtered heart sound data. Based on input from the application module control blocks above, the cardiac auscultation sounds block 810 may forward the filtered data to one or more of the audio playback block 820, the storage block 880 and the transmission block 890. The input from the application module control block may be based on whether real-time local play, remote play, local storage, remote or cloud storage, archive and retroactive play need to be performed, which may also come from user input to the telehealth application. In one embodiment of the present disclosure, the heart sound auscultation data is played simultaneously (synchronously) locally at the phone 110 and the remote device (after simultaneous or delayed transmission required by the physician at the remote device), allowing real-time and / or asynchronous diagnosis by the physician at the remote device.

[0034] The lung auscultation sounds block 815 applies digital filtering to remove noise from the received lung sound data to obtain filtered lung sound data. Based on user input or input from the application module control block, the lung auscultation sounds block 815 may forward the filtered data to one or more of the audio playback block 820, the storage block 880, and the transmission block 890. The input from the application module control block may be based on whether real-time local play, remote play, and local storage need to be performed. In one embodiment of the present disclosure, the lung sound auscultation data is also played back simultaneously (synchronously) locally at the phone 110 and the remote device (after simultaneous or delayed transmission required by the physician at the remote device), allowing real-time and / or asynchronous diagnosis by the physician at the remote device.

[0035] Note that heart sounds have a frequency range of approximately 50-60 Hz. The digital filters used within the cardiac auscultation sounds block 810 may use any magnitude, low pass and high boost filtering optimized to capture the frequencies of the heart sounds. High boost filtering refers to enhancing (boosting) the high frequency components of the sound without removing the low frequency components. The digital filters used in the pulmonary auscultation sounds block 815 are designed to optimize and boost frequencies from approximately 300 Hz to approximately 550 Hz (respiratory frequencies). The filters in blocks 810 and 815 can be implemented as fixed FIR (finite impulse response) and IIR (infinite impulse response), or adaptive filters.

[0036] The audio playback block 820 routes the digital audio data on path 821 to a hardware audio subsystem in the phone 110, which may include a digital-to-analog converter (DAC), a power amplifier, and a speaker. The audio playback block 820 may be implemented using application programmer interface (API) function calls provided by the operating system in the phone 110.

[0037] 8, formats the data according to a corresponding storage format, and transfers the formatted data to storage in the phone 110 over path 881. The storage block 880 may be implemented using application programmer interface (API) function calls provided by an operating system in the phone 110.

[0038] The transmit block 890 receives data representing audio or video / images from the corresponding block in Figure 8, formats the data according to a corresponding transmission format (including packetization), and forwards the formatted data to a transmitter in the phone 110 over path 891. The transmit block 890 may be implemented using application programmer interface (API) function calls provided by an operating system in the phone 110.

[0039] The camera interface block 830, otoscopy processing block 835, BP / heart rate / temperature / oximeter block 840, optical character recognition (OCR) block 845, and display block 860 operate on digitized video or image (still) data.

[0040] The display block 860 forwards the video / images received from one or more of blocks 845, 835, and 840 on respective paths 856, 836, and 846 over path 861 to the display hardware of the phone 110. The display hardware renders the video / images on a display device (e.g., the screen of the phone 110). The display block 890 may be implemented using application programmer interface (API) function calls provided by the operating system in the phone 110.

[0041] Camera interface block 830 receives digitized video or image data from camera port 160 and through the video hardware (including another (ADC)) of phone 110 on path 831. Video / images may be acquired from the pressing of a user's finger on the camera port using otoscope 150, or by focusing the camera of phone 110 on the viewing area of ​​one or more external diagnostic devices, such as a digital or analog thermometer, a digital or analog blood pressure monitor, a digital or analog oximeter, etc.

[0042] When using otoscope 150, speculum 150A of otoscope 150 is placed (e.g., by a user) on the body part to be viewed (e.g., ear, nose or throat) and XY adjuster 140 is adjusted so that speculum 150A and lens (in lens holder 150B) are aligned with camera port 160. When capturing a user's finger press, the user adjusts XY adjuster 140 so that speculum 150 and lens are away from the camera port. When viewing an external diagnostic device is desired, the user focuses the camera port of camera 110 on the viewing area of ​​such external diagnostic device.

[0043] The camera interface block 830 may be implemented using API function calls provided by the operating system in the phone 110. Based on input from the application module control block (described above), the camera interface block 830 forwards the received video / image data to a corresponding one of blocks 835, 840, and 845 depending on whether the video data was captured using the otoscope 150, a user's finger pressure, or an external diagnostic device.

[0044] The otoscopy processing block 835 filters the received video / image data acquired using the otoscope 150 and forwards the filtered data to one or more of the display block 860, the storage block 880, and the transmission block 890 depending on the corresponding input received from the application module control block (described above). The input from the application module control block may be based on whether real-time local play, remote play, and local storage needs to be performed, which in turn may be based on user input to the telehealth application. In one embodiment of the present disclosure, the filtered data from the otoscopy processing block 835 is simultaneously (synchronously) rendered locally on the display block 860 of the phone 110 as well as in the display of the remote device (after simultaneous transmission to the remote device), enabling real-time diagnosis by the physician at the remote device.

[0045] The OCR block 845 operates to receive an image (e.g., captured by a rear camera) of the display area of ​​the external diagnostic device (as described above) and extract text, numbers, symbols, etc. within the image. The OCR block 845 forwards the extracted data to one or more of the display block 860, the storage block 880, and the transmit block 890, depending on corresponding input received from the application module control block (described above). For example, the external device may be a digital thermometer. A user may use the digital thermometer to obtain their temperature, and the OCR block 845 may receive an image of the reading on the display area of ​​the digital thermometer and extract the temperature value therefrom.

[0046] The BP / Heart Rate / Temperature / Oximeter block 840 operates to determine the user's blood pressure (BP), heart rate, temperature and blood oxygen saturation level from a video / image of the user's finger pressing on the camera port 160.

[0047] The signal processing algorithm of block 480 typically selects a region of interest around the approximate center of the field of view (FOV) (in this case, approximately the center of the pressed finger). The size of this FOV region directly impacts the computational complexity, and there is an optimum for the size that provides diagnostic quality results, with diminishing returns observed for larger FOV sizes. Block 840 extracts the average values ​​of the Red / Green / Blue (RGB) channels at a high frame rate as a function of time. We now describe how the following parameters are determined: BP, heart rate, body temperature, and blood oxygen saturation level.

[0048] As described above, block 840 operates on a series of RGB images generated by the user's finger press on camera port 160. Block 840 creates a fixed volume or region of interest (ROI) within each image of the series of images. Each image of the series is a function of the light intensity I, as well as the contents of the ROI at a macroscopic level. Each image includes the red, green, and blue (RGB) values ​​of the camera's sensor (e.g., CMOS) output, and block 840 extracts the average values ​​of the red / green / blue (RGB) channels as a function of time at a high frame rate.

[0049] The dermal (skin) content of the resulting series of images (or more specifically the RGB values ​​of the images) as a function of time is constant for any given scenario, and the light level variations in the series of images are due to the effect of blood moving in and out of the ROI as the heart goes through systole and diastole. Thus, the light level variations in the series of images are also a function of (i.e., correlated to) the user's blood pressure. Thus, the ROI represents cardiac function and correlates to blood pressure as a function of time.

[0050] The light level value at the systolic peak is also a function of the oxygen content in the blood during the systolic peak. Block 840 corrects for oxygen content variations by first determining the peak of the beat (systole) corresponding to the heart rate based on the red intensity values ​​in each image. The systolic peak corresponds to the maximum red intensity among all images other than those representing the other systolic peaks. Block 840 then estimates the oxygen content component by subtracting from the image representing the systolic peak the average value of the red intensity in the image corresponding to the systolic peak and extracting the oxygen content from the flat-field corrected intensity. As is well known in the art, flat-field correction is a technique used to improve the quality of digital imaging by counteracting the effects of image artifacts caused by variations in pixel-to-pixel sensitivity of the detector (camera) and distortions in the optical path. The blood oxygen content component thus obtained (or alternatively obtained by averaging the blood oxygen content estimates over several iterations) indicates the blood oxygen saturation level.

[0051] Heart rate is determined by averaging the periods of the detected peaks (as above) and taking the inverse of the period. Body temperature correlates with increased heart rate and is an age-dependent factor. Block 840 uses the well-known age-heart rate-temperature relationship to obtain the patient's body temperature.

[0052] Continuing with the BP determination, once corrected for blood oxygen content, the only remaining dependent variable is BP. Block 840 processes multiple consecutive frames (images) (as an example, the camera can provide a frame rate of 30 frames / second or more) to obtain the blood volume (correlated with the red value in the image) in the vicinity (i.e., the red intensity in the image immediately before (for the sequence of images) and immediately after the image representing the systolic peak, and the red intensity in the image immediately before and immediately after the image representing the diastolic peak) and determines the rate of blood volume change as a function of time, i.e., dV / dt corresponding to the systolic and diastolic peaks.

[0053] A region of interest (ROI) is drawn approximately in the center of the image of the finger (i.e., in each of the series of images). This ROI or field of view (FOV) is any small area on the two-dimensional image. The blood flow in and out of this FOV as a volume change (dV) as a function of time (dt) (dV / dt) is determined by analyzing successive frames thus acquired with their corresponding timestamps (dt) and by plotting the corrected (as described above) red intensity as a function of time.

[0054] The volume change rates dV / dt corresponding to the systolic and diastolic peaks are then converted to respective values ​​of dP / dt, i.e., pressure change rates over time, via a linear transformation that assumes a first degree approximation for a very small portion of the pressed finger, and is further reduced by arbitrarily selecting a smaller subset of the image, called the field of view, as described above. The linear transformation constant K is a function of the sensor (camera sensor) response characteristics and can vary from device to device. This value, i.e., the constant K, is determined using a calibration process with known BP.

[0055] The BP determined as above may include error components due to variations in light intensity (lumens) and camera resolution (across different cameras) used to acquire the series of images of the finger press. Such error components are corrected using commonly known flat-field correction methods, in which the sensor (camera element) response is measured under uniform illumination to establish a baseline. This baseline accounts for system-induced variations that do not contribute to the signal of interest and is subtracted from the measured signal as the flat-field correction.

[0056] Parameters and signals such as cardiac and pulmonary auscultation sounds, images obtained by an otoscope, BP, body temperature, blood oxygen content, etc., as detailed above, may be referred to as diagnostic data.

[0057] Thus, various parameters and diagnostic data regarding the user's health are obtained and transmitted in real time to a physician at the remote terminal. The description continues with a simplified diagram of an exemplary user interface provided by an application module (other than that of FIG. 8) on the smartphone 110 to enable the user / patient to interact with a physician at the remote terminal, obtain diagnostic measurements, etc.

[0058] 4. User Interface Once a remote consultation is set up, the patient and doctor connect with each other by opening an application (the telehealth application mentioned above) on their respective smartphones. The patient, receiving verbal or non-verbal (text) or gestural (video) instructions from the doctor, clicks on menu items (examples below) in the application and diagnostic information is retrieved and relayed synchronously from the point-of-care to the remote doctor. Different diagnostic information is retrieved sequentially or in parallel based on the medical need that the healthcare provider or doctor deems appropriate. Once diagnostic retrieval is complete, a prescription is sent by phone 110 to the patient or to a pharmacy of their choice. At the end of the consultation, billing-related exchanges take place between the patient and healthcare provider.

[0059] 9A-9L are exemplary screens provided by a telehealth application with which a user interacts. In each of the screens shown, a "Cancel" button at the top left of the screen will cancel the current action and the user will be presented with the next appropriate screen.

[0060] Figure 9A is the screen presented to the user on the display of the phone 110 upon launching the telehealth application. The user can enter their email ID in box 901, enter a password in box 902, and then press button 903 to log in or register (if not already registered). Once logged in, the user is presented with the screen shown in Figure 9B.

[0061] In FIG. 9B, pressing button 907 allows the user to confirm an appointment with a doctor, and upon pressing button 907, the user is presented with the screen of FIG. 9E. Referring to FIG. 9E, the user can select one of the doctors, indicated by reference numbers 920, 921, and 922. Assuming the user selects Lisa Su (920), the user is presented with the screen of FIG. 9F, where the user can enter the date (925) and time (926) of the appointment. Pressing the Set button 927 produces the screen of FIG. 9G, which displays the doctor's name (931), date (932), and time (933) of the appointment. The user can return to the previous screen at any time by pressing "Cancel" (top right of full screen). Alternatively, as described below, the functionality to select and confirm an appointment with a doctor (as described above) is automatically presented to the user (not via pressing button 907) after clicking button 905 and completing the triage questionnaire.

[0062] In FIG. 9B, upon pressing button 905, the user is directed to the screen of FIG. 9C for triage questions. As shown in FIG. 9C, a questionnaire is presented with questions regarding specific health conditions (911, 912, 913). The user can select one or more of the health conditions, and upon pressing button 914, the user is presented with the screen of FIG. 9D. In FIG. 9D, health conditions are presented (915, 916, 917, and 918) and the user can select based on the health condition. Upon completing the triage questionnaire, the answers provided are transmitted to the remote terminal and the remote physician uses it for diagnosis. Additionally, the user is presented with a selection of available healthcare professionals for consultation, as described above with respect to pressing button 907 and the resulting screen.

[0063] In Figure 9B, button 906 is for initiating a consultation with a physician for whom an appointment has already been confirmed as described above. Alternatively, a consultation can be initiated if an appointment has been confirmed with a physician as described above and the physician is currently available for a consultation. In either case, the user is presented with the screen of Figure 9H. A consultation is enabled by the telehealth application conducting an audio and / or video call with the remote physician. Once the call is established, the user is presented with the screen of Figure 9H.

[0064] The screen of FIG. 9H allows a user, under the direction of a physician at the remote end, to generate diagnostic data such as cardiac / pulmonary auscultation sound data using stethoscope 130, otoscopic examination data using otoscope 150 / 600, diagnostic data regarding temperature, blood pressure, blood oxygen content and heart rate from finger pressing on the camera port, as well as diagnostic data captured by external devices and retrieved by OCR block 845 as described above, thereby facilitating a complete medical examination.

[0065] Specifically, pressing button 941 (with the auscultation component 130A of stethoscope (130) positioned over the user's cardiac area) initiates capture of cardiac auscultation sound data via stethoscope 130 and presents the user with the screen of FIG. 9I, which shows the cardiac auscultation data being filtered by "Filter I" implemented by cardiac auscultation sounds block 810 (FIG. 8).

[0066] Pressing button 942 (with the auscultation component 130A of stethoscope (130) placed over the user's lung area) initiates the capture of pulmonary auscultation sound data via stethoscope 130 and presents the user with the screen of Figure 9J, which shows that the pulmonary auscultation data has been filtered by "Filter II" performed by pulmonary auscultation sounds block 810 (Figure 8).

[0067] Pressing button 943 (with speculum 150A / 610 of otoscope (150) positioned at the user's ear / nose / throat and camera flash optionally pressed when using otoscope 600) initiates capture of video data via otoscope 150 with the user being presented with the screen of FIG. 9K. FIG. 9K shows that the video data has been filtered by "Filter III" performed by otoscopy processing block 835 (FIG. 8). The screen also displays parameter 1, the patient's temperature (951), as determined by block 840.

[0068] Pressing button 944 (with speculum 150A / 610 of otoscope (150) positioned at the user's ear / nose / throat and camera flash optionally pressed when using otoscope 600) initiates capture of video data via otoscope 150, with the user being presented with the screen of FIG. 9L. FIG. 9L shows that the video data has been filtered by "Filter IV" performed by otoscopy processing block 835 (FIG. 8). The screen also displays parameter 2 (blood pressure) (961), parameter 3 (pulse oximetry) (962), and parameter 4 (beats per minute) (963) determined by block 840.

[0069] It should be noted here that the telehealth application provided by the present disclosure can also generate various other screens for displaying various other options and information to the user, and the screens of Figures 9A-9L are merely representative. Additionally, a counterpart application is provided by the present disclosure to be installed on the remote device to complement the operation of the module (or telehealth application) in the phone 110 and enable two-way audio / video consultation as described herein. Such applications can be implemented in a manner known by those of skill in the art upon reading the disclosure herein.

[0070] Several other features of the telehealth application not described above are summarized below, some of which are also mentioned in other sections of this disclosure.

[0071] (A) Regulatory Compliance 1. HIPAA (Health Insurance Portability and Accountability Act) compliant patient and provider registration portal with two-factor authentication.

[0072] 2. HIPAA compliant patient and provider login web portal hosted on cloud or private cloud or private server machine.

[0073] 3. HIPAA compliant login to Android and iOS apps with the following features: 1. Login with ID / email and password / fingerprint / face ID 2. Online mode i. You will have access to both video calling and diagnostic features.

[0074] ii. Ability to use additional features during a two-way conference call; 3. Offline Mode i. Diagnostic functions only are accessible (no video calls) with on-device archiving and delayed relay to remote locations.

[0075] (B) Reservation and user management functions 1. Two modes for logging in 1. Patient Mode 1. In patient mode, when an appointment is made, a link is sent via email and added to the phone calendar, by clicking on which the two clients can connect via video call.

[0076] 2. Provider Mode 1. Review your booking calendar 2. A booking calendar with an active link that, when clicked, starts a WebRTC video call (C) Video calling function 1. Mute the microphone.

[0077] 2. Turn off the front camera and allow voice calls only. 3. Switch between front and rear cameras.

[0078] 4. Zoom in and out the camera. 5. Switch to Otoscope mode where the object under the otoscope is displayed zoomed in on the local and remote screens.

[0079] 6.Seamlessly switch between 3 different audio filters for real-time / synchronous transmission of auscultatory sounds via WebRTC or equivalent technology.

[0080] (D) Additional Features 1. If an earphone with a microphone is connected to the phone, the telehealth application generates an error message stating that additional features (stethoscope and otoscope) cannot be used.

[0081] 2. In offline mode, audio is not sent to the remote device via WebRTC but is made available via earphones (in-ear headphones, approved earphones).

[0082] 3. In offline mode, audio can be recorded and stored on the phone or in the cloud, or emailed only via the telehealth app.

[0083] 4. Saved audio, along with other user entered data, has metadata of username, date of birth, time, mode (clean channel, filter 1 or filter 2), and date recorded.

[0084] 5. The user can play back the recording. 6. Users can delete records.

[0085] 7. Users are warned before deleting records (or other user data) that they cannot be retrieved.

[0086] 8.There is an app button to switch between different filters. The filters are as follows:

[0087] 1. Unfiltered clean channel 2. Cardiac mode auscultation 3. Pulmonary mode auscultation 9. Patient finger pressure on the rear camera followed by user input (button press) on a corresponding screen (not shown).

[0088] Exemplary internal details of the phone 110 will now be described. 5.Telephone 10 is a block diagram showing implementation details of a mobile / smartphone 110 in one embodiment of the present disclosure. Although described as a mobile phone, the device 110 may be generally implemented as a digital processing device having wireless or wired communication capabilities, such as a tablet PC (personal computer), an iPad, an Internet-connected Chromebook, a computer, an IoT (Internet of Things)-enabled device, as well as a wired phone, an IP (Internet Protocol) phone, etc.

[0089] A mobile phone 110 is shown including a battery 1001, a power supply 1005, a microphone interface 1010, a camera interface 1015, a processing block 1020, a non-volatile memory 1030, a random access memory (RAM) 1040, an input block 1050, a display 1060, a transmit chain 1070, a receive chain 1080, a switch 1090, and an antenna 1095. The particular components / blocks of the mobile phone 1000 are shown for illustrative purposes only. However, the mobile phone 1000 may include more or fewer components / blocks.

[0090] A battery 1001, together with a power supply 1005, provides a regulated supply voltage that powers each of blocks 1010, 1015, 1020, 1030, 1040, 1050, 1060, 1070 and 1080. However, in Figure 10, for clarity, only the power connections to processing block 1020 are shown.

[0091] Microphone interface 1010 receives the audio signal on path 180 (microphone port), amplifies the signal, and generates digital data representing the signal using an ADC. Microphone interface 1010 forwards the digital audio data to processing block 1020 for further processing.

[0092] Camera interface 1015 receives light (visible / infrared, etc.) via camera port 160 and generates digital video and / or still images in a known manner. Thus, for example, camera interface 1015 may include RGB filters, an image sensor (e.g., CMOS), an ADC, formatting circuitry, etc. to generate video / images in the form of a set of RGB values. Camera interface 1015 forwards the RGB values ​​to processing block 1020.

[0093] The input / output block 1050 represents one or more input and output devices used to provide user input to the mobile phone 1000 and output data from the mobile phone 110 to the user. Thus, the input / output block 1050 may include a keypad as an input and a DAC, power amplifier and speaker / earphone as outputs. The display 1060 represents a display screen (e.g., a liquid crystal display) for displaying images / text generated by the processing block 1020.

[0094] Antenna 1095 operates to receive and transmit information-bearing radio signals from and to a wireless medium. Switch 1090 may be controlled by processing block 1020 to connect antenna 1095 to either receive chain 1080 via path 1098 or transmit chain 1070 via path 1079 (connections not shown) depending on whether mobile phone 1000 is receiving or transmitting radio signals.

[0095] The transmit chain 1070 receives data / conversation / voice / video transmitted from the processing block 1020 (in a general information signal including that generated by the transmit block 890 when executed by the processing block 1020), generates a radio frequency (RF) signal modulated by the information signal according to a corresponding standard such as GSM, CDMA, etc., and transmits the RF signal via the switch 1090 and the antenna 1095. The receive chain 1080 receives RF signals carrying information signals (including signals representing data from remote devices as described herein) via the switch 1090, the path 1098, and the antenna 1095, demodulates the RF signal, and provides the extracted information (data / conversation / voice / video) to the processing block 1020.

[0096] The non-volatile memory 1030 is a non-transitory machine-readable storage medium that stores instructions that, when executed by the processing block 1020, cause the mobile phone 1000 to provide certain features described herein. Thus, the non-volatile memory 1030 may store instructions representing application modules described herein, including those of FIG. 8. The non-volatile memory 1030 also stores data, such as that on path 881 (FIG. 8). The RAM 1030 is a volatile random access memory and can be used to store instructions and data.

[0097] The processing block 1020 (or processor in general) may include multiple processing units (processors) therein, each processing unit may be designed for a specific task. Alternatively, the processing block 1020 may include only a single general-purpose processing unit. The processing block 1020 may execute instructions stored in the non-volatile memory 1030 or RAM 1040 to enable the mobile phone 110 to operate to provide various features described herein. Specifically, the processing block 1020 executes instructions included in a telehealth application, including the application module of FIG. 8, modules for generating the screens of FIG. 9A-9L, application module control blocks, etc., to provide two-way audio and / or video consultations between a user and a remote medical professional or doctor in real time. The paths 821, 881, 891, 861, 801, and 831 of FIG. 8 are included in paths 1052, 1032, 1027, 1062, 1021, and 1025, respectively.

[0098] 6. Conclusion References throughout this specification to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the disclosure. Thus, appearances of "in one embodiment," "in an embodiment," and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0099] While the illustrations in Figures 1-4, 5A, 5B, and 10 show terminals / nodes having direct connections (i.e., "connected to") various other terminals, it should be understood that additional components (as appropriate to the particular environment) may also be present in the path, and thus the connections may be considered to be "electrically coupled" to the same connected terminal.

[0100] While various embodiments of the present disclosure have been described above, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. 1. A telehealth device for use in combination with a telephone having a microphone and a camera, said telehealth device for facilitating remote examination of a living organism, said telehealth device comprising: an auscultation component for collecting sounds emitted by the body of the living being and transmitting them to the microphone which generates an acoustic signal; an otoscope comprising an optical element configured to collect a therapeutic optical spectrum from the portion of the living organism and convert the optical spectrum into an optical spectrum from which the camera captures an image; The adjuster and a solid medium having a first end coupled to the auscultation component; a mechanical structure for gripping the telephone, the mechanical structure also attached to a second end of the solid medium, the second end attached proximate to the microphone to transmit the sound from the auscultation component to the microphone; a first end of the adjuster is also physically attached to the mechanical structure and a second end of the adjuster is attached to the otoscope, the adjuster being adjustable to position the otoscope in front of the camera; the adjuster is an X-Y adjuster, the otoscope includes a speculum attached to the X-Y adjuster, the X-Y adjuster being pivoted relative to the mechanical structure via a pivot mechanism to position the otoscope in front of the camera in one position and away from the camera in a second position; whereby the acoustic signal and the image are transmitted by the telephone to a remote person to facilitate remote inspection of the organism; A telehealth device, wherein the otoscope includes a first lens and a first optical filter, the first lens designed to magnify an object to be viewed and the first optical filter designed to convert infrared light to visible light.

2. The telehealth device of claim 1 , wherein the stethoscope component is part of a stethoscope and the solid medium is an air tube of the stethoscope.

3. The telehealth device of claim 1 , wherein the image comprises a still image or a video of the portion.

4. 3. The telehealth device of claim 2, wherein the solid medium has a sufficient length so that a user can simultaneously see and hear a physician at a remote device via the telephone and simultaneously operate the stethoscope.

5. 4. The telehealth device of claim 3, wherein the camera is on a back side of the phone and the otoscope further comprises a light guide for housing an optical fiber, the optical fiber being connected to a flash light emitting diode (LED) of the phone, thereby enabling capture of an image from an object illuminated by the flash.

6. 10. The telehealth device of claim 1, wherein the second end of the solid medium comprises a coupling piece, the coupling piece coupled to the mechanical structure by a compression fit mechanism, the coupling piece also comprising one or more suction cups or rubber couplers to allow a user's voice to be captured by the microphone without significant attenuation, thereby allowing the user to communicate audio using the microphone without significant degradation.

Images