Diagnostic mobile device

Abstract

A mobile device (120) includes a camera, a memory, and a processor (210). The memory stores instructions including a diagnostic program. The processor (210) executes the instructions. When executed by the processor (210), the instructions cause the mobile device (120) to: control the camera to take a screenshot of an external display of content to generate an image of the external display; process the image to isolate the physiological content from the external display; apply the diagnostic program to determine a diagnosis and whether the physiological content triggers an alarm, and generate and provide a recommendation via the mobile device (120) when the physiological content does not trigger an alarm; and generate a notification comprising the physiological content to an external party (190) over a communication network (201) when the physiological content triggers an alarm.

Description

DIAGNOSTIC MOBILE DEVICEBACKGROUND

[0001] The ability to diagnose at-risk patients promptly and accurately in emergencies is crucial, but also sometimes difficult such as in areas without easy access to clinicians. For example, sometimes there are limitations to access to the Internet, healthcare professionals, and hospitals in these areas.

[0002] Diagnostic electrocardiograms (ECGs or EKGs) and patient monitoring devices are important tools for healthcare professionals to provide high-quality patient care. ECGs are used to measure the electrical activity of the heart, while patient monitoring devices are used to track vital signs over time. Both of these tools can help healthcare professionals diagnose and manage a variety of medical conditions, including heart disease, arrhythmias, and respiratory problems.

[0003] The diagnostic ECG device is known as an electrocardiograph and records the electrical activity of the heart and is crucial for diagnosing various heart conditions. The electrocardiograph is non-invasive and quick, and serves as an initial evaluation tool for suspected heart disease, aiding in treatment monitoring and identification of cardiac abnormalities. The electrocardiograph presents ECG signals as graphical waveforms, offering valuable information about the rhythm and rate of the heart, and overall cardiac health.

[0004] Patient monitoring devices are medical devices that continuously monitor and track vital signs, including heart rate, blood pressure, and oxygen levels. This real-time data allows healthcare professionals to closely observe patient condition and respond promptly when needed. These devices are capable of measuring and recording a variety of vital signs, including heart rate, blood pressure, oxygen saturation (SpO2), carbon dioxide level (CO2), respiration rate, and temperature. They can also capture important waveforms like ECG, pulse oximetry (PPG or SpO2), capnography (CO2), invasive or non-invasive blood pressure (IBP or NIBP), and respiration. Recent advancements in the resolutions of electrocardiograph and patient monitoring device screens have enabled the clear display of waveforms and vital signs. These significant improvements have the potential to aid health professionals in promptly and accurately diagnosing heart conditions and assessing vital signs with greater precision. However, despite the advancements, challenges arise in achieving prompt and accurate diagnosis / interpretation byclinicians in limited / isolated environments such as insular, rural, and underdeveloped countries / areas with limited access to hospitals, as well as in-home settings and emergency situations. Moreover, healthcare providers may not always be able to save, access, share, communicate and interpret captured waveforms and vital signs.

[0005] Advancements in digital technology, such as portable devices, high-resolution screens, image processing capabilities, and enhanced computing power, have played a pivotal role in digitizing waveforms and vital signs from the screen photos on the diagnostic ECG and patient monitoring devices. This digitization enables remote access and analysis, offering a number of advantages, including prompt and accurate diagnosis, improved patient care, reduced healthcare costs, increased healthcare efficiency through telemedicine, and reduced physical storage volume resulting from large paper-based reports. With the ability to transmit digitized data to remote clinicians, this approach facilitates remote patient monitoring / diagnosis and consultation, thereby improving healthcare access and outcomes through telemedicine. Furthermore, the digitization of waveforms and vital signs from the screen photo enhances accuracy, efficiency, storage, remote monitoring / diagnosis, and patient care. Digitization empowers healthcare professionals to take prompt interventions and devise effective treatment strategies, particularly in limited areas and emergency situations. The ability to quickly access and interpret digitized data enhances healthcare delivery by ensuring timely and targeted care, even in resource-constrained environments.SUMMARY

[0006] According to an aspect of the present disclosure, a mobile device includes a camera, a memory and a processor. The memory stores instructions including a diagnostic program. The processor executes the instructions. When executed by the processor, the instructions cause the mobile device to: control the camera to take a screenshot of an external display of content, wherein the content comprises physiological information of a subject, to generate an image of the external display; process the image to isolate the physiological content from the external display; apply the diagnostic program to determine a diagnosis and whether the physiological content triggers an alarm, and generate and provide a recommendation via the mobile device when the physiological content does not trigger an alarm; and generate a notification comprising the physiological content to an external party over a communication network when thephysiological content triggers an alarm.

[0007] According to another aspect of the present disclosure, a tangible, non-transitory computer-readable medium stores instructions including a diagnostic program. When executed by a processor, the instructions cause a mobile device to: control a camera of the mobile device to take a screenshot of an external display of content, wherein the content comprises physiological information of a subject, to generate an image of the external display, wherein the physiological content from the external display comprises waveforms; process the image to isolate the physiological content from the external display; apply the diagnostic program to determine a diagnosis and whether the physiological content triggers an alarm, and generate and provide a recommendation via the mobile device when the physiological content does not trigger an alarm; and generate a notification comprising the physiological content to an external party over a communication network when the physiological content triggers an alarm.

[0008] According to another aspect of the present disclosure, a mobile device includes a camera, a processor and a memory. The memory stores instructions including a diagnostic program and the processor executes the instructions. A method of operating the mobile device includes controlling a camera of the mobile device to take a screenshot of an external display of content, wherein the content comprises physiological information of a subject, to generate an image of the external display. The physiological content from the external display comprises waveforms. The method also includes processing the image to isolate the physiological content from the external display; applying the diagnostic program to determine a diagnosis and whether the physiological content triggers an alarm, and generating and providing a recommendation via the mobile device when the physiological content does not trigger an alarm; and generating a notification comprising the physiological content to an external party over a communication network when the physiological content triggers an alarm.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The example embodiments are best understood from the following detailed description when read with the accompanying drawing figures. It is emphasized that the various features are not necessarily drawn to scale. In fact, the dimensions may be arbitrarily increased or decreased for clarity of discussion. Wherever applicable and practical, like reference numerals refer to like elements.

[0010] FIG. 1 illustrates a hybrid system for using a diagnostic mobile device, in accordance with a representative embodiment.

[0011] FIG. 2 illustrates a computer system for a diagnostic mobile device, in accordance with another representative embodiment.

[0012] FIG. 3 illustrates a method for diagnostic mobile device, in accordance with a representative embodiment.

[0013] FIG. 4 illustrates a method for diagnostic mobile device, in accordance with a representative embodiment.

[0014] FIG. 5 illustrates a hybrid system for using a diagnostic mobile device, in accordance with a representative embodiment.

[0015] FIG. 6 illustrates a hybrid system for using a diagnostic mobile device, in accordance with a representative embodiment.

[0016] FIG. 7 illustrates a hybrid system for using a diagnostic mobile device, in accordance with a representative embodiment.

[0017] FIG. 8 illustrates a hybrid method for using a diagnostic mobile device, in accordance with a representative embodiment.DETAILED DESCRIPTION

[0018] In the following detailed description, for the purposes of explanation and not limitation, representative embodiments disclosing specific details are set forth in order to provide a thorough understanding of embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from specific details disclosed herein remain within the scope of the appended claims. Descriptions of known systems, devices, materials, methods of operation and methods of manufacture may be omitted so as to avoid obscuring the description of the representative embodiments. Nonetheless, systems, devices, materials and methods that are within the purview of one of ordinary skill in the art are within the scope of the present teachings and may be used in accordance with the representative embodiments. It is to be understood that the terminology used herein is for purposes of describing particular embodiments only and is not intended to be limiting. Definitions and explanations for terms herein are in addition to the technical and scientific meanings of the terms as commonly understood and accepted in the technical field of the present teachings.

[0019] It will be understood that, although the terms first, second, third etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Thus, a first element or component discussed below could be termed a second element or component without departing from the teachings of the inventive concept.

[0020] As used in the specification and appended claims, the singular forms of terms ‘a,’ ‘an’ and ‘the’ are intended to include both singular and plural forms, unless the context clearly dictates otherwise. Additionally, the terms "comprises", and / or "comprising," and / or similar terms when used in this specification, specify the presence of stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0021] Unless otherwise noted, when an element or component is said to be “connected to,” “coupled to,” or “adjacent to” another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or intervening elements or components may be present. That is, these and similar terms encompass cases where one or more intermediate elements or components may be employed to connect two elements or components. However, when an element or component is said to be “directly connected” to another element or component, this encompasses only cases where the two elements or components are connected to each other without any intermediate or intervening elements or components.

[0022] The present disclosure, through one or more of its various aspects, embodiments and / or specific features or sub-components, is thus intended to bring out one or more of the advantages as specifically noted below.

[0023] As described herein, a diagnostic system implemented using a mobile device such as a smartphone may be used to provide telemedicine using screen photos from diagnostic ECG and patient monitoring devices taken by the mobile device. Screenshots of an external display of content may be taken by a camera of a mobile device to generate an image of the external display. The image may be processed to isolate the physiological content from the external display. A diagnostic program may be applied to determine a diagnosis from the isolated physiological content and whether the physiological content triggers an alarm. Arecommendation may be generated and provided via the mobile device when the physiological content does not trigger an alarm. A notification over a communication network to an external party and comprising the physiological content may be generated when the physiological content triggers an alarm. The external party may be selected from a set of choices of external parties, either by an operator or automatically. The diagnostic system may use digitized waveforms and vital signs, and may enable healthcare professionals to capture, save, access, transmit and diagnose extracted physiological information, resulting in improved care and treatment. The diagnostic system provides fast and accurate diagnosis of patient condition to result in appropriate care even in limited / isolated environments, including insular, rural, and underdeveloped countries / areas with limited access to hospitals, as well as in-home settings. The diagnostic system also may enable digitization and sharing in emergency situations where clinicians are not available and prompt care / treatment is required, enabling patients and healthcare professionals to receive timely and precise health information from remote hospitals, empowering them to make informed decisions and provide appropriate care or treatment.

[0024] FIG. 1 illustrates a system 100 for diagnostic mobile device, in accordance with a representative embodiment.

[0025] The system 100 in FIG. 1 is a system for diagnostic mobile device and includes components that may be provided together or that may be distributed. The system 100 in FIG. 1 includes a cardiograph 110A which is a diagnostic ECG cardiograph, a patient monitor HOB, and a mobile device 120. The mobile device 120 is not typically provided with the cardiograph 110A or the patient monitor HOB, such that the system 100 may be considered an ad-hoc temporary system when one or both of the cardiograph 110A and the patient monitor 110B are used together with the mobile device 120. The cardiograph is representative of a diagnostic ECG (cardiograph) device and includes an external display (i.e., external to the mobile device 120). The patient monitor 110B is representative of a continuous patient monitoring device, such as a bedside patient monitor and also includes an external display (i.e., external to the mobile device 120). An external display is a display external to the mobile device 120, though external displays are not limited to electronic displays of a cardiograph 110A and / or a patient monitor 110B. The cardiograph 110A is an example of a diagnostic ECG device, and the patient monitor 110B may be an example of a bedside patient monitor. In an alternative example, the cardiograph 110A and the patient monitor 110B may be represented by a single medical monitoring device, which isadapted to receive as an input raw physiological signal readings from a patient and subsequently visualize the readings in a waveform on its display.

[0026] A computer system 200 in FIG. 2 may be used to implement the mobile device 120 as a diagnostic mobile device, though a mobile device 120 may include more or fewer elements than the computer system 200 depicted in FIG. 2. The mobile device 120 may also include a controller with a memory that stores instructions and a processor that executes the instructions, as well as a camera configured to take screenshots of external displays under control of the controller.

[0027] The cardiograph 110A and / or the patient monitor HOB may each include a different display, or as an alternative one display showing combined diagnostic information, which can be captured by the camera of the mobile device 120. The display(s) may be configured to show physiological content such as waveforms generated by and displayed by the cardiograph 110A and or digital outputs of patient vital signs from the patient monitor HOB. The types of displays representative of a cardiograph 110A and / or patient monitor 110B may be embedded displays or may be separated from other components of the cardiograph 110A and / or patient monitor 110B. Such displays may include one or more input interface(s) that may connect to other elements or components, as well as interactive touch screens configured to display prompts to users and collect touch input from users.

[0028] The mobile device 120 may comprise, for example, a smartphone or a tablet. For example, the mobile device 120 may comprise an Apple iPhone or a Samsung Galaxy phone. The mobile device 120 may store instructions including a diagnostic program downloaded to the mobile device 120 from the Apple App store or the Google Play store. To the extent that the mobile device 120 includes a controller, the controller may perform some of the operations described herein directly and may implement other operations described herein indirectly. For example, a controller of the mobile device 120 may indirectly control operations such as by making a screen photo with its camera, generating and transmitting notifications comprising physiological content for an external party to review. Transmitted physiological content may be transmitted over a communications network when the physiological content triggers an alarm when a diagnostic program is executed on the mobile device 120. Other operations may be performed directly by a controller of the mobile device 120, such as applying the diagnostic program to determine a diagnosis from the captured physiological content by the camera andwhether the physiological content triggers the alarm. When the physiological content does not trigger an alarm, the controller of the mobile device 120 may determine a diagnosis by applying the diagnostic program. Accordingly, the processes implemented by a controller of the mobile device 120 may include steps not directly performed by the controller of the mobile device 120. For example, a specific type of heart attack called ST-segment elevation myocardial infarction (STEMI) can be diagnosed from a screen photo of a cardiograph using this system. In STEMI, a complete blockage of a coronary artery prevents blood flow to a part of the heart muscle. Immediate medical attention is crucial for STEMI, so the patient or someone with the patient should call emergency services immediately (911 in the US).

[0029] The mobile device 120 in FIG. 1 initiates a process which includes capturing with the camera the screenshot of an external display of content which includes physiological information of the subject to generate an image, then processing the image at SI 30 and generating a report to be communicated over the internet 101 for an external party 190 when the physiological content triggers an alarm. The external display may be a screen of the cardiograph 110A or the patient monitor HOB.

[0030] In emergency situations, prompt and accurate diagnosis and treatment may be vital for at- risk patients, but clinicians are not always available to provide care or treatment in all places. Advances in high-resolution screens on diagnostic ECG and patient monitoring devices, along with portable devices such as smartphones and tablets, enable the digitization of waveforms (e.g., electrocardiogram (ECG), photoplethysmography (PPG / SpO2), carbon dioxide (CO2), invasive blood pressure (IBP) and non-invasive blood pressure (NIBP), respiration and more) as well as vital signs information (e.g., heart rate, blood pressure, oxygen saturation, carbon dioxide, respiration rate, temperature, etc.) from screen photos taken with a camera of a mobile device as well as the transmission to a remote hospital when appropriate. This diagnostic system implemented via a mobile device may offer automatic digitization, report generation, secure data transmission to remote hospitals or automatic diagnosis in the smartphone, and appropriate action advice based on abnormalities / conditions. The digitization of waveforms may involve digitization into a digital format in which the waveforms are represented as data matrices. As a result, the system 100 in FIG. 1 facilitates timely and accurate care for at-risk patients, particularly in undeveloped areas and home / emergency settings.

[0031] Although not shown in FIG. 1, the mobile device 120 may also capture screenshots ofpaper. Waveforms and vital signs may be present (displayed) on paper, such as from printers. Screenshots of paper may be filtered for numerous types of extraneous components, including faded tracings, grid lines, noise, crumpling of the paper, stains, artifacts, or scribbling from degraded paper. Faded tracings may result from depleted ink and / or worn printer components. Grid lines may be present from the source of the waveforms and vital signs being printed on the paper. Noise may simply be electronic noise components that erroneously result in random printing. Crumpling, stains or scribbling may simply be the result or intended or unintended operator activity. The extraneous components resulting from any of these types of defects on or of the paper may be filtered in order to isolate the physiological content from the external display. Some common types of artifacts found on printed ECG reports include movement artifact, baseline wander, electrode artifact, AC interference (power line interference), electromagnetic interference, muscle tremor artifact, lead break artifact, oversensing artifact, ECG machine malfunction, baseline drift, lead switching artifacts, and others.

[0032] Filtering may also include filtering of screenshots of the diagnostic ECG or patient monitoring devices. Diagnostic ECG or patient monitoring devices may have cracks, defective pixels, dead pixels, stuck pixels, hot pixels, screen burn-in, backlight bleeding, color inconsistencies, flickering, vertical or horizontal lines, touchscreen malfunction, or screen shadowing from degraded screens. The extraneous components resulting from any of these types of defects may be filtered in order to isolate the physiological content from the external display.

[0033] FIG. 2 illustrates a computer system for a diagnostic mobile device, in accordance with another representative embodiment.

[0034] Referring to FIG. 2, the computer system 200 includes a set of software instructions that can be executed to cause the computer system 200 to perform any of the methods or computer- based functions disclosed herein. The computer system 200 may operate as a standalone device. Alternatively, the computer system 200 may operate as a client user computer in a server-client user network environment, or as a peer-to-peer computer system in a peer-to-peer (or distributed) network environment. The computer system 200 can also be implemented as or incorporated into various devices, such as a tablet computer or any other camera-equipped machine capable of executing a set of software instructions (sequential or otherwise) that specify actions to be taken by that machine. In an embodiment, the computer system 200 can be implemented using electronic devices that provide voice, video or data communication.

[0035] As illustrated in FIG. 2, the computer system 200 includes a processor 210. The processor 210 may be considered a representative example of a processor of a controller and executes instructions to implement some or all aspects of methods and processes described herein. The processor 210 is tangible and non-transitory. As used herein, the term “non- transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period. The term “non-transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time. The processor 210 is an article of manufacture and / or a machine component. The processor 210 is configured to execute software instructions to perform functions as described in the various embodiments herein. The processor 210 may be a general- purpose processor or may be part of an application specific integrated circuit (ASIC). The processor 210 may also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. The processor 210 may also be a logical circuit, including a programmable gate array (PGA), such as a field programmable gate array (FPGA), or another type of circuit that includes discrete gate and / or transistor logic. The processor 210 may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices.

[0036] The term “processor” as used herein encompasses an electronic component able to execute a program or machine executable instruction. References to a computing device comprising “a processor” should be interpreted to include more than one processor or processing core, as in a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems. The term computing device should also be interpreted to include a collection or network of computing devices each including a processor or processors. Programs have software instructions performed by one or multiple processors that may be within the same computing device or which may be distributed across multiple computing devices.

[0037] The computer system 200 further includes a main memory 220 and a static memory 230, where memories in the computer system 200 communicate with each other and the processor 210 via a bus 208. Either or both of the main memory 220 and the static memory 230 may beconsidered representative examples of a memory of a controller, and store instructions used to implement some, or all aspects of methods and processes described herein. Memories described herein are tangible storage mediums for storing data and executable software instructions and are non-transitory during the time software instructions are stored therein. As used herein, the term “non-transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period. The term “non-transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time. The main memory 220 and the static memory 230 are articles of manufacture and / or machine components. The main memory 220 and the static memory 230 are computer-readable mediums from which data and executable software instructions can be read by a computer (e.g., the processor 210). Each of the main memory 220 and the static memory 230 may be implemented as one or more of random access memory (RAM), read only memory (ROM), flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, tape, compact disk read only memory (CD-ROM), digital versatile disk (DVD), floppy disk, blu-ray disk, or any other form of storage medium known in the art. The memories may be volatile or non-volatile, secure and / or encrypted, unsecure and / or unencrypted.

[0038] “Memory” is an example of a computer-readable storage medium. Computer memory is any memory which is directly accessible to a processor. Examples of computer memory include, but are not limited to RAM memory, registers, and register files. References to “computer memory” or “memory” should be interpreted as possibly being multiple memories. The memory may for instance be multiple memories within the same computer system. The memory may also be multiple memories distributed amongst multiple computer systems or computing devices.

[0039] As shown, the computer system 200 further includes a video display unit 250, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT), for example. Additionally, the computer system 200 includes an input device 260, such as a keyboard / virtual keyboard or touch-sensitive input screen or speech input with speech recognition, and a cursor control device 270, such as a mouse or touch-sensitive input screen or pad. The computer system 200 also optionally includes a disk drive unit 280, a signal generation device 290, such as a speaker or remote control, and / or a network interface device 240.

[0040] In an embodiment, as depicted in FIG. 2, the disk drive unit 280 includes a computer- readable medium 282 in which one or more sets of software instructions 284 (software) are embedded. The sets of software instructions 284 are read from the computer-readable medium 282 to be executed by the processor 210. Further, the software instructions 284, when executed by the processor 210, perform one or more steps of the methods and processes as described herein. In an embodiment, the software instructions 284 reside all or in part within the main memory 220, the static memory 230 and / or the processor 210 during execution by the computer system 200. Further, the computer-readable medium 282 may include software instructions 284 or receive and execute software instructions 284 responsive to a propagated signal, so that a device connected to a network 201 communicates voice, video or data over the network 201. The software instructions 284 may be transmitted or received over the network 201 via the network interface device 240.

[0041] FIG. 3 illustrates a method for diagnostic mobile device, in accordance with a representative embodiment.

[0042] The method of FIG. 3 starts at S310 when the camera of a mobile device 120 is controlled to capture a screenshot. The screenshot may be of an external display of content, which comprises diagnostic information content of the subject, on a piece of paper, or an external display of physiological content on a cardiograph 110A or a patient monitor HOB. As an example, the mobile device 120 may be used to digitize paper-based ECG reports to result in signal data. Paper-based ECG reports are the physical printouts generated by the electrocardiograph and contain waveforms and basic measurements, often referred to as "short form measurements". The smartphone-based diagnostic system described herein may digitize and process these paper-based ECG reports, allowing for the extraction and analysis of the waveform data and essential measurements. This digital transformation enhances accessibility, efficiency, and accuracy in managing and interpreting ECG information, providing remote healthcare professionals with valuable insights for diagnosis and treatment decisions. As another example, the mobile device 120 may be used to digitize the screen images of smartwatch and portable devices such as another mobile device such as a smartphone, tablet or laptop that shows the waveforms and vital signs of the patient. Consequently, users have the capability to receive diagnosis and treatment advice pertaining to any abnormalities or conditions of the patients at any time and from any location.

[0043] At S320, the image is processed to isolate the diagnostic information in the content from the external display. Isolating diagnostic information in the content may involve identifying the content salient to a diagnostic program, such as physiological waveforms from a cardiograph 110A or digital data (numerical numbers) representative of vital signs from a patient monitor HOB. Waveforms may include, for example, ECG, PPG / SpO2, CO2, IBP / NIBP and / or respiration. Vital signs may include, for example, heart rate, blood pressure, oxygen saturation, carbon dioxide, respiration rate and / or temperature. The waveforms and vital signs are isolated from the other pixels in the screen shots taken from diagnostic ECG and patient monitoring devices using a mobile device. The digitization may convert the waveforms and vital signs into a digital format or, when appropriate, one of several available digital formats. The digitization of waveforms may involve digitization into a digital format in which the waveforms are represented as data matrices. The digital format may represent waveforms as data matrices and vital signs as text and values. A data matrix is a common way to store ECG recordings. In this format, each row represents the waveform of a single lead (electrical activity recording from a specific placement on the body), and each column represents a data point or sample captured at a specific time point. This matrix can be transposed, where rows and columns swap positions.

[0044] At S330, a diagnostic program is applied to the physiological content from the external display isolated at S320. The diagnostic program may automatically generate diagnostic ECG reports and patient monitoring reports. Diagnostic ECG reports may include digitized waveforms and measurements. Measurements may include, for example, heart rate, PR interval, QRS duration (QRSd), QT / QTc and / or P / QRS / T durations. The diagnostic ECG reports may be provided in a specific format such as an ECG XML data format, or another standard format such as MIT or DICOM. Patient monitoring reports may include digitized waveforms and vital signs information. Waveforms and vital sign information may be provided in a specific format such as the Philips DWC data format or another format such as MIT, DICOM, or others. The reports may be securely saved and transmitted. The diagnostic program may apply a deep neural network (DNN) in the mobile device 120.

[0045] At S340, from identified diagnostic information in the physiological content a determination is made as to whether an alarm is triggered in applying the automated diagnostic program. If the condition is severe, for example, the diagnostic program may determine an occurrence of the ST-segment elevation myocardial infarction (STEMI), the digitized data maybe immediately and automatically transmitted such as to a remote hospital, and otherwise the diagnosis may simply be output via the mobile device to the patient or a first responder or clinician on-site with the patient, depending on the context and the conditions.

[0046] That is, if an alarm is triggered (S340 = Yes), a notification is generated at S360. The notification includes the physiological content, and the notification is to an external party over a communication network. The notification may be an automatic transmission of diagnostic information, such as to a remote hospital in an undeveloped region or in an emergency scenario or home setting. This notification may also allow clinicians to remotely access the data and provide appropriate treatment recommendations to the patients, first-responders, or other personnel who are present with the patients. The external party may be selected from a set of choices of external parties, either by an operator or automatically.

[0047] When the alarm is not triggered (S340 = No), a recommendation is generated and provided via the mobile device 120. For example, the recommendation may include an audio announcement including voice or a simple tone, a visual announcement including text or a simple color or pattern.

[0048] The method of FIG. 3 may be used to accurately digitize waveforms and vital sign information from screen photos displayed on diagnostic ECG and patient monitoring devices using a smartphone camera. In turn, this may allow for generation of comprehensive reports for various types of ECG devices, such as standard multi -lead ECG (limb leads, chest leads, standard 12-lead, 16-lead, and 18-lead), as well as generating reports for patient monitoring devices. The digitization of waveforms and vital signs derived as a physiological information from the screen photos also allows for electronic storage, which saves physical space and enables easy access and secure sharing of patient data. The implementation of the system 100 may save lives by providing essential medical information and guidance to patients, first responders, and bystanders in critical situations, and therefore may significantly improve healthcare delivery efficiency while facilitating seamless communication among healthcare providers, ultimately enhancing patient care outcomes.

[0049] FIG. 4 illustrates a method for diagnostic mobile device, in accordance with a representative embodiment.

[0050] The method of FIG. 4 is performed by a mobile device with a diagnostic program. The mobile device includes a camera that takes screen photos displayed on diagnostic ECG andpatient monitoring devices for telemedicine.

[0051] After the method of FIG. 4 starts, the method diverges at S402 based on whether a patient is a new patient.

[0052] If the patient is a new patient (S402 = Yes), at S404 a configuration of patient information for the new patient is generated.

[0053] If the patient is in the system already (S402 = No) or otherwise after S404, the method includes a selection of the patient at S406.

[0054] At S408, a photo of the cardiograph screen is taken using a smartphone camera or a camera of another type of mobile device with a diagnostic program installed thereon. The process from S408 to S430 involves determining physiological information in the screen photo and converting the corresponding portions of the screen image into waveforms and vital sign information.

[0055] At S410, the image is checked for distortion or visual anomalies. Image correction techniques are applied at S410, if necessary. The image checked at S410 may be checked for distortion or visual anomalies that may affect the accuracy of a waveform analysis. If necessary, image correction techniques may be applied to remove any distortion or noise from the image.

[0056] At S412, vital sign information in its numerical form may be extracted from an image using a technique such as optical character recognition (OCR). S412 is often performed only for the patient monitoring device 110b and not the cardiograph 110a, because of the type of the data form this physiological information has. Extracted vital signs information may include, for example, heart rate, blood pressure, oxygen saturation, carbon dioxide, respiration rate and / or temperature.

[0057] At S414, the image is cropped to isolate the ECG waveform and to remove any extraneous elements. Cropping may involve adjusting one or more edges of the image inward. The cropping at S414 is automatic, though the operator of the mobile device 120 may also or alternatively be prompted to manually adjust and confirm the movement of the one or more edges.

[0058] At S416, the ECG image is segmented to separate the ECG image into individual leads. The ECG image may be separated into individual leads such as, for example, the standard 12- lead, 16-lead, and / or 18-lead for diagnostic ECG. The ECG image may also or alternatively be separated into waveforms such as ECG, PPG / SpO2, CO2, IBP / NIBP and / or respiration for apatient monitoring device. Diagnostic ECG devices, also known as cardiographs or electrocardiographs, typically provide multi-lead ECG waveforms, such as standard 12-lead, 16- lead, and 18-lead ECGs, which allow for a more comprehensive assessment of the heart's electrical activity. Patient monitoring devices typically offer a range of functionalities, providing both single-lead waveforms like ECG, PPG / SpO2, CO2, IBP / NIBP, and respiration, as well as vital signs information such as heart rate, blood pressure, oxygen saturation, carbon dioxide levels, respiration rate, temperature, and others. Some patient monitoring devices also offer the ability to measure and display ECG waveforms, either single-lead or multi-lead.

[0059] At S418, grid lines, noise and other artifacts are removed from the ECG image. The elements removed at S418 are those which may interfere with subsequent analysis.

[0060] At S420, the image is converted to a binary format to facilitate further processing. When an ECG waveform image is converted to a binary format, it means that each pixel in the image is represented by a single bit (either 0 or 1). In this binary representation: 0 typically corresponds to the absence of a waveform (background or non-significant pixels). 1 corresponds to the presence of a waveform (significant pixels representing the ECG signal).

[0061] At S422, dilation and skeletonization techniques are applied to smooth the waveform and enhance waveform features. Dilation expands or thickens regions within an image. It is commonly used to enhance features or structures. Dilation is applied to make the ECG waveform more prominent by expanding its boundaries. Skeletonization simplifies the shape of an object while preserving its essential structure. Skeletonization aims to extract a simplified representation of the waveform, emphasizing its core features.

[0062] At S424, linear interpolation is performed to fill in any missing data points in the waveform.

[0063] At S426, the leads are separated to isolate individual waveforms.

[0064] At S428, the signal is extracted from the waveform and saved as a matrix for analysis.

[0065] At S430, the multi-lead ECG or patient monitoring report is generated.

[0066] At S432, the method of FIG. 4 includes determining if the patient’s condition is severe. If the patient’s condition is severe (S432 = Yes), a report of the patient condition is transmitted to a remote hospital. If the patient’s condition is not severe (S432 = No), at S434 the diagnostic program on the mobile device 120 automatically diagnoses the patient using a trained machinelearning model or a deep learning model.

[0067] Examples of a deep neural network (DNN) usable to determine severity of the patient condition include LSTM, AlexNet, SqueezeNet, GoogLeNet, ShuffleNet, MobileNet, VGG, ResNet, SE-ResNet, ResNeXt, DenseNet, SENet, Inception, NASNet, and / or Transformer. These types of deep neural networks may be used for automatic diagnosis on mobile devices for patients with non-severe conditions. Furthermore, the input data for a deep neural network model for automatic diagnosis may consist of either digitized multi-lead waveforms or segmented images of the screen in the screen shots. This type of data may be used to train and test the deep neural network model. In the context of patient monitoring devices, a deep neural network model may be used to automatically generate a diverse range of alarms. These alarms may be designed to promptly alert healthcare providers or on-site personnel to potential issues concerning the patient's condition. The alarm system may encompass a wide array of alerts, including heart rate alarms, SpO2 alarms, blood pressure alarms, respiratory rate alarms, apnea alarms, arrhythmia alarms, and more. This comprehensive approach ensures that various aspects of the patient's well-being are continuously monitored, facilitating timely interventions when necessary. The type of alert may vary based on the diagnosis by the diagnostic program, and the recipients of the alerts may vary based on the type of alert.

[0068] After S434 or S436, at S438 a determination is made as to whether any abnormalities were detected. If abnormalities were detected (S438 = Yes), at S440 the patient is provided advice based on the type(s) of abnormalities. Otherwise, (S438 = No), at S442 the patient is notified of the absence of abnormalities.

[0069] After S440 or S442 the method of FIG. 4 ends.

[0070] FIG. 5 illustrates a hybrid system for using a diagnostic mobile device, in accordance with a representative embodiment.

[0071] In FIG. 5, a diagnostic system based on a mobile device 120 such as a smartphone captures the screen photo of a portable / bedside patient monitoring system as an example of a patient monitor HOB for telemedicine.

[0072] FIG. 6 illustrates a hybrid system for using a diagnostic mobile device, in accordance with a representative embodiment.

[0073] In FIG. 6, a diagnostic system based on a mobile device 120 such as a smartphone captures the screen photo of a continuous patient monitoring system in the context of prehospitalcare such as an ambulance. The continuous patient monitoring system in FIG. 6 is another example of a patient monitor HOB for telemedicine.

[0074] FIG. 7 illustrates a hybrid system for using a diagnostic mobile device, in accordance with a representative embodiment.

[0075] In FIG. 7, a diagnostic system based on a mobile device 120 such as a smartphone captures the screen photo of a paper-based ECG report for telemedicine. FIG. 7 illustrates that the system 100 of FIG. 1 may be used to capture, analyze and selectively transmit notifications based on paper-based ECG reports or other paper-based reports.

[0076] FIG. 8 illustrates a hybrid method for using a diagnostic mobile device, in accordance with a representative embodiment.

[0077] In FIG. 8, a simplified process for converting a screen photo on the diagnostic ECG device to ECG waveforms in digital starts with taking a photo using the mobile device 120. At S815, the screen is segmented. At S816, the ECG image on the screen is segmented. At S818, grid lines and noise / artifacts are removed, and then binarization is performed. At S828 the signal is extracted and saved as a data matrix / array.

[0078] In an embodiment, dedicated hardware implementations, such as application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic arrays and other hardware components, are constructed to implement one or more of the methods described herein. One or more embodiments described herein may implement functions using two or more specific interconnected hardware modules or devices with related control and data signals that can be communicated between and through the modules. Accordingly, the present disclosure encompasses software, firmware, and hardware implementations. Nothing in the present application should be interpreted as being implemented or implementable solely with software and not hardware such as a tangible non-transitory processor and / or memory.

[0079] In accordance with various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system that executes software programs. Further, in an exemplary, non-limited embodiment, implementations can include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing may implement one or more of the methods or functionalities as described herein, and a processor described herein may be used to support a virtual processing environment.

[0080] Accordingly, diagnostic mobile device enables mobile device to use a camera to take screenshots of an external display of content, the content also including physiological information content of a subject, to generate an image of the external display. The image may be processed to isolate the physiological content from the external display. A diagnostic program may be applied to determine a diagnosis and whether the physiological content triggers an alarm. A recommendation may be generated and provided via the mobile device when the physiological content does not trigger an alarm. A notification comprising the physiological content to an external party over a communication network may be generated when the physiological content triggers an alarm.

[0081] Although diagnostic mobile device has been described with reference to several exemplary embodiments, it is understood that the words that have been used are words of description and illustration, rather than words of limitation. Changes may be made within the purview of the appended claims, as presently stated and as amended, without departing from the scope and spirit of diagnostic mobile device in its aspects. Although diagnostic mobile device has been described with reference to particular means, materials and embodiments, diagnostic mobile device is not intended to be limited to the particulars disclosed; rather diagnostic mobile device extends to all functionally equivalent structures, methods, and uses such as are within the scope of the appended claims.

[0082] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to serve as a complete description of all of the elements and features of the disclosure described herein. Many other embodiments may be apparent to those of skill in the art upon reviewing the disclosure. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Additionally, the illustrations are merely representational and may not be drawn to scale. Certain proportions within the illustrations may be exaggerated, while other proportions may be minimized. Accordingly, the disclosure and the figures are to be regarded as illustrative rather than restrictive.

[0083] One or more embodiments of the disclosure may be referred to herein, individually and / or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any particular invention or inventive concept.Moreover, although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the description.

[0084] The Abstract of the Disclosure is provided to comply with 37 C.F.R. § 1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. This disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all of the features of any of the disclosed embodiments. Thus, the following claims are incorporated into the Detailed Description, with each claim standing on its own as defining separately claimed subject matter.

[0085] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. As such, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments which fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents and shall not be restricted or limited by the foregoing detailed description.

Claims

CLAIMS:

1. A mobile device (120), comprising: a camera; a memory that stores instructions including a diagnostic program; and a processor (210) that executes the instructions, wherein, when executed by the processor (210), the instructions cause the mobile device (120) to: control the camera to take a screenshot of an external display of content, wherein the content comprises physiological information content, to generate an image of the external display; process the image to isolate the physiological content from the external display; apply the diagnostic program to the isolated physiological content to determine a diagnosis and whether the physiological content triggers an alarm, and generate and provide a recommendation via the mobile device (120) when the physiological content does not trigger an alarm; and generate a notification comprising the physiological content to an external party (190) over a communication network (201) when the physiological content triggers an alarm.

2. The mobile device (120) of claim 1, wherein the external display comprises a patient monitor (HOB).

3. The mobile device (120) of claim 1, wherein the physiological content from the external display comprises waveforms.

4. The mobile device (120) of claim 3, wherein, when executed by the processor (210), the instructions further cause the mobile device (120) to: digitize the waveforms such that the notification comprises a digitization of the physiological content from the external display.

5. The mobile device (120) of claim 4, wherein digitization of the waveforms comprises converting waveforms from the image into a digital format in which the waveforms are represented as data matrices; and wherein, when executed by the processor (210), the instructions further cause the mobile device (120) to: segment the image to separate the image into individual leads for diagnostic ECG or waveforms; apply dilation and skeletonization to smooth the waveforms and enhance features of the waveforms; and perform linear interpolation to fill in missing data points in the waveforms.

6. The mobile device (120) of claim 1, wherein, when executed by the processor (210), the instructions further cause the mobile device (120) to: digitize the physiological content such that the notification comprises a digitization of the physiological content from the external display.

7. The mobile device (120) of claim 1, wherein, when executed by the processor (210), the instructions further cause the mobile device (120) to: select the external party (190) from a plurality of options of external parties based on the diagnosis.

8. The mobile device (120) of claim 1, wherein, when executed by the processor (210), the instructions cause the mobile device (120) to: control the camera to take a screenshot of content on paper to generate an image of the content on paper, wherein the external display of content comprises the paper.

9. The mobile device (120) of claim 1, wherein isolating the physiological content from the content of the external display comprises filtering out at least one of faded tracings, grid lines, noise, crumpling of paper, stains, artifacts, or scribbling from degraded paper, orcracks, defective pixels, dead pixels, stuck pixels, hot pixels, screen burn-in, backlight bleeding, color inconsistencies, flickering, vertical or horizontal lines, touchscreen malfunction, or screen shadowing, from degraded screens.

10. A tangible, non-transitory computer-readable medium (282) that stores instructions including a diagnostic program, which when executed by a processor (210), cause a mobile device (120) to: control a camera of the mobile device (120) to take a screenshot of an external display of content, wherein the content comprises physiological information content, to generate an image of the external display, wherein the physiological content from the external display comprises waveforms; process the image to isolate the physiological content from the external display content; apply the diagnostic program to determine a diagnosis and whether the physiological content triggers an alarm, and generate and provide a recommendation via the mobile device (120) when the physiological content does not trigger an alarm; and generate a notification comprising the physiological content to an external party (190) over a communication network (201) when the physiological content triggers an alarm.

11. The tangible, non-transitory computer-readable medium (282) of claim 10, wherein, when executed by the processor (210), the instructions further cause the mobile device (120) to: digitize the waveforms such that the notification comprises a digitization of the content from the external display.

12. The tangible, non-transitory computer-readable medium (282) of claim 11, wherein digitization of the waveforms comprises converting waveforms from the image into a digital format in which the waveforms are represented as data matrices; and wherein, when executed by the processor (210), the instructions further cause the mobile device (120) to: segment the image to separate the image into individual leads for diagnostic ECG or waveforms;apply dilation and skeletonization to smooth the waveforms and enhance features of the waveforms; and perform linear interpolation to fill in missing data points in the waveforms.

13. A method of operating a mobile device (120) comprising a camera, a memory that stores instructions including a diagnostic program and a processor (210) that executes the instructions, the method comprising: controlling, by the processor (210) executing the instructions, the camera to take a screenshot of an external display of content, wherein the content comprises physiological information content, to generate an image of the external display; processing the image to isolate the physiological content from the external display; applying the diagnostic program to determine a diagnosis and whether the physiological content triggers an alarm, and generating and providing a recommendation via the mobile device (120) when the physiological content does not trigger an alarm; and generating a notification comprising the physiological content to an external party (190) over a communication network (201) when the physiological content triggers an alarm.

14. The method of claim 13, wherein the physiological content from the external display comprises waveforms.

15. The method of claim 14, further comprising: digitize the waveforms such that the notification comprises a digitization of the physiological content from the external display, by converting waveforms from the image into a digital format in which the waveforms are represented as data matrices.

16. The method of claim 15, further comprising: segment the image to separate the image into individual leads for diagnostic ECG or waveforms; apply dilation and skeletonization to smooth the waveforms and enhance features of the waveforms; and perform linear interpolation to fill in missing data points in the waveforms.

Images