Method and system for medical diagnostic image acquisition and dimensional analysis

Abstract

Presented herein are methods and systems for health monitoring based on body dimensions, changes in body dimensions, or both, derived from images of a human being collected over time. In another further aspect, an alert is sent to medical personnel for further investigation of the health status of a human user. In one aspect, body dimensions are estimated based on known values ​​of distinguishable reference dimensions in each of a plurality of images. In some examples, distances between facial features in an image are normalized for a measured interpupillary distance associated with the image.

Description

Technical Field

[0001] The described embodiments relate to the diagnosis of diseases based on images collected over time.

Background Art

[0002] Many diseases and health conditions cause changes in facial dimensions. For example, people suffering from brain and cranial nerve tumors, Cushing's disease, acromegaly, certain strokes or brain infections, etc. show significant changes in facial dimensions over time. Substance abuse with chemicals such as steroids and opioids also causes significant changes in facial dimensions over time. In one example, more than 250,000 people are newly diagnosed with conditions related to brain tumors each year. Depending on the pathology of the brain tumor, the period until diagnosis can be up to 7 years at most. Therefore, the pathology of brain tumors often remains undiagnosed for several years. The long intervening period from onset to diagnosis often limits treatment options, reduces effectiveness, and dramatically increases overall treatment costs. Furthermore, patients treated to remove tumors are monitored for disease recurrence by expensive, time-consuming, and stressful blood and imaging (e.g., magnetic resonance imaging) tests.

[0003] When considering facial dimensions even slightly in daily medical practice, typically, facial dimensions are visually examined by a physician when the patient visits the hospital. Visible facial distortion can be adopted by the physician as one clue for medical diagnosis. However, as part of medical diagnosis, it is not common for a physician to measure facial dimensions and track changes in the measured values of facial dimensions over time. Since retrospective data is usually not available, physicians do not adopt accurate measurement and tracking of facial dimensions as part of medical diagnosis. Small changes in facial dimensions are often not noticed by physicians in a normal clinical environment. As a result, disease precursors are often ignored until a significant period has passed after the onset of the disease.

[0004] Improvements in patient health monitoring systems are desired. In particular, improvements in image collection and image analysis for medical diagnosis are desired.

Summary of the Invention

[0005] Methods and systems for health monitoring based on body dimensions, changes in body dimensions, or both, derived from human images collected over a long period of time are presented herein.

[0006] In one embodiment, an image-based medical diagnostic system is communicatively coupled to one or more image acquisition devices available to a human user. These one or more image acquisition devices capture images of the body. The acquired images are stored and analyzed.

[0007] In a further embodiment, images of a human user's body are collected regularly over a considerable period of time (e.g., at least once a month over several years). In some embodiments, the image-based medical diagnostic system accurately measures facial dimensions and changes in facial dimensions over time and identifies whether any measured dimension, change in facial dimension, or both exceed one or more thresholds indicating the onset or recurrence of a medically significant condition (e.g., the onset or recurrence of a disease, tumor, etc.). One or more dimensions are estimated at least in part based on known values ​​of identifiable reference dimensions for each of the multiple images.

[0008] In some embodiments, an image-based medical diagnostic tool (IBMDT) determines the number of pixels between pupils in each image (i.e., interpupillary distance). Based on the known interpupillary distance and the determined number of pixels, the IBMDT estimates a scaling factor that relates to the actual distance per pixel in each image. Furthermore, the IBMDT estimates the actual distance between selected facial features based on the number of pixels between selected facial features in each image and the scaling factor associated with each image.

[0009] In this way, the distances between any selected facial features in an image are normalized to the measured interpupillary distance (IPD) associated with that image. Since the actual IPD is known, the distances between any selected facial features can also be determined.

[0010] In another further embodiment, an image-based medical diagnostic system communicates a warning (to, for example, a human user, a qualified healthcare professional, etc.) indicating abnormal body dimensions, changes in body dimensions, or both. In this way, the image-based medical diagnostic system prompts the healthcare professional to further investigate the human user's condition.

[0011] The above is a summary and therefore includes simplifications, generalizations, and omissions of details where necessary, and as a result, those skilled in the art will understand that the summary is illustrative and not in any way restrictive. Other aspects of the apparatus and / or processes described herein, features of the invention, and advantages will become apparent in the non-restrictive detailed description provided herein. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows one embodiment of an image-based medical diagnostic system (IBMDS) 100, which includes one or more imaging devices 102 and an image-based diagnostic tool (IBMDT) 101, in at least one aspect. [Figure 2] This figure shows an exemplary facial image 160 and the dimensions of interest measured by IBMDS. [Figure 3] This figure shows a memory 131 that stores face records 151-156, which include face images collected over several years. [Figure 4] This flowchart shows a method 200 for implementing the health monitoring and communication gateway functions exemplified herein. [Modes for carrying out the invention]

[0013] Herein, background examples and several embodiments of the present invention will be given in detail, examples of which are shown in the accompanying drawings.

[0014] Methods and systems for conducting health monitoring based on body dimensions, changes in body dimensions, or both, derived from human images collected over a long period of time are presented herein.

[0015] In one embodiment, an image-based medical diagnostic system is communicatively connected to one or more image acquisition devices available to a human user. In some embodiments, one or more image acquisition devices capture images of faces. The acquired images are transmitted to the image-based medical diagnostic system for storage and analysis.

[0016] Figure 1 shows one embodiment of an image-based medical diagnostic system (IBMDS) 100, which includes one or more imaging devices 102 and an image-based diagnostic tool (IBMDT) 101. In the embodiment shown in Figure 1, the imaging device 102 collects an image of the face of a human user 107. The image 103 is communicated to the IBMDT 101.

[0017] As shown in Figure 1, the IBMDT101 includes a processor 120, memory 130, a bus 140, and a wireless communication transceiver 150. The processor 120, memory 130, and wireless communication transceiver 150 are configured to communicate via the bus 140.

[0018] As shown in Figure 1, the IBMDT101 is configured to receive an image 103 from the imaging device 102. As described herein, memory 130 also includes multiple memories 132 that store program code causing the processor 120 to perform image-based medical diagnostic functions when executed by the processor 120. Memory 130 also includes multiple memories 131 that store measurement data extracted from the image 103 by the IBMDT101.

[0019] In some embodiments, images of a human user's face are collected regularly over a considerable period of time (e.g., at least once a month over several years). The image-based medical diagnostic system accurately measures the dimensions of the face and changes in those dimensions over time. The image-based medical diagnostic system also identifies whether any measured facial dimensions, changes in facial dimensions, or both exceed one or more thresholds indicating the onset or recurrence of a medically significant condition (e.g., the onset or recurrence of a disease, tumor, etc.). One or more dimensions are estimated at least in part based on known values ​​of identifiable reference dimensions in each of the multiple images.

[0020] In the embodiment shown in Figure 1, the IBMDT101 is communicatively connected to the imaging device 102 via a wired communication link. The IBMDT101 receives the image 103 and extracts facial dimensions from the acquired image. Figure 2 shows an exemplary image 160. In one example, the IBMDT101 extracts distances related to interpupillary distance (IPD), lower face height (LFH), and interauricular distance (EE).

[0021] In general, since each image of a user's face can be captured at different angles and distances, the relationship between the number of image pixels and the actual distance between physical points captured in, for example, image 103, changes. However, the interpupillary distance remains nearly constant throughout an adult's lifetime.

[0022] In one embodiment, IBMDT101 determines the number of pixels between pupils located in each image 103 (i.e., the interpupillary distance). Based on the known interpupillary distance and the determined number of pixels, IBMDT101 estimates a scaling factor that relates the actual distance per pixel in each image 103. Furthermore, IBMDT101 estimates the actual distance between selected facial features based on the number of pixels between selected facial features located in each image 103 and the scaling factor associated with each image 103.

[0023] In this way, the distances between the features of any selected face in the image are normalized with respect to the measured interpupillary distance (IPD) associated with that image. Since the actual IPD is known, the distances between the features of any selected face are also determined.

[0024] Although the IPD has been described above as a preferred reference dimension in a face image, generally, any stable facial dimension, or set of markings, captured in an image of a human face can be intended as a reference dimension in this patent document. Facial dimensions such as the distance from ear to ear and the LFH are described with reference to FIG. 2, but generally, any suitable dimension can be estimated by the IBMDT101, and any suitable stable body dimension can be adopted as a reference dimension.

[0025] Furthermore, although the operation of an image-based medical diagnostic system has been described with reference to face images, facial dimensions, and reference dimensions of a face, generally, the image-based medical diagnostic system described in this specification can operate based on an image of any part of the human body to track any suitable body dimension, provided that the reference dimension having a known value is identifiable within each image under consideration.

[0026] Generally, the IBMDT101 is communicatively connected to one or more imaging devices 102 by any suitable communication link (e.g., a wired or wireless communication link). In some embodiments, the IBMDT101 and the imaging device 102 are communicatively connected by a wireless communication link operating in accordance with any suitable wireless communication protocol (e.g., Bluetooth®, WiFi®, ZigBee®, any cellular network-based protocol, or other communication network). In some other embodiments, the IBMDT101 and the imaging device 102 are communicatively connected to a wired communication link operating in accordance with any suitable wired communication protocol (e.g., a serial data link, Ethernet®, etc.).

[0027] In one example, the IBMDT101 is implemented as part of a portable computing device such as a smartphone, a tablet computing system, etc. In many of these examples, the portable computing device integrates a computing system that implements the functions of the IBMDT101 and an imaging device 102. In other embodiments, the portable computing device and the collected images are uploaded to an external computing system (e.g., an external server, a cloud-based computing system, etc.) via a signal 106 communicated from the IBMDT101. In some of these examples, the signal 106 is communicated via a wireless communication link (e.g., via the wireless transceiver 150 shown in FIG. 1) or a wired communication link (e.g., via a wired connection between the portable computing device and a communication network (e.g., the Internet, etc.)).

[0028] Generally, the imaging device 102 includes any suitable imaging device that can be used by the user 107 to collect face images. In some embodiments, the imaging device 102 collects a two-dimensional image of the user 107's face. In these examples, the imaging device 102 includes a two-dimensional imaging element (e.g., a charge-coupled device, complementary metal oxide on a silicon element, etc.).

[0029] In some of these examples, the IBMDT101 generates a three-dimensional model of a human face based on a series of images collected by the imaging device 102 from different angles with respect to the human face. The distances between selected facial features are determined by the IBMDT101 based on the three-dimensional model. As described above, the distance between any selected points within the three-dimensional model is normalized with respect to a reference distance associated with that three-dimensional model (e.g., the measured IPD). Since the actual IPD is known, the distances between any selected facial features are also determined.

[0030] In some of these embodiments, the imaging device 102 is a three-dimensional imaging device, such as a scanning laser-based imaging device. In one embodiment, the imaging device 102 is a “structural sensor” manufactured by Occipital, Inc. (USA) in San Francisco, California. In other embodiments, the imaging device 102 is a LIDAR-based sensor. In these embodiments, the imaging device 102 generates a three-dimensional point cloud of image data 103 to communicate with IBMDT101. In these embodiments, the three-dimensional imaging device 102 is a three-dimensional camera system, such as the TrueDepth® camera system manufactured by Apple Inc., Cupertino, California (USA), which is integrated into a portable computing device manufactured by Apple Inc., such as the iPhoneX®.

[0031] In one embodiment, a portable computing device such as a smartphone or tablet computer implements an IBMDT function as described with respect to Figure 1, based on a 3D image 103 collected from a 3D imaging device.

[0032] In another embodiment, the IBMD system 100 collects facial images and estimates facial dimensions at multiple points in time over a substantially long period. In some embodiments, facial images are collected and facial dimensions are estimated once or multiple times per day, week, or month over a yearly period. Figure 3 shows a diagram of memory 131. As shown in Figure 3, memory 131 stores facial records 151-156 containing facial images collected over several years, image acquisition times, and estimated facial dimensions.

[0033] In another further embodiment, the image-based medical diagnostic system communicates a warning (to, for example, a human user, a qualified medical professional, etc.) indicating abnormal facial dimensions, changes in facial dimensions, or both. In this way, the image-based medical diagnostic system prompts the medical professional to further investigate the human user's condition.

[0034] As shown in Figure 1, the processor 120 is configured to process the digital image generated by the imaging device 102 and issue a warning if one or more of the estimated facial dimensions or changes in facial dimensions exceed one or more predetermined thresholds. In one example, a warning is transmitted if the measured facial dimensions differ from the average facial dimensions (e.g., average LFH, average EE, etc.) by a predetermined threshold. In another embodiment, the processor 120 is configured to process the digital image generated by the imaging device 102 to detect anomalies that may indicate an imminent or ongoing medical health emergency. If such an anomaly is detected, the processor 120 communicates an alarm message 105 to an entity capable of addressing the medical problem (e.g., audibly via the speaker of a portable computing device, visually via the display of a portable computing device, via the internet, etc.). In one example, the processor 120 receives a digital image 103 and estimates an abnormal change in EE distance indicating a brain tumor. In this embodiment, the processor 120 generates an alarm message 105 that is communicated to a physician related to the medical care of a human user 107. For example, the IBMDT101 can send email, text, or phone messages indicating the nature of a medical emergency to doctors, family members, and others.

[0035] In some embodiments, the IBMDS100 identifies changes in the dimensions of a human user's features (e.g., facial dimensions) that indicate the onset or recurrence of a tumor after treatment. In some cases, hormonally active pituitary tumors (such as adenomas that secrete growth hormone) cause characteristic changes in the shape of a human face (such as Cushing's moon face or acromegaly). Typically, these tumors are very small (e.g., 3-5 mm in diameter) and located in the pituitary gland. For these reasons, these tumors are often difficult to detect and excise completely. Often, patients treated surgically see an initial reversal of facial changes. However, symptoms may also recur slowly. This can occur over a period of several years. In these cases, the IBMDS100 can identify early signs of tumor recurrence based on changes in facial dimensions measured according to the methods and systems described herein. In this way, patient anxiety induced by frequent follow-up visits and examinations, as well as the time and expense associated with these visits, are avoided.

[0036] In several other embodiments, the IBMDS100 identifies changes in the dimensions of human user features (e.g., facial dimensions) that indicate heart failure. Examples of features measured by the IBMDS100 that indicate an increased risk of heart failure include the dimensions of the diagonal fold above the earlobe, i.e., Frank's sign, xanthomas appearing on the eyelids, and a gray ring around the outer edge of the iris, also known as arcus senilis.

[0037] In another further embodiment, IBMDT101 receives a query 104 from an entity concerned (e.g., a human user 107, a medical professional related to the medical care of human user 107, etc.). In response to query 104, IBMDT101 communicates a representation of the human body's health status 105, including estimated facial dimensions, changes over time, or both, to the requesting entity.

[0038] In some embodiments, the IBMDS100 is implemented as part of a portable computing device such as a smartphone or tablet computing system. In some of these examples, the IBMDS100 is configured to automatically capture an image of a human user 107 when a desired part of the human user 107's body (e.g., the human user 107's face) is within the field of view of the portable computing system's imaging device 102. For example, when a human user 107 visually interacts with the portable computing device for any purpose (e.g., interacting with a software application, dialing a phone number, creating or reading a text message), the IBMDS100 is configured to collect an image of the human user 107's face, identify whether a reference feature (e.g., IPD) and any desired features (e.g., LFH, EE, etc.) are present in the collected image, and store the image containing one or more reference features and one or more desired features. In some other embodiments, the IBMDS100 prompts the human user 107 to locate the portable computing device so that a desired part of the human user 107's body is within the field of view of the portable computing system's imaging device. IBMDS100 also prompts the human user 107 to instruct the portable computing device to collect one or more images of the human user 107. In some embodiments, IBMDS100 may prompt the human user to collect one or more images periodically (e.g., hourly, daily). In other embodiments, IBMDS100 may prompt the human user to collect one or more images in response to a request from a query 104 from a medical professional.

[0039] In some embodiments, the IBMDS100 is implemented as part of a portable computing device that integrates three-dimensional (3D) display capabilities, such as those found in virtual reality Google. In these embodiments, an imaging device integrated with the 3D display captures an image of a human user's face and estimates facial features such as IPD and pupil size. In one of these embodiments, the image captured by the 3D display system is analyzed by the IBMDT101, which determines, based on the measured pupil size, whether the human user wearing the 3D display is experiencing an epileptic seizure.

[0040] Figure 4 shows a flowchart of method 200 for implementing the health monitoring and communication gateway functions described herein. In some embodiments, the IBMDS100 can operate according to method 200 shown in Figure 4. However, generally, the implementation of method 200 is not limited to the embodiments of the IBMDS100 described with reference to Figure 1. Since many other embodiments and operational examples are possible, this example and corresponding description are provided as examples only.

[0041] In block 201, multiple human facial images are captured by the imaging device.

[0042] In block 202, one or more dimensions of a human being associated with each of several images are estimated. These one or more dimensions are estimated at least in part based on known values ​​of identifiable reference dimensions in each of the several images.

[0043] Block 203 determines whether a dimension of 1 or more, or a change of 1 or more dimensions, exceeds a predetermined threshold indicating a potential medical emergency.

[0044] In block 204, if one or more dimensions, or one or more changes in dimensions, exceed a predetermined threshold, a warning message is communicated to an external entity.

[0045] In one or more exemplary embodiments, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted to a computer-readable medium as one or more instructions or codes on the medium. Computer-readable mediums include both computer storage media and communication media, including any media that facilitate the transfer of computer programs from one location to another. Storage media can be any available medium accessible by a general-purpose or special-purpose computer. Such computer-readable media may include, but are not limited to, RAM, ROM, EEPROM®, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures, and that can be accessed by a general-purpose or special-purpose computer or general-purpose or special-purpose processor. Any connection is also appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, discs and disks include compact discs (CDs), laserdiscs, optical discs, digital versatile discs (DVDs), floppy disks (registered trademark), and Blu-ray discs, where the disc typically reproduces data magnetically and the disc reproduces data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0046] While specific embodiments are described above for teaching purposes, the teachings in this patent document have general applicability and are not limited to the specific embodiments described above. Therefore, various modifications, adaptations, and combinations of features of the described embodiments can be implemented without departing from the scope of the invention as defined in the claims.

[0047] This patent application claims priority to U.S. Patent Application No. 16 / 278,071, filed on 16 February 2019, for the invention titled "Methods And Systems For Image Collection And Dimensional Analysis For Medical Diagnoses." U.S. Patent Application No. 16 / 278,071 subsequently claims priority under 35 U.S. SC § 119, U.S. Provisional Patent Application No. 62 / 632,326, filed on 19 February 2018, for the invention titled "Image Based Medical Diagnoses." Both applications are incorporated herein by whole reference.

Claims

1. A portable image-based medical diagnosis system, comprising: an imaging device configured to capture a plurality of images of a human user; a processor configured to estimate one or more dimensions of the human user associated with each of the plurality of images based at least in part on known values of a reference dimension identifiable in each of the plurality of images, the one or more dimensions including dimensions of diagonal wrinkles in the earlobes, and to determine whether the one or more dimensions or a change in the one or more dimensions exceeds a predetermined threshold indicating a potential medical emergency; a communication interface configured to communicate a warning message to an external entity when the dimension of the diagonal wrinkle in the earlobe or a change in the dimension of the diagonal wrinkle in the earlobe exceeds the predetermined threshold; wherein the reference dimension includes a stable facial dimension or a set of markings identifiable in each of the plurality of images; wherein the estimation of the one or more dimensions associated with each of the plurality of images is associated with the reference dimension and includes determining the number of pixels between a set of reference features located within each image and estimating a scaling factor based on a known distance between the set of reference features and the determined number of pixels; A portable image-based medical diagnosis system.

2. The portable image-based medical diagnosis system according to claim 1, further comprising a memory configured to store the acquisition time of each of the plurality of images and the one or more dimensions associated with each of the plurality of images.

3. The portable image-based medical diagnosis system according to claim 2, wherein the imaging device, the processor, and the memory are integrated as part of a portable computing system.

4. The portable image-based medical diagnosis system according to claim 2, wherein the processor and the memory are integrated as part of a network-based computing system separate from the imaging device.

5. The portable image-based medical diagnosis system according to claim 1, wherein the imaging device is a two-dimensional imaging device.

6. The portable image-based medical diagnosis system according to claim 1, wherein the imaging device is a three-dimensional imaging device.

7. The portable image-based medical diagnosis system according to claim 1, wherein the plurality of images are collected at multiple time points over a long period of time.

8. The portable image-based medical diagnosis system according to claim 7, wherein the long period exceeds one year.

9. The portable image-based medical diagnosis system according to claim 1, wherein the set of reference features includes the pupils of a human user.

10. A method for a processor to execute each step, comprising: imaging a plurality of images of a human; estimating one or more dimensions of the human associated with each of the plurality of images, at least partially based on known values of reference dimensions identifiable in each of the plurality of images, wherein the one or more dimensions include dimensions of diagonal wrinkles on the earlobes; determining whether the one or more dimensions or a change in the one or more dimensions exceeds a predetermined threshold indicating a potential medical emergency; communicating a warning message to an external entity when the dimension of the diagonal wrinkle on the earlobe or a change in the dimension of the diagonal wrinkle on the earlobe exceeds the predetermined threshold; wherein the reference dimensions include stable facial dimensions or a set of markings identifiable in each of the plurality of images; the estimation of the one or more dimensions associated with each of the plurality of images is associated with the reference dimensions and includes determining the number of pixels between a set of reference features located within each image and estimating a scaling factor based on the known distance between the set of reference features and the determined number of pixels.

11. The method according to claim 10, further comprising storing the acquisition time of each of the plurality of images and the one or more dimensions associated with each of the plurality of images.

12. The method according to claim 10, wherein each of the plurality of images is a 2D image.

13. The method according to claim 10, wherein each of the plurality of images is a 3D image.

14. The method according to claim 10, wherein the plurality of images are collected at a plurality of time points over a long period.

15. The method according to claim 14, wherein the long period exceeds one year.

16. The method according to claim 10, wherein the set of reference features includes the pupils of a human.

17. A portable image-based medical diagnosis system, comprising: an imaging device configured to image a plurality of images of a human user; A non-transitory computer-readable medium that records instructions to cause the processor to perform the following operations when read by the processor; The operations include estimating one or more dimensions of the human user associated with each of the plurality of images, at least partially based on known values of a reference dimension identifiable in each of the plurality of images, where the one or more dimensions include dimensions of diagonal wrinkles in the earlobes; Determining whether the one or more dimensions or a change in the one or more dimensions exceeds a predetermined threshold indicating a potential medical emergency; A communication interface configured to communicate a warning message to an external entity when the dimension of the diagonal wrinkle in the earlobe or a change in the dimension of the diagonal wrinkle in the earlobe exceeds the predetermined threshold; and The reference dimension includes a stable facial dimension or a set of markings identifiable in each of the plurality of images; The estimation of the one or more dimensions associated with each of the plurality of images is associated with the reference dimension and includes determining the number of pixels between a set of reference features located within each image and estimating a scaling factor based on a known distance between the set of reference features and the determined number of pixels; A portable image-based medical diagnostic system.

18. The portable image-based medical diagnostic system according to claim 17, wherein the imaging device, the processor, and the memory are integrated as part of a portable computing system.