Guided self-capture of diagnostic images.

Abstract

An app for guided self-capture of images may display a graphical user interface (GUI) on a screen of a mobile computing device with a camera. The GUI may present instructions for capturing images of a particular area of ​​interest on a patient's body, including a graphical representation of the particular area of ​​the user's body posed in a predetermined position. The GUI may display the graphical representation in conjunction with a view that appears in the camera's lens. The app may then store one or more images captured by the camera and display one or more stored images in the GUI along with instructions for reviewing the stored images. In response to receiving a user input approving one or more of the stored images, the app may transmit the approved images to a system associated with the physician or clinician.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 306,953, filed February 4, 2022, the entire contents of which are incorporated herein by reference in their entirety.

[0002] This application relates generally to user interfaces for imaging applications. [Background technology]

[0003] Telemedicine systems typically rely on a combination of videoconferencing between the patient and clinicians and / or physicians, and submission of diagnostic information captured by monitoring devices or by the patient themselves (e.g., taking pictures of areas of interest on the patient's body that can be visually assessed). For example, the patient may need to capture pictures of areas of the body where symptoms of disease may be visible, where an implant has been inserted, where surgery has previously been performed, or where a procedure has been administered. The problem with having the patient capture the images themselves is that the patient may not be familiar with the type or quality of information that the clinician and / or physician needs to properly assess the patient's symptoms.

[0004] For example, a diagnostic photograph captured by a patient may be captured from a perspective that is too far away to capture sufficient detail or too close to capture the entire area of ​​interest. For example, a photograph captured by a patient may show the patient's entire face but not enough detail around the eye. In another example, a photograph captured by a patient focused on a wound may not have included adjacent areas of the body that may show swelling, redness, discoloration, or other signs of infection.

[0005] Furthermore, a patient's region of interest may be improperly positioned, blurred, obscured, or captured from a viewpoint or angle that does not meet predetermined criteria. For example, a patient may need to capture a photograph of an area of ​​the body where an implant has been inserted, where surgery has previously been performed, or where a procedure has been performed, where symptoms of disease may be visually presented. In such cases, the patient may need to move a limb to a particular position, pull apart a skin fold, rotate the eyes in a particular direction, or stretch a body part to provide better visibility of the region of interest. There is a need for a tool that can provide a patient with instructions and guidance to capture a usable diagnostic image of a region of interest on the patient's body for review (e.g., by a clinician or physician). Summary of the Invention

[0006] An exemplary embodiment of a user interface of a software application (e.g., a mobile device app executing on a mobile device including a camera) described herein may provide instructions to a user of the mobile device for taking a photograph that captures a usable diagnostic image of a region of interest of the user's body. The user interface of the mobile device app may provide textual, visual, and / or audio instructions and / or feedback regarding how to position the camera lens and / or how the user of the mobile device should pose their body to capture a usable diagnostic image of the region of interest of their body. The instructions regarding how to pose the user's body may include a graphical display showing how to pose the user's body to capture a usable diagnostic image of the region of interest of the user's body. The graphical display may include one or more still images or animations. The graphical display may be displayed along with an image of a view appearing through the lens of the camera.

[0007] In certain embodiments, the mobile device app may analyze images captured by the mobile device's camera (before, while, or after the photo is taken) to ensure that the image fits a predefined location profile. The analysis may be performed using techniques including image analysis to detect and identify relevant features, and then measure attributes of the identified relevant features. The measured attributes may then be evaluated against the predefined location profile.

[0008] The user interface may also provide feedback to the user if the camera position and / or body pose needs to be adjusted. Once the camera is in place and the user's body is posed, the user interface may enable the user to indicate that they are ready to take a photo (e.g., by tapping an interactive element such as a button displayed in the user interface, by speaking a voice command, or by making a gesture), at which point one or more photos may be taken. Alternatively, the mobile device app may analyze the image and detect the moment it meets one or more factors for determining that the image is within a usable diagnostic depiction of the region of interest, and automatically initiate photography of the image being captured.

[0009] The photographs may be taken using a "burst mode" that captures several photographs in rapid succession (e.g., 10 photographs per second) within a very short period of time to account for any movement while the photographs are being captured. The user interface may provide the user with an option to retake the photograph, and in some embodiments the user interface may analyze the captured images in the photograph and provide the user with a recommendation as to whether or not to retake the photograph. Such analysis may include, for example, analyzing photographic characteristics of the image, including brightness, sharpness, contrast, etc., compared to predetermined criteria for photographic characteristics.

[0010] The user interface may provide an option for the user to send photos to an image analyst (e.g., a physician or clinician). The user interface may provide the user with the ability to select one recipient from a list of image analysts to whom they can submit their photos, as well as the ability to review photos previously sent to a particular image analyst. The user interface may also provide alerts or notifications when the patient needs to take updated photos and / or submit new photos (e.g., periodically, to monitor progress).

[0011] In some embodiments, the mobile device app may be configured for use by an image analyst (e.g., a physician or clinician) to receive, review, annotate, categorize, store, and / or request resubmission of photographs by a target user (e.g., a patient). Such an embodiment of the mobile device app may also include an image capture workflow, whereby the mobile device may be used in a self-capture mode by the target user as described above (e.g., a nurse at the physician's office may hand the mobile device to the patient and instruct the patient how to take a photograph of the area of ​​interest). The user interface of the mobile device app may provide access to all photographs submitted by the target user, not just recently submitted photographs. If the user interface provides access to photographs of multiple target users, the image analyst may have the option to review photographs of one target user at a time, or to review and / or compare photographs from multiple users at a time. The user interface may also provide an option for the image analyst to request that the submitting target user retake a photograph. When reviewing one or more photographs, the physician may have the option to add annotations to one or more of the photographs, which may be later stored and retrieved along with the photograph. [Brief description of the drawings]

[0012] [Figure 1] 1 shows an overview of an exemplary system for guided self-capture of diagnostic photographs.

[0013] [Diagram 2] 1 shows a flowchart illustrating steps of a method for providing a user interface for guided self-capture of a diagnostic photograph.

[0014] [Diagram 3] 1 illustrates an example user interface displaying a graphical representation of an eye in a predetermined position depicted with an image of a user's eye captured by a camera.

[0015] [Figure 4] 1 illustrates an exemplary user interface that displays status messages and progress indicators during a picture taking process.

[0016] [Diagram 5] 1 illustrates an exemplary user interface for reviewing a photo taken by a user with options to retake the photo if necessary and to submit the photo.

[0017] [Figure 6] 1 illustrates an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] An exemplary embodiment of a user interface of a software application (e.g., a mobile device app executing on a mobile device including a camera) described herein may provide instructions to a user of a mobile device on how to capture a usable diagnostic image of a region of interest of the user's body. The user interface of the mobile device app may provide textual, visual, and / or audio instructions and / or feedback regarding how to position the camera lens and / or how a user of the mobile device should pose their body to capture a usable diagnostic image of the region of interest of their body. The instructions may instruct the user to position the camera lens at a particular distance, height, or angle relative to the region of interest. The instructions may instruct the user to utilize one or more features of the software associated with the camera to zoom in on the region of interest, identify a particular feature to autofocus, enable or disable the flash, increase or decrease the exposure time, change the aspect ratio, or set a timer to delay taking a picture. The instructions may instruct the user to pose and / or manipulate the body part (e.g., with the user's own hands or feet, or with the help of another person) to reveal, isolate, highlight, accentuate, or otherwise provide greater visibility of the body part.

[0019] The instructions on how to pose the user's body may include a graphical display showing how to pose the user's body to capture a usable diagnostic image of the region of interest of the user's body. The graphical display may include one or more still images or animations. The graphical display may include a stylized depiction of the region of interest in a desired pose. The graphical display may be displayed along with an image of the view that appears in the camera lens (e.g., the patient's face captured by the camera pointing toward the user of the mobile device).

[0020] A set of instructions on how to position the camera and how to pose the user's body may be displayed when the app is first opened. The instructions may be text, other visual content, audio content, or some combination thereof. The instructions may be presented before and / or during the image capture process. By presenting instructions step-by-step during the image capture process, the user interface may allow the user to seemingly execute each instruction. The app may display one instruction at a time, and only display the next instruction after ensuring that the current instruction has been fulfilled.

[0021] The user interface may also provide feedback to the user if the camera position and / or body pose needs to be adjusted. Once the camera is in position and the user's body is posed, the user interface may enable the user to indicate that they are ready to take a photo (e.g., by tapping an interactive element such as a button displayed in the user interface, by speaking a voice command, or by making a gesture), at which point one or more photos may be taken. Alternatively, the mobile device app may analyze the image of the view appearing in the camera lens to evaluate compliance with predefined criteria for usable diagnostic images of a particular region of interest. As the user poses their body to adjust their position, the mobile device app may detect the moment when the image complies with predefined criteria for usable diagnostic images of a region of interest, and automatically begin taking photos at that moment.

[0022] The predefined criteria for a usable diagnostic image may include predefined criteria for photographic characteristics of the image, a profile of predefined locations depicted in the usable diagnostic image, or other requirements for achieving a usable diagnostic image. The photographic characteristics of the image may include, for example, brightness, sharpness, contrast, focus, framing, etc. The profile of predefined locations may specify minimum thresholds, maximum thresholds, ratio-based thresholds, tolerance ranges, acceptable types, acceptable deviations, or any other suitable metrics for measurable attributes of relevant features of the image. The analysis may be performed using techniques including image analysis to detect and identify relevant features, and may include measuring attributes of identified relevant features. The measurable attributes may include the visibility of one of the identified features, the color of one of the identified features, the area of ​​one of the identified features, the distance between two of the identified features, the ratio of one of the identified features to another of the identified features, the shape of one of the identified features, the position of one of the identified features, the location of one of the identified features, the orientation of one of the identified features, the number of identified features of a particular type, an angle associated with one or more identified features, or a ratio of the measurable attributes.

[0023] The photos may be taken using a "burst mode" that captures several photos in succession within a very short period of time (e.g., 10 photos per second) to account for any movement while the photos are being captured. The user interface may provide the user with the option to retake any of the captured photos. In some embodiments, the app may analyze the captured images to evaluate whether the images conform to predefined criteria for usable diagnostic images of the area of ​​interest and provide a recommendation to the user as to whether the captured image should be retaken.

[0024] The user interface may provide an option for the user to send photos to an image analyst (e.g., a physician or clinician). The user interface may provide the user with the ability to select one recipient from a list of image analysts to whom they can submit their photos, as well as the ability to review photos previously sent to a particular image analyst. The user interface may also provide alerts or notifications when the patient needs to take updated photos and / or submit new photos (e.g., periodically, to monitor progress).

[0025] In some embodiments, the mobile device app may be configured for use by an image analyst (e.g., a physician or clinician) to receive, review, annotate, categorize, store, and / or request resubmission of photographs by a target user (e.g., a patient). Such an embodiment of the mobile device app may also include an image capture workflow, whereby the mobile device may be used in a self-capture mode by the target user as described above (e.g., a nurse at the physician's office may hand the mobile device to the patient and instruct the patient how to take a photograph of the area of ​​interest). The user interface of the mobile device app may provide access to all photographs submitted by the target user, not just recently submitted photographs. If the user interface provides access to photographs of multiple target users, the image analyst may have the option to review photographs of one target user at a time, or to review and / or compare photographs from multiple users at a time. The user interface may also provide an option for the image analyst to request that the submitting target user retake a photograph. When reviewing one or more photographs, the physician may have the option to add annotations to one or more of the photographs, which may be later stored and retrieved along with the photograph.

[0026] The aforementioned elements may be illustrated by applying them to the exemplary case of capturing a photograph of an eye. The associated set of instructions may include: 1) a reminder to ensure that the image capture illumination meets a predetermined standard or threshold; 2) step-by-step instructions showing or instructing how the patient should lift the upper eyelid and then move the pupil downward; 3) a graphical display of the eye posed in a predetermined position for medical evaluation; (4) instructions for capturing the image; and 5) any post-light capture steps. The instructions may be displayed on a series of screens with at least one instruction per page, each instruction may include visual and / or textual components, as well as annotations or visual, auditory, or tactile cues. Displaying the instructions as part of the photograph capture process may be done by first asking the patient to find a spot with good illumination, then waiting until sufficient illumination is detected before presenting the next instruction, i.e., a graphical display of the eye posed in a predetermined position and how the patient should pose the eye by lifting the upper eyelid and looking down and toward the nose. Once the app detects that the patient has taken a photo (or automatically takes a photo when it detects that the user has correctly posed the eye), the next instructions may be displayed and the process repeated with the final step. If displaying both advance instructions and abbreviated key components, the advance instructions may be given in the same manner as above, and the key components, such as how to lift the upper eyelid and how to use the visual guides, may be overlaid on the camera. The overlay may then be easily removed with some simple action, such as touching the screen.

[0027] In certain embodiments, the mobile device app may analyze the visual information from the camera (before or during capture) to ensure the captured photo meets certain requirements and provide feedback if the camera position and / or body pose needs to be adjusted. The app may include functionality to provide real-time feedback on different aspects of the photo capture process before the photo is taken, and perhaps even while the photo is being taken. Such feedback may include informing the patient when the patient is too close to the camera, too far away, when the body part is not aligned closely enough with the visual guide, when the body part is not properly aligned, when the body part is not properly posed, when something is occluding the body part, when the camera is out of focus, when lighting becomes insufficient, when the camera may be positioned at a poorly positioned viewpoint, etc. The feedback may include a description of the problem and how the patient can solve the problem. The generation of the feedback may be accomplished by any means, such as artificial intelligence, statistical, or machine learning computational models trained to detect when one of the aforementioned problems occurs.

[0028] The computational model may be an extension of the artificial intelligence model used for the instructions mentioned above, or it may be a completely separate model. Different feedback may be generated by different means. The way in which the feedback is made available to the patient may be done by a number of ways. One approach may be to display a notification message somewhere on the screen along with the associated feedback. However, the app may also incorporate some kind of haptic notification to alert the patient, since the patient may not always be able to easily notice a purely visual notification, such as when a photo of the eye is captured that may require the patient's pupils to be pointed away from the device screen. The app may also provide multiple forms of notification, such as by combining visual and haptic notifications.

[0029] Once the camera is in place and the patient's body is posed, any number of photos may be taken, with the option to retake as many times as necessary. The app may include several "modes" for capturing photos. One of the modes may be a "burst mode" that captures several photos in succession within a very short period of time (e.g., 10 photos per second) to reduce the effects of movement while the photos are being captured. This mode may be designed so that the app continues to take photos as long as the camera button is pressed, with the total number of photos captured being proportional to the length of time the button is pressed. A mode similar to that of burst mode may be a mode that continuously captures photos in rapid succession, but instead of requiring the button to be pressed the entire time, the first tap initiates the continuous capture process, and the process ends either with a second tap of the button ("double touch mode") or after a predefined time frame ("timed mode"). In these modes, the app may allow for various levels of customizability, such as allowing the patient to change the number of photos captured per second or the length of the predefined capture time. An alternative "countdown mode" may introduce a countdown when the button is first pressed, allowing the patient time to complete positioning before a photograph is captured.

[0030] These modes do not preclude the app also providing a simple "single photo mode" where only a single photo is captured upon release of the button, regardless of how long the button is pressed before it is released. The app may allow multiple modes to be active at once, as long as the modes are not inherently opposed to one another, such that a burst mode may be combined with a countdown mode, but a burst mode may not be able to operate in a single photo mode. In modes that may include some type of timed component, such as timed or countdown modes, the app may provide some type of notification when the relevant period has ended, since the patient may not be looking at the screen and therefore may not know when the process has ended. By notifying the patient, the app may help prevent the patient from moving before the timed process has ended (e.g., if the patient moves too early when trying to check the status of the capture process), but may also help prevent the patient from having to hold position for too long after the timed process has ended.

[0031] To ensure that the quality of the captured photos does not suddenly degrade midway through the capture process, especially in modes where photos may be captured in rapid succession over a period of time, the app may automatically terminate the capture process when it detects a substantial change in visual quality. The change that may trigger this early termination may be anything that may result in an unusable diagnostic photo, similar to causes that the app may detect before starting to capture photos, such as if the patient moves too far away from the camera, if the patient is no longer aligned closely enough with the visual guide, or if the camera suddenly goes out of focus. If this early termination is triggered, the app must either remember what has been captured up to that point, or simply discard everything and begin a new capture process from the beginning. The app may allow this feature to be customized by the user, perhaps including the option to disable the feature entirely or adjust the trigger threshold for termination.

[0032] The photo capture process and associated set of features may also be extended to support taking pictures of multiple body parts. In such an embodiment, once the process is completed for one body part, there may be an option to buffer the just-taken photo while completing the process again with another body part. Once the patient is satisfied with all the photos from all the body parts they wish to submit, they may complete the photo capture process and proceed to the next step.

[0033] In certain embodiments, the app may have a post-photo capture step reserved for reviewing the photos that have just been captured. The app itself may first analyze the photos after they are captured to perform a pre-review of the photos. This may be done in any number of ways, such as using the same artificial intelligence model that analyzed the visual information before and during the photo capture process. Any photos that the app's artificial intelligence model detects as potentially unsatisfactory (e.g., the patient was not posed properly, there was some obstruction, the photo was found to be blurry, etc.) may be marked in some way, such as with an exclamation point, to reflect that fact before being presented to the patient along with the other captured photos. The patient may manually review the photos taken.

[0034] During the review process, the app may include various features to assist the patient in the review. The app may display an example photo of the same body part just captured with the visual guide just used superimposed over the photo so that the patient may use the example photo as a standard for evaluating their own photo. If the patient is satisfied with the photo, the patient may approve the photo for submission, and if the patient is not satisfied with the photo, the patient may discard the photo. If the patient has taken multiple photos and is therefore currently reviewing multiple photos, the app may display one photo at a time and have the patient review all the photos one by one, or may display multiple photos at once so that the patient may review multiple photos at once. In the latter case, the app may need to ensure that each of the photos displayed is large enough for the patient to review them properly, which may limit the number of photos that can be displayed at any given time. The app may also take an approach where multiple photos are displayed, but then individual photos may be selected and enlarged for more detailed review. This review step may be scheduled to occur either after all photos have been captured, or after each photo or set of photos if using a mode such as burst mode.

[0035] If the patient captured videos of the body part, visual guides may be overlaid on each video for its entire duration to help the patient determine if they may have moved too far from the indicated position at any point during the video. Otherwise, the video review process may be largely similar to the photo review process. Because the patient may have taken both photos and videos, the app may allow the patient to review the photos and videos simultaneously or separate the photos and videos into separate review sessions.

[0036] In certain embodiments, the mobile device app may be configured for the patient to send photos to a doctor or clinician. In addition to the mobile app's user interface providing guidance on how to position the camera lens and / or how to orthodont the patient's body, the mobile device app may provide the patient with the ability to select which doctor and / or clinician to submit the captured photos to, the ability to review previously captured and / or previously submitted photos, and the ability to provide alerts or notifications when the patient needs to take a replacement photo and / or submit a new photo (e.g., periodically, to monitor progress). With respect to submitting the captured photos, after the patient reviews the captured photos and selects the doctor and / or clinician to whom they want to send the photos, the app may submit the photos immediately or may buffer the photos locally first until the patient has taken photos of other body parts that they wish or need to take before submitting all the photos in bulk. If the app buffers the images, the app may send a reminder to the patient, perhaps intermittently, with some notification that they need to confirm the submission of the captured photos. The reminder may be as a banner, as a pop-up, or in any other manner. The app may allow the patient to freely switch between having the app send photos immediately or buffer and send in bulk.

[0037] In certain embodiments, the mobile device app may be configured to receive, review, annotate, categorize, store, and / or request resubmission of patient photos, as well as the doctor or clinician who hands the mobile device to the patient to capture the photos as described above, or to capture the patient's own photos. The app may alert the doctor or clinician with a notification as soon as a new patient photo is submitted. The doctor or clinician may have access to all photos associated with the patient, not just the most recently submitted photo. The doctor or clinician may also have access to photos of multiple patients, or all patients for whom the app has data. While doctor / clinician access to photos may not be completely restricted, the app may provide options to implement various administrative restrictions, such as only allowing the doctor to view photos of patients for whom they are personally responsible.

[0038] When the physician reviews the photos, the physician may review photos of one particular patient at a time, or may review photos from multiple patients at a time. The app may allow the physician to add any amount of annotations to each photo, which may then be stored with the photo and retrieved later. The app may provide several tools to assist the physician in the review process. For example, the app may allow the physician to customize the annotations to make comments easier to identify later. The annotations may be color-coded to represent different types of manifestations associated with the photo, such as red for photos that may reveal very concerning manifestations that require the patient to schedule an office visit, yellow for photos that may suggest that there may be future concerns, and green for photos that may suggest that all is well for the patient. Other tools and features may include built-in markings to reflect different developments, such as a star to represent a particularly good quality photo, a check mark to represent a good or safe development of the patient's condition, or a house to represent the need for the patient to schedule an office visit.

[0039] FIG. 1 illustrates an exemplary system 100 for guided self-capture of diagnostic photographs. A mobile device application (e.g., patient app 110) runs on a mobile computing device (e.g., patient device 112) that supports the app. A camera 114 may be built into the patient device 112. If the patient device 112 has only a single camera, the camera is the one used to take the photograph, whether it is a front-facing camera 114 located above the display screen 116 of the patient device 112 or a rear-facing camera located on the opposite side of the display screen 116 of the patient device 112. If the patient device 112 has both a front-facing camera and a rear-facing camera, the patient app 110 may take the photograph using either camera. The patient app 110 may be able to interface with a third-party camera controlled by the patient device 112. The patient app 110 interfaces with a network 130 to transmit patient data to a clinical data system 150. The app's user interface may also be implemented in a number of ways, and examples of specific approaches are discussed in later sections.

[0040] The physician may interact with a mobile application 130 (e.g., physician app 120), which may also execute on a mobile device (e.g., physician device 122) supporting physician app 120. Physician app 120 may connect to network 130 to retrieve patient data and photos from clinical data system 150 and to transmit and store annotations or comments. The same communication protocol used to establish a connection between app 110 and network 130 may be used to establish a connection between physician app 130 and network 130. The user interface of physician app 120 may allow for reviewing, annotating, categorizing, ranking, scoring, storing, and / or requesting resubmission of patient photos.

[0041] The physician app 120 may also provide the ability to capture photographs using the camera 124 of the physician device 122 (shown in FIG. 1 as a forward-facing camera positioned adjacent to the display screen 126). For example, if a patient checks in with a clinician at a clinical facility to review the status of an area of ​​interest, as part of the visit, the clinician's assistant may hand the physician device 122 to the patient so that the patient can take photographs of the area of ​​interest while waiting for the clinician. Such photographs may be stored as part of the data collected during the visit. Additionally, if the patient needs assistance learning how to take photographs themselves, the clinician may provide guidance during the visit by having the patient attempt to capture a photograph of themselves using the physician device 122.

[0042] The clinical data system 150 may include a server 160 that responds to requests from the patient app 110 and the physician app 120, and a data storage 170 that stores the clinical data safely and securely. The server 160 may check, clean, format, and store new patient photos, as well as physician annotations and comments on the patient photos. The server 160 may clean the patient photos to remove any photo metadata that is not necessary, ensure that the received photos meet or pass predetermined quality standards, and / or verify that the received photos capture the patient's diagnostic features. The server may perform operations on the photos to effectively store the photos in the data storage, as required by the type and design of the data storage. The server may store the photos in the data storage device 170. This particular server design and associated methodology are merely exemplary, and particular embodiments may rearrange, add, or remove steps to suit their particular needs.

[0043] Upon receiving a request from the Doctor App 120 to review the patient's photos, the Server 160 may verify that the doctor is authorized to access the patient's data and immediately send a denial response if access is not possible. If not, the Server may retrieve the data from storage, reformat the data into human readable form if necessary from the way they are stored, add any necessary metadata, and send the photos to the Doctor App 120.

[0044] The server 160 may include some additional functionality for security and robustness purposes. For example, the server 160 may encrypt data when writing to the data store and decrypt any data when reading from the store. The encryption method used may be any known in the art, such as Advanced Encryption Standard (AES) or Rivest-Shamir-Adleman (RSA). All such designs of the server are purely exemplary, and specific embodiments may adjust the processes as necessary to suit their particular needs. Finally, a dedicated data storage component 170 may house all of the clinical data, including photos, annotations and comments, metadata, and other associated data.

[0045] In certain embodiments, the clinical data system 150 may be able to provide a selection of instruction types for guided self-capture of different regions of interest. A doctor or clinician may be able to select one of the instruction types for a particular patient. The patient app may interface with the clinical data system 150 to determine and / or download specific instructions, including specific graphical displays, to display for the patient. In some embodiments, the type of instruction may vary depending on the region of interest being captured as well as the particular symptoms, instructions, or other concerns that the doctor or clinician wants to focus on.

[0046] FIG. 2 illustrates an exemplary method 200 for guided self-capture of diagnostic photographs. The method may begin at step 210, where a mobile device application executing on a mobile device may display a graphical user interface (GUI) that provides instructions to the patient regarding the self-guided capture of available diagnostic images of the patient. The GUI may include a graphical display showing a particular area of ​​the patient's body posed in a predetermined position. As described further herein, the instructions may be presented in a number of different ways. At step 220, the mobile device application may display an image of the view appearing in the camera lens along with a graphical display of how to pose the body part displayed in relation to the view from the camera lens. As described further herein, the graphical display may be displayed in a number of different ways and may operate with the instructions to guide the user on how to pose the body part. At step 230, the mobile device application may capture one or more images of the view appearing in the camera lens and store the captured images in local data storage on the mobile computing device. As described further herein, the images may be captured via a number of different "modes." At step 240, the stored images may be displayed in the GUI along with various instructions for reviewing the images. As described further herein, various other elements may be displayed in the GUI at this point to assist the user in reviewing the images. After receiving user input approving one or more of the stored images at step 250, the approved images may be transmitted to a system associated with the physician or clinician for review and / or annotation by the physician or clinician.

[0047] Certain embodiments may repeat one or more steps of the method of FIG. 2, where appropriate. Although this disclosure describes and illustrates certain steps of the method of FIG. 2 as occurring in a particular order, this disclosure contemplates that any suitable steps of the method of FIG. 2 may be occurring in any suitable order. Additionally, while this disclosure describes and illustrates an exemplary method for guided self-capture of diagnostic photographs that includes certain steps of the method of FIG. 2, this disclosure contemplates any suitable method for guided self-capture of diagnostic photographs that includes any suitable steps, and may not include all, some, or any of the steps of the method of FIG. 2, where appropriate. Additionally, while this disclosure describes and illustrates certain components, devices, or systems that perform certain steps of the method of FIG. 2, this disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of FIG. 2.

[0048] FIG. 3 shows an example of what the application may show during pre-photo capture as it guides the patient towards a predefined position. In this case, the patient is using the Patient App to capture a photo of the eye using the camera 114, with the eye posed to expose the area of ​​the sclera where the implanted device was inserted. A message area 310 at the top of the screen may present instructions instructing the patient to pose the eye by using a finger to manipulate the skin of the upper eyelid while rotating the eye downward and towards the nose, and then to capture the photo when ready. As shown, key aspects of the instructions may be repeated at the point where the patient prepares to pose after the initial display of detailed instructions. The display area 320 may show the image being captured by the camera 114, along with an overlay of a graphical display 330. The graphical display 330, taking the form of a stylized eye, reflects a pose in which the wide-open eye is rolled downward and inward toward the nose (i.e., toward the inner canthus of the eye) to expose both a large area of ​​sclera between the upper eyelid and the iris of the eye, and a large area of ​​sclera between the iris and the outer canthus of the eye. A bottom camera button 340 with the text "Take Photo" may suggest that upon release of the button the current mode may take a single photo or a burst of photos, based on how the patient app 112 is configured. As shown in FIG. 3, the graphical representation 330 is shown as a stylized line drawing of an eye in a desired pose using dashed lines, although in certain embodiments, the graphical representation 330 may be depicted as an outline, a transparent image, or an animation.

[0049] FIG. 4 shows what the Patient App user interface looks like while actively photographing the patient's eye. As the photo capture process continues, the patient maintains the same pose, pulling back the upper eyelid and keeping the eye pointed downward and inward while holding the camera in a fixed position laterally and at a fixed distance to ensure persistent alignment with the graphical display 320. Based on the text in the message area 310 at the top of the screen and the text 410 above the photo capture button at the bottom, the current photo mode may be "burst mode" where multiple photos are taken continuously as long as the button is pressed. Thus, the button text instructing the patient to continue pressing the button may be to ensure a full burst of photos is captured. In some embodiments, a progress indicator 420 may be displayed while the burst of photos is captured and stored to let the user know when the process is complete. Similar embodiments may be used to capture short video clips.

[0050] FIG. 5 shows an example of the user interface of the Patient App in the review phase when the patient reviews the captured photograph. In the review phase, the patient is asked to consider several questions 510 before deciding whether to accept and submit the photograph by clicking button 520 or to click button 530 to retake the photograph. In this example, the app displays an example photograph 540 of the patient's eye pulling the upper eyelid while moving the pupil downward. The patient captured photograph 550 is placed next to the example photograph 540 to give the patient an easy standard for evaluating their own photograph. The questions 510 may provide further guidance on how the patient should evaluate the captured photograph. Although only one captured photograph is shown in the example, this does not necessarily mean that the patient captured only one photograph, the patient may have captured multiple photographs and the app is simply displaying only one photograph at a time for the patient to review. After reviewing their photograph, the patient has the option to either submit the photograph if they are satisfied with the result of the photograph or retake the photograph if they are not satisfied. Upon clicking the button 520, the patient app may upload the photo 550 immediately after clicking, or may wait until all photos have been reviewed before submitting them all together. Similarly, the retake option allows the patient to discard the photo or keep the original photo before retaking a new one, so the patient may compare the new photo with the original photo, perhaps before returning to the review page, and get the best of the two.

[0051] 6 illustrates an exemplary computer system 600. In certain embodiments, one or more computer systems 600 perform one or more steps of one or more methods described or illustrated herein. In certain embodiments, one or more computer systems 600 provide functionality described or illustrated herein. In certain embodiments, software running on one or more computer systems 600 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Certain embodiments include one or more portions of one or more computer systems 600. As used herein, references to a computer system may encompass computing devices, and vice versa, where appropriate. Moreover, references to a computer system may encompass one or more computer systems, where appropriate.

[0052] The present disclosure contemplates any suitable number of computer systems 600. The present disclosure contemplates computer system 600 taking any suitable physical form. By way of example and not limitation, computer system 600 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (e.g., a computer on module (COM) or system on module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 600 may include one or more computer systems 500, may be single or distributed, may span multiple locations, span multiple machines, span multiple data centers, or may reside in a cloud that may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example, and not by way of limitation, one or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. One or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein at different times or in different locations, where appropriate.

[0053] In a particular embodiment, computer system 600 includes a processor 602, memory 604, storage 606, an input / output (I / O) interface 608, a communications interface 610, and a bus 612. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0054] In particular embodiments, the processor 602 includes hardware for executing instructions, such as those that make up a computer program. By way of example and not by way of limitation, to execute instructions, the processor 602 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 604, or storage 606, decode and execute those instructions, and then write one or more results to an internal register, an internal cache, memory 604, or storage 606. In particular embodiments, the processor 602 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates the processor 602 including any suitable number of any suitable internal caches, where appropriate. By way of example and not by way of limitation, the processor 602 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in an instruction cache may be copies of instructions in memory 604 or storage 606, and the instruction cache may speed up retrieval of those instructions by the processor 602. The data in the data cache may be a copy of data in memory 604 or storage 606 on which instructions executing in the processor 602 operate, results of previous instructions executed in the processor 602 for access by subsequent instructions executing in the processor 602 or for writing to memory 604 or storage 606, or other suitable data. The data cache may speed up read or write operations by the processor 602. The TLB may speed up virtual-address translation for the processor 602. In particular embodiments, the processor 602 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates a processor 602 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, the processor 602 may include one or more arithmetic logic units (ALUs), be a multi-core processor, or include one or more processors 602. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0055] In a particular embodiment, memory 604 includes a main memory for storing instructions for processor 602 to execute or data on which processor 602 operates. By way of example and not by way of limitation, computer system 600 may load instructions into memory 604 from storage 606 or another source (such as, for example, another computer system 600). Processor 602 may then load the instructions from memory 604 into an internal register or cache. To execute instructions, processor 602 may retrieve instructions from the internal register or cache and decode the instructions. During or after execution of instructions, processor 602 may write one or more results (which may be intermediate or final results) to an internal register or cache. Processor 602 may then write one or more of those results to memory 604. In particular embodiments, processor 602 executes only instructions in one or more internal registers or internal caches or in memory 604 (as opposed to storage 606 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 604 (as opposed to storage 606 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may connect processor 602 to memory 604. Bus 612 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor 602 and memory 604 to facilitate accesses to memory 604 requested by processor 602. In particular embodiments, memory 604 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported RAM or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory 604 may include one or more memories 604, where appropriate. Although this disclosure describes and illustrates a particular memory, this disclosure contemplates any suitable memory.

[0056] In particular embodiments, storage 606 includes mass storage for data or instructions. By way of example and not limitation, storage 606 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Storage 606 may include removable or non-removable (or fixed) media, where appropriate. Storage 606 may be internal or external to computer system 600, where appropriate. In particular embodiments, storage 606 is a non-volatile solid-state memory. In particular embodiments, storage 606 includes read-only memory (ROM). Where appropriate, this ROM may be a mask programmable ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. The present disclosure contemplates mass storage 606 taking any suitable physical form. Storage 606 may include one or more storage control units to facilitate communications between processor 602 and storage 606, as appropriate. Storage 606 may include one or more storages 606, as appropriate. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0057] In particular embodiments, I / O interface 608 includes hardware, software, or both that provide one or more interfaces for communication between computer system 600 and one or more I / O devices. Computer system 600 may include one or more of these I / O devices, as appropriate. One or more of these I / O devices may enable communication between a person and computer system 600. By way of example and not by way of limitation, the I / O devices may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet, a touch screen, a trackball, a video camera, another suitable I / O device, or a combination of two or more of these. The I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interfaces 608 for those I / O devices. As appropriate, I / O interface 608 may include one or more device or software drivers that enable processor 602 to drive one or more of these I / O devices. I / O interface 608 may include one or more I / O interfaces 608, where appropriate. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0058] In particular embodiments, communication interface 610 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication, etc.) between computer system 600 and one or more other computer systems 600 or one or more networks. By way of example and not by way of limitation, communication interface 610 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 610 for that network. By way of example and not by way of limitation, computer system 600 may communicate with an ad-hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 600 may communicate with a wireless PAN (WPAN) (e.g., a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Computer system 600 may include any suitable communications interface 610 for any of these networks, where appropriate. Communications interface 610 may include one or more communications interfaces 610, where appropriate. Although this disclosure describes and illustrates a particular communications interface, this disclosure contemplates any suitable communications interface.

[0059] In particular embodiments, bus 612 includes hardware, software, or both that connects components of computer system 600 to one another. By way of example, and not limitation, bus 612 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus, or a combination of two or more of these. Bus 612 may include one or more buses 612, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0060] As used herein, one or more computer-readable non-transitory storage media may include, where appropriate, one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

[0061] In certain embodiments, the particular type of storage used for the data storage component of the system may be any of many types of systems known in the art, including, but not limited to, traditional relational databases such as PostgreSQL or MySQL, NoSQL systems such as MONGODB or BerkeleyDB, distributed systems such as GOOGLE CLOUD SPANNER or AZURE COSMOS DB, or data lakehouse-based systems such as DATABRICKS or SNOWFLAKE, among others.

[0062] For purposes of this description herein, any reference to taking photographs in connection with the present system is intended to apply equally to recording video. Given the current state of the art in photography and videography, it will be relatively easy for one skilled in the art to transfer system design and functionality from photography to videography or vice versa. Where this is not the case, the description provides clear consideration of how the design and / or functionality may change when applied to photography and videography.

[0063] For purposes of this description herein, references to specific features related to system design or functionality are intended only as examples and in no way as limitations. In this specification, "or" is inclusive and not exclusive, unless expressly indicated otherwise or otherwise indicated by the context. Thus, in this specification, "A or B" means "A, B, or both" unless expressly indicated otherwise or otherwise indicated by the context. Moreover, "and" is both jointly and severally, unless expressly indicated otherwise or otherwise indicated by the context. Thus, in this specification, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or otherwise indicated by the context.

[0064] The scope of the present disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or shown herein that would be understood by a person skilled in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or shown herein. Moreover, although the present disclosure describes and shows each embodiment herein as including certain components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or shown anywhere herein that would be understood by a person skilled in the art. Furthermore, references in the appended claims to an apparatus or system, or an apparatus or system component, that is adapted, arranged, capable, configured, enabled, operable, or operative to perform a particular function encompass that apparatus, system, component, or its particular function, so long as it is so adapted, arranged, capable, configured, enabled, operable, or operative, regardless of whether the apparatus, system, component, or its particular function is activated, turned on, or unlocked. Further, while this disclosure describes or illustrates certain embodiments as providing certain advantages, a particular embodiment may provide none, some, or all of these advantages.

Claims

1. One or more computer-readable non-transitory storage media embodying software for guided self-capture of one or more images, the code comprising: displaying a graphical user interface (GUI) on a screen of the mobile computing device, the GUI presenting instructions for capturing an image of a particular area of ​​the user's body posed in a predetermined position, the instructions including a graphical representation of the particular area of ​​the user's body posed in the predetermined position; upon detecting that a camera associated with the mobile computing device is available, displaying in the GUI an image of a view as it appears in the lens of the camera, the graphical representation being displayed in conjunction with the image of the view; capturing one or more images of the view, the captured images being stored on the mobile computing device; displaying one or more of the stored images in the GUI along with instructions for reviewing the stored images; in response to receiving user input approving one or more of the stored images, transmitting the approved one or more of the images via a network connection to a system associated with an image analyst; One or more computer-readable non-transitory storage media operable to:

2. 2. The computer-readable, non-transitory storage medium of claim 1, wherein the instructions include at least one instruction describing how to manipulate the particular area of ​​the user's body to achieve the predetermined position, and the graphical representation includes a still image depicting the particular area of ​​the user's body after posing in the predetermined position.

3. 3. The computer-readable, non-transitory storage medium of claim 2, wherein the particular area of ​​the user's body is the user's eye, and the at least one instruction describing how to manipulate the particular area of ​​the eye includes instructions to lift an eyelid of the eye using a finger, instructions to hold the eyelid in a lifted position, and instructions to rotate the eye downward toward the user's nose.

4. 2. The computer-readable, non-transitory storage medium of claim 1, wherein the particular area of ​​the user's body is the user's eye, and the predetermined location is designed to expose a device implanted within the particular area of ​​the sclera of the eye.

5. 5. The computer-readable non-transitory storage medium of claim 4, wherein the graphical representation of the eye posed in the predetermined position includes a stylized image of the eye posed to expose a large area of ​​the sclera between the upper eyelid of the eye and the iris of the eye, and a large area of ​​the sclera between the iris and the lateral canthus of the eye, the upper eyelid being depicted as being lifted high and the iris being depicted as being rotated downward and inward toward the medial canthus of the eye.

6. The code is analyzing the images of the views to assess compliance with predefined standards for usable diagnostic images of particular regions of interest, the predefined standards for usable diagnostic images including predefined standards for image photographic characteristics or profiles of predefined locations depicted in the usable diagnostic images; The computer-readable non-transitory storage medium of claim 1 , further operable to:

7. the images of the view are analyzed as they appear in the lens of the camera; automatically triggering the mobile device to activate the camera to capture the one or more images upon determining that the view appearing through the lens of the camera meets the predetermined specifications for a usable diagnostic image of the particular region of interest. The computer-readable non-transitory storage medium of claim 1 .

8. The image of the view is stored and then analyzed, and the code comprises: determining that an unsatisfactory one of the stored images does not meet the predetermined standard for a usable diagnostic image of the particular region of interest; displaying in the GUI the unsatisfactory images and interactive elements for capturing a replacement image; The computer-readable non-transitory storage medium of claim 1 , further operable to:

9. The computer-readable non-transitory storage medium of claim 6 , wherein the predetermined specifications for photographic characteristics of an image include brightness, sharpness, or contrast.

10. The code executable to analyze the images of the views to evaluate compliance with a predetermined standard for usable diagnostic images of the particular region of interest includes: performing image analysis to identify one or more characteristics of the particular area of ​​the user's body; Evaluating one or more measurable attributes of one or more of the identified features to determine conformance of the given location to the profile.

7. The computer-readable non-transitory storage medium of claim 6, comprising code executable to:

11. 8. The computer-readable non-transitory storage medium of claim 7, wherein the measurable attributes include a visibility of one of the identified features, a color of one of the identified features, an area of ​​one of the identified features, a distance between two of the identified features, a ratio of one of the identified features to another of the identified features, a shape of one of the identified features, a position of one of the identified features, a location of one of the identified features, an orientation of one of the identified features, a number of the identified features of a particular type, an angle associated with one or more identified features, or a ratio of measurable attributes.

12. 7. The computer-readable non-transitory storage medium of claim 6, wherein the profile of a given location depicted in a usable diagnostic image includes a minimum threshold, a maximum threshold, a ratio-based threshold, a tolerance range, an acceptable type, or an acceptable degree of deviation in the measurable attribute.

13. The computer-readable, non-transitory storage medium of claim 1 , wherein the one or more stored images comprise a set of images capturing the view appearing to the lens of the camera in rapid succession over a short period of time.

14. The code is displaying an interactive element in the GUI for taking a photograph; detecting user interaction with the interactive element via a touch screen of the mobile computing device; in response to detecting the user interaction, storing the one or more images on the mobile computing device; The computer-readable non-transitory storage medium of claim 1 , further operable to:

15. The code is Detecting a voice command by a microphone of the mobile computing device; in response to detecting the voice command, causing the mobile computing device to store the one or more images. The computer-readable non-transitory storage medium of claim 1 , further operable to:

16. The code is Detecting a gesture by the camera of the mobile computing device; in response to detecting the gesture, causing the mobile computing device to store the one or more images. The computer-readable non-transitory storage medium of claim 1 , further operable to:

17. The computer-readable, non-transitory storage medium of claim 1 , wherein each of the one or more images captured by the camera comprises a still image or a video.

18. 16. The computer-readable non-transitory storage medium of claim 15, wherein upon receiving an indication that the one or more images should be captured by the camera, a progress indicator is displayed during a period of time that elapses while the one or more images are captured and stored.

19. Mobile computing devices displaying a graphical user interface (GUI) on a screen of the mobile computing device, the GUI presenting instructions for capturing an image of a particular area of ​​the user's body posed in a predetermined position, the instructions including a graphical representation of the particular area of ​​the user's body posed in the predetermined position; upon detecting that a camera associated with the mobile computing device is available, displaying a view in the GUI as it would appear in the lens of the camera, the graphical representation being displayed in conjunction with the view; capturing one or more images of the view, the captured images being stored on the mobile computing device; displaying, in the GUI, one or more of the stored images along with instructions for reviewing the stored images; in response to receiving a user input approving one or more of the stored images, transmitting the approved one or more images via a network connection to a system associated with an image analyst; A method comprising:

20. 1. A system comprising one or more processors and a memory coupled to the processors containing code executable by the processors, the processor, when executing the code, performing: displaying a graphical user interface (GUI) on a screen of the mobile computing device, the GUI presenting instructions for capturing an image of a particular area of ​​the user's body posed in a predetermined position, the instructions including a graphical representation of the particular area of ​​the user's body posed in the predetermined position; upon detecting that a camera associated with the mobile computing device is available, displaying in the GUI a view as it would appear in the lens of the camera, the graphical representation being displayed in conjunction with the view; capturing one or more images of the view, the captured images being stored on the mobile computing device; displaying one or more of the stored images in the GUI along with instructions for reviewing the stored images; in response to receiving a user input approving one or more of the stored images, transmitting the approved one or more images via a network connection to a system associated with an image analyst; The system is operable to

Images