Image capturing device, system and method

Abstract

A system may include an image capture device and a processor communicatively coupled with the image capture device. The image capture device may be configured to obtain one or more images. The processor may be configured to receive a first image obtained by the image capture device, determine whether the first object is in the first image, determine whether a size and a location of the first object in the first image satisfy a threshold if it is determined that the first object is in the first image, and determine whether the first object is in the first image. If at least one of the size and the position of the first object in the image does not meet the threshold, at least one corrective action to be taken is determined, and an instruction for taking the at least one corrective action is communicated to the user. A method and a machine-readable storage medium are also disclosed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 359,311 filed on July 8, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The disclosed devices, systems and methods relate to image capture and processing. More particularly, the disclosed devices, systems and methods relate to image capture and processing that can be used for patient care and / or telemedicine. Background Art

[0004] Consumer cameras and other image capture devices continue to improve and are now capable of obtaining near-cinema quality images and videos. Today, these cameras and image capture devices are integrated into or configured to work with cell phones, laptop computers, desktop computers, and other computing devices. While these consumer devices are typically used for social purposes (e.g., taking photos and / or videos of people, places, and things to share with others), they have the capability to be used for diagnostic purposes. Summary of the invention

[0005] In certain aspects, a system may include an image capture device and a processor communicatively coupled to the image capture device. The image capture device may be configured to obtain one or more images. The processor may be configured to receive a first image obtained by the image capture device, determine whether a first object is in the first image, if the first object is determined to be in the first image, determine whether a size and a position of the first object in the first image meet a threshold, if at least one of the size and the position of the first object in the image does not meet the threshold, determine at least one corrective action to be taken, and cause instructions for taking the at least one corrective action to be transmitted to a user.

[0006] In certain aspects, the first object may be at least one of an iris and a pupil of an eye.

[0007] In some aspects, the first object can be a contact lens.

[0008] In certain aspects, the at least one corrective action may include moving the image capture device in at least one direction relative to at least one of the user and the object.

[0009] In some aspects, the image capture device can be a camera located on a side of the mobile device opposite the side on which the screen is located.

[0010] In some aspects, at least one corrective action may include moving an eye of the user relative to the image capture device.

[0011] In certain aspects, the processor may be configured to determine whether the first object is in the first image using the first neural network.

[0012] In certain aspects, the processor may be configured to track the first object in at least one second image obtained from the image capture device using a second neural network different from the first neural network.

[0013] In certain aspects, the processor may be configured to store the first image in a non-transitory machine-readable storage medium communicatively coupled to the processor if the size and location of the first object in the first image satisfy a threshold.

[0014] In certain aspects, the processor may be configured to analyze a plurality of images received from an image capture device and determine an eye blink.

[0015] In some aspects, instructions for taking at least one corrective action may be communicated to the user via an audio output device communicatively coupled to the processor.

[0016] In certain aspects, instructions to take at least one corrective action may be communicated to the user tactilely.

[0017] In certain aspects, instructions for taking at least one corrective action may be presented to a user on a display communicatively coupled to the processor.

[0018] In certain aspects, the processor may be configured to provide an indication that the size and position of the first object in the image satisfies a threshold.

[0019] In some aspects, the indication may include a tactile indication.

[0020] In some aspects, the indication may include a visual indication.

[0021] In some aspects, the indication may include an audible indication.

[0022] In some aspects, the indication may include instructions to acquire a second image.

[0023] In some aspects, a method may include determining, by a first processing module, whether an object is present in a first image received from an image capture device; if the object is determined to be present in the first image, determining, by a second processing module, whether the object satisfies at least one predetermined criterion; if the object does not satisfy the at least one predetermined criterion, determining, by a third processing module, at least one adjustment to be made; and transmitting instructions to perform the at least one adjustment. The at least one predetermined criterion may include at least one of size and position in the image.

[0024] In some aspects, the first, second, and third processors can be the same processor.

[0025] In some aspects, the first, second, and third processors can be different processors.

[0026] In some aspects, instructions to perform at least one adjustment may be communicated to a user using an audio output device.

[0027] In some aspects, the instruction to perform at least one adjustment may be communicated to the user tactilely.

[0028] In some aspects, instructions to perform at least one adjustment may be displayed to the user.

[0029] In certain aspects, the method may include providing an indication that at least one predetermined criterion has been met.

[0030] In some aspects, the indication may include a tactile indication.

[0031] In some aspects, the indication may include a visual indication.

[0032] In some aspects, the indication may include an audible indication.

[0033] In some aspects, the indication may include instructions to acquire a second image.

[0034] In certain aspects, a machine-readable storage medium may store executable code. When the executable code is executed by a processor, it may cause the processor to perform a method. The method may include receiving a first image obtained by an image capture device; determining whether a first object is in the first image; if the first object is determined to be in the first image, determining whether a size and a position of the first object in the first image meet a threshold; if at least one of the size and the position of the first object in the image does not meet the threshold, determining at least one corrective action to be taken; and causing instructions for taking the at least one corrective action to be transmitted to a user.

[0035] In certain aspects, the first object may be at least one of an iris and a pupil of an eye.

[0036] In some aspects, the first object can be a contact lens.

[0037] In some aspects, at least one corrective action may include moving the image capture device in at least one direction relative to at least one of an eye of the user and the object.

[0038] In some aspects, the image capture device can be a camera located on a side of the mobile device opposite the side on which the screen is located.

[0039] In some aspects, at least one corrective action may include moving an eye of the user relative to the image capture device.

[0040] In some aspects, determining whether the first object is in the first image may include using a first neural network.

[0041] In certain aspects, the method may include tracking the first object in at least one second image obtained from the image capture device using a second neural network different from the first neural network.

[0042] In certain aspects, the method may include storing the first image in a non-transitory machine-readable storage medium communicatively coupled to the processor if the size and location of the first object in the first image satisfies a threshold.

[0043] In certain aspects, the method may include analyzing a plurality of images received from an image capture device and determining an eye blink.

[0044] In some aspects, instructions for taking at least one corrective action may be communicated to the user via an audio output device.

[0045] In certain aspects, instructions to take at least one corrective action may be communicated to the user tactilely.

[0046] In certain aspects, instructions for taking at least one corrective action may be presented to the user on a display.

[0047] In certain aspects, the method may include providing an indication that at least one of a size and a position of a first object in the image satisfies a threshold.

[0048] In some aspects, the indication may include a tactile indication.

[0049] In some aspects, the indication may include a visual indication.

[0050] In some aspects, the indication may include an audible indication.

[0051] In some aspects, the indication may include instructions to acquire a second image.

[0052] In certain aspects, a system may include an image capture device and a processor communicatively coupled to the image capture device. The image capture device may be configured to obtain one or more images. The processor may be configured to receive a first image obtained by the image capture device; determine whether a first object is in the first image; if the first object is determined to be in the first image, determine whether a size and a position of the first object in the first image meet a threshold; and transmit an indication to a user that the first image meets the threshold.

[0053] In certain aspects, the processor may be configured to determine at least one corrective action to take and communicate to a user instructions for taking the at least one corrective action if at least one of the size and position of the first object in the image does not satisfy a threshold.

[0054] In some aspects, a method may include: determining, by a first processing module, whether an object is present in a first image received from an image capture device; if the object is determined to be present in the first image, determining, by a second processing module, whether the object satisfies at least one predetermined criterion; and providing an indication to a user if the object satisfies at least one predetermined criterion. The at least one predetermined criterion may include at least one of size and position in the image.

[0055] In certain aspects, the method may include: determining, by a third processing module, at least one adjustment to be made if the object does not satisfy at least one predetermined criterion; and transmitting instructions to perform the at least one adjustment.

[0056] In some aspects, a machine-readable storage medium may store executable code. When the executable code is executed by a processor, it may cause the processor to perform a method. The method may include: receiving a first image obtained by an image capture device; determining whether a first object is in the first image; if it is determined that the first object is in the first image, determining whether a size and a position of the first object in the first image meet a threshold; and transmitting an indication that the first image meets the threshold to a user.

[0057] In certain aspects, the method may include determining at least one corrective action to take if at least one of a size and a position of the first object in the image does not satisfy a threshold, and communicating instructions to a user for taking the at least one corrective action. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a schematic diagram of an example of a system according to some embodiments;

[0059] Figure 2 According to some embodiments, Figure 1 A functional block diagram of an example of a computing device that can be used with the system shown in FIG.

[0060] Figure 3 is a functional block diagram of a processing pipeline that may be implemented by a computing device according to some embodiments;

[0061] Figure 3A and 3B illustrates examples of the front and back sides of a computing device, respectively, according to some embodiments;

[0062] Figure 4A and 4Bshows examples of a number of selections that a provider user and a patient user can make through an app and / or a web portal, respectively, according to some embodiments;

[0063] Figure 5 An example of a process flow that may be performed to obtain one or more images and / or videos according to some embodiments is shown;

[0064] Fig. 6A shows an example of an image or frame of an eye with an iris and pupil according to some embodiments;

[0065] Figure 6B shows an example of an image of an eye with pupil edges and contact lens edges highlighted in accordance with some embodiments; and

[0066] Figure 6C An example of an image of an eye is shown with the edge of a contact lens highlighted in accordance with some embodiments. DETAILED DESCRIPTION

[0067] These descriptions of the exemplary embodiments are intended to be read in conjunction with the accompanying drawings, which are to be considered a part of the entire written description.

[0068] The disclosed systems and methods are advantageous for facilitating remote medical examinations (e.g., telemedicine). In some embodiments, the medical examination includes a medical examination related to the eye. Examples of conditions that can be evaluated can include, but are not limited to, eyelid lesions, swelling, ptosis, conjunctival redness / swelling, dry eyes (e.g., capping gland and corneal evaluation), pterygium / conjunctival macula, contact placement, fit and irritation, glaucoma, and cataracts, to name just a few possibilities. Although the description herein relates to an app that images the eye to facilitate telemedicine, it should be understood that other types of medical or diagnostic evaluations can be performed, such as evaluations related to the ears, nose, throat, and other body parts.

[0069] In some embodiments, the disclosed systems and methods can utilize various user-available imaging devices, such as those provided on mobile devices (e.g., Apple iPhone, Apple iPad, Apple iPod, Samsung Galaxy), and can provide audio, visual, and / or tactile instructions to the user to facilitate the acquisition of one or more images that can be used for eyeglass examination. As described herein, the one or more images can include a single static image acquisition, or can include a series of images comprising a video.

[0070] As described in more detail below, the disclosed systems, devices, and methods may be implemented over a network (e.g., the Internet). The Internet is a global system of computer networks—a network of networks in which a user at one computer, terminal, or other device connected to the network can obtain information from any other computer, terminal, or device and communicate with users of other computers or devices. The most widely used portion of the Internet is the World Wide Web (often abbreviated "WWW" or just "the Web").

[0071] One feature of the Web is the use of hypertext, a method of cross-referencing. In most Web sites, certain words or phrases appear in text that is a different color from the surrounding text. The text is also usually underlined. Sometimes, there are hot spots, such as buttons, images, or "clickable" parts of images. Clicking on a hypertext or hot spot causes another web page to be downloaded through a protocol such as the Hypertext Transfer Protocol (HTTP). Using the Web, millions of pages of information can be accessed. Web "surfing" is done through a Web browser such as Apple Safari, Microsoft Edge, Mozilla Firefox, and Google Chrome. The appearance of a particular website may vary slightly depending on the specific browser used. Various versions of browsers have "plug-ins" that can provide animation, virtual reality, sound, and music. Interpreters (e.g., applets) can run in a browser.

[0072] Figure 1 A system is shown in which a plurality of wireless devices 100-1 and 100-2 (collectively referred to as "wireless devices 100" or "mobile devices 100") are connected to one or more computer system networks 50-1, 50-2 (collectively referred to as "computer system networks 50") and a telemedicine network 20 via a network 10. The network 10 may be a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), etc. In one embodiment, the network 10 is the Internet, and the mobile device 100 is online. "Online" may mean connecting to other devices or networks coupled to the network 10 or accessing source data or information from a location remote from other devices or networks coupled to the network 10.

[0073] The telemedicine network 20 may include one or more processing units 24 connected to one or more data storage units 26-1, 26-2 (collectively referred to as "data storage units 26"). The processing unit 24 may provide a front-end graphical user interface ("GUI"), such as a patient GUI or interface 28, a provider GUI or portal 30, and a back-end or management GUI or portal 32 to a remote computer 54 or a local computer 34. The GUI may take the form of, for example, a web page that is displayed to the remote computer 54 or one or more local computers 34 using a local browser program. It will be appreciated that the telemedicine network 20 may be implemented on one or more computers, servers, or other computing devices. For example, the physical training network 20 may include a server that is programmed or partitioned based on permitted access to data stored in the data storage unit 26. The front-end and back-end GUIs 28, 30, 32 may be portal pages that include various content retrieved from one or more data storage devices 26. As used herein, a "portal" is not limited to a general purpose Internet portal or search engine, such as GOOGLE, but also includes a GUI that may be of interest to a specific, limited audience and may provide parties with access to a variety of different types of related or unrelated information, links, and tools as described below. "Web page" and "website" may be used interchangeably herein.

[0074] The remote computer 54 can be part of the computer system network 50 and obtain access to the network 10 through an Internet service provider ("ISP") 52-1, 52-2 ("ISP 52"). The mobile device 100 can obtain access to the network 10 through a wireless cellular communication network, a WAN hotspot, or through a wired or wireless connection to a computer as understood by those skilled in the art. A patient or provider can use the remote computer 54 to obtain access to the telemedicine network 20.

[0075] In some embodiments, mobile or computing device 100 includes any mobile or computing device capable of transmitting and receiving wireless signals. Examples of mobile instruments include, but are not limited to, mobile or cellular phones, personal digital assistants ("PDAs"), laptop computers, tablet computers, music players, and e-readers, to name just a few possible devices.

[0076] Figure 2 is a block diagram of an example of an architecture of mobile device 100. Figure 2As shown, the mobile device 100 may include one or more processors, such as (one or more) processors 102. (One or more) processors 102 can be any central processing unit ("CPU"), microprocessor, microcontroller, or computing device or circuit for executing instructions. (One or more) processors can be connected to a communication infrastructure 104 (e.g., a communication bus, crossover bar, or network). Various software embodiments are described with respect to this exemplary mobile device 100. After reading this description, it will be apparent to a person of ordinary skill in the art how to implement the method using a mobile device 100 that may include other systems or architectures. A person of ordinary skill in the art will understand that the computer 34, 54 may have a similar and / or identical architecture as the mobile device 100. In other words, the computer 34, 54 may include Figure 2 Some, all, or additional functional components of the functional components of the mobile device 100 shown in FIG.

[0077] The mobile device 100 may include a display 106 that displays graphics, video, text, and other data received from the communication infrastructure 104 (or from a frame buffer not shown) to a user (e.g., a patient, a provider user, a backend user, or other user). Examples of such displays 106 include, but are not limited to, LCD screens, LED displays, OLED displays, touch screens (e.g., capacitive, resistive optical imaging, infrared), and plasma displays, to name just a few possible displays. The mobile device 100 may also include a main memory 108, such as a random access ("RAM") memory, and may also include an auxiliary memory 110. The auxiliary memory 110 may include more persistent memory, such as a hard disk drive ("HDD") 112 and / or a removable storage drive ("RSD") 114, which represents a tape drive, an optical drive, a solid-state drive ("SDD"), etc. In some embodiments, the removable storage drive 114 may read from and / or write to a removable storage unit ("RSU") 116 in a manner understood by those skilled in the art. Removable storage unit 116 may represent a magnetic tape, optical disk, etc., which is readable and writable by removable storage drive 114. Those skilled in the art will appreciate that removable storage unit 116 may include a tangible and non-transitory machine-readable storage medium having computer software and / or data stored therein.

[0078] In some embodiments, the secondary memory 110 may include other devices for allowing computer programs or other instructions to be loaded into the mobile device 100. Such devices may include, for example, a removable storage unit ("RSU") 118 and a corresponding interface ("RSI") 120. Examples of such units 118 and interfaces 120 may include removable memory chips (e.g., erasable programmable read-only memory ("EPROM"), programmable read-only memory ("PROM")), secure digital ("SD") cards and associated slots, and other removable storage units 118 and interfaces 120 that allow software and data to be transferred from the removable storage unit 118 to the mobile device 100.

[0079] The mobile device 100 may also include a speaker 122, an oscillator 123, a camera (or other image capture device) 124, a light emitting diode ("LED") 125, a microphone 126, an input device 128, and a global positioning system ("GPS") module 130. Examples of input device 128 include, but are not limited to, a keyboard, buttons, a trackball, or any other interface or device through which a user can input data. In some embodiments, input device 128 and display 106 are integrated into the same component or device. For example, display 106 and input device 128 may be a touch screen through which a user can enter data into the mobile device 100 using a finger, pen, and / or stylus.

[0080] The mobile device 100 may also include one or more communication interfaces 132 that allow software and data to be transferred between the mobile device 100 and external devices (e.g., another mobile device 100, a computer 34, 54, a telemedicine network 20, and other devices that may be locally or remotely connected to the mobile device 100). Examples of one or more communication interfaces 132 may include, but are not limited to, a modem, a network interface (e.g., an Ethernet card or a wireless card), a communication port, a Personal Computer Memory Card International Association (“PCMCIA”) slot and card, one or more Personal Component Interconnect (“PCI”) Express slots and cards, or any combination thereof. The one or more communication interfaces 132 may also include a wireless interface configured for short-range communication, such as near field communication (“NFC”), Bluetooth, or other interface for communicating via another wireless communication protocol. As briefly described above, it will be understood by those of ordinary skill in the art that the computer 34, 54 20 may include some or all of the components of the mobile device 100.

[0081] Software and data transmitted through one or more communication interfaces 132 may be in the form of signals, which may be electronic, electromagnetic, optical or other signals capable of being received by the communication interface 132. These signals may be provided to the communication interface 132 via a communication path or channel. The channel may be implemented using wire or cable, optical fiber, telephone line, cellular link, radio frequency ("RF") link or other communication channel. The terms "non-transitory computer program medium" and "non-transitory computer readable medium" refer to media such as removable storage units 116, 118 or a hard disk installed in the hard disk drive 112. These computer program products provide software for the mobile device 100. Computer programs (also referred to as "computer control logic") may be stored in the main memory 108 and / or the secondary memory 110. Computer programs may also be received through one or more communication interfaces 132. When executed by (one or more) processors 102, such computer programs enable the mobile device 100 to perform the methods discussed herein.

[0082] In some embodiments, where the method is implemented partially or entirely using software, the software may be stored in a computer program product and loaded into the mobile device 100 using the removable storage drive 114, the hard disk drive 112, and / or the communication interface 132. When executed by the processor(s) 102, the software may cause the processor(s) 102 to perform the functions of the method described herein. In some embodiments, the method may be implemented primarily in hardware, such as using hardware components, such as an application specific integrated circuit ("ASIC"). Those skilled in the art will understand the implementation of a hardware state machine in order to perform the functions described herein. In some embodiments, the method may be implemented using a combination of hardware and software.

[0083] Figure 3 One example of a processing system 300 is shown that may be implemented by one or more processors 102 of a mobile device 100. Figure 3 Described as being implemented by the mobile device 100 , it should be understood that the processing system 300 may be implemented by other computing devices (eg, the computer 54 ), or in a distributed manner, such as in combination with multiple processing devices and / or in combination with the telemedicine network 20 .

[0084] like Figure 3As shown, the computing device 100 may include a database or other data store 302 configured to store one or more selected images or image sequences (e.g., videos). The data store may be stored in the main memory 104 and / or the secondary memory 110 of the computing device, or the data store may be stored in a location remote from the computing device 100. In some embodiments, the data store 302 may also include one or more requirements, which may be predefined characteristics or parameters of one or more images or image sequences to be obtained. For example, the characteristics and / or parameters may define a threshold or range of lighting requirements for the image, a focus requirement, the size and / or position of the detected object and / or image, to name just a few possibilities. In some embodiments, the data store 302 may include a user identification and / or settings for an application running on a mobile device. As will be appreciated by one of ordinary skill in the art, one or more tokens may be stored on the computing device 100 for obtaining access to or authenticating with the telemedicine network 20.

[0085] The user profile module 304 may be provided to facilitate user registration. In some embodiments, the user profile module 304 may be configured to present one or more GUIs to the user on the display 106 to facilitate user registration. Registration may include obtaining the user's name, password, gender, age, and other identifying characteristics of the user. In some embodiments, the user profile module 304 may ask the user for screening purposes. For example, the questions presented to the user may ask the user what type of service (e.g., general eye examination, contact lens fitting, glaucoma testing, etc.) the user wishes to obtain and / or whether the user seeks services based on trauma, infection, or other reasons. In some embodiments, the user profile module 304 may include logic for determining whether the user is part of a relevant population receiving telemedicine. For example, a user of a certain age may be instructed to make a face-to-face appointment with a doctor for glaucoma screening, while the same user may be able to use the application for contact lens fitting. Those of ordinary skill in the art will appreciate that the user profile module 304 may be configured to perform various other functions to facilitate user registration and screening, including the additional functions described below.

[0086] An image acquisition module 306 may also be provided. The image acquisition module 306 may be configured to acquire an image or a series of images (e.g., a video) using the camera 124. As described herein, the image acquisition module 306 may be configured to cause the camera 124 to acquire one or more images automatically or in response to user input, such as, for example, a voice command, selection of a soft button displayed on a display, or pressing a button.

[0087] A separate detection module 308 may be provided, or it may be combined into the image acquisition module 306 (or vice versa). The detection module 308 may be configured to detect objects of interest, such as an eye, a pupil, the edge of a contact lens, or other objects that will be understood by those skilled in the art. For example, the system described herein may be used for a variety of different telemedicine applications including optometry. In such an embodiment, the detection module 308 may be configured to detect other objects of interest, such as the tongue, epiglottis, or other parts of the patient's anatomy.

[0088] In some embodiments, the detection module 308 can be implemented using a Yolov4-tiny ANN that is trained on a dataset of over 5,000 (e.g., 7,000) manually labeled images of various individuals to exclude bias. For example, as will be appreciated by one of ordinary skill in the art, individuals may vary based on eye color, race, gender, and / or age. In some embodiments, the resolution of an image from a camera or image capture device is converted from a first size or resolution to a second size or resolution. For example, an image from a camera may be resized to have a resolution of 416×416, which may be a trade-off between quality and processing speed. It should be understood that other resolutions may be used and that the image from the camera may not be resized prior to further processing. In some embodiments, the resizing of the image is performed by a separate module, such as an image cropping module 310, such as Figure 3 As shown. The image cropping module 310 can be configured to crop the obtained image to the region of interest and / or size determined by the detection module 308. The ANN can be converted from a first form to one or more second forms. For example, the ANN can be converted from a first form (e.g., a Darknet form) to a second or intermediate form (e.g., a Keras form) and then to a third or final form (e.g., CoreML) so that the ANN can run on the mobile device 100 and / or the computer 54.

[0089] A display module 312 may be provided to communicate with the detection module 308. As will be appreciated by those of ordinary skill in the art, the display module 312 may be configured to be connected to the display 106 of the computing device to present one or more captured images and / or GUIs to the user. In some embodiments, the display module 312 may be configured to display an overlay on the image displayed to the user. The overlay may take a variety of forms, such as a red line, a circle, or a polygon, and may be used to attract the user's attention to an area of ​​interest in the image obtained. It will be appreciated by those of ordinary skill in the art that the display module 312 may be configured to generate and / or display other overlays and / or GUIs to facilitate user interaction.

[0090] A gaze detection module 314 may be provided for detecting and / or tracking the gaze direction of the user. The gaze detection module 314 may be configured to analyze one or more images to detect the eye gaze direction. In some embodiments, the gaze detection module 314 may be configured to focus on a small area of ​​the eye, which may be the area detected by the eye detection module 308 described above, and generate a binary mask for the iris (or other features) of the eye.

[0091] The gaze detection module 314 can be implemented in a variety of ways. In some embodiments, the gaze detection module 314 can be implemented using a UNet neural network with a ReLU (rectified linear unit) activation function. However, it should be understood that the gaze detection module 314 can be implemented using other convolutional neural networks for imaging. In some embodiments, the gaze detection module 314 can be trained on hundreds of pairs (e.g., 400 pairs or more) of eye iris region of interest segmentation masks, along with various enhancements to provide approximately 1200 or more data points. In some embodiments, the resolution of an image from a camera or image capture device is converted from a first size or resolution to a second size or resolution. For example, an image from a camera can be resized to have a resolution of 192×128 and scaled to fit the region of interest. It should be understood that other resolutions can be used and that the image from the camera may not be resized prior to further processing. In some embodiments, the gaze detection module 314 can receive an image previously cropped by the image cropping module 310 (or from the detection module 308 if the cropping module 310 is included as part of the detection module 308), such as Figure 3 shown.

[0092] A blink detection module 316 may also be provided for analyzing one or more images to determine when the eyes are open or closed. In some embodiments, the blink detection module 316 is configured to detect a blink of the patient in one or more images. For example, the blink detection module 316 may be configured to trigger recording of a video (i.e., a series of images), wherein the start and end of the recording are triggered by a blink, which may be identified by a transition between two states: an open eye and a closed eye. The blink detection module may identify a blink when the eye transitions from an open eye state to a closed eye state.

[0093] In some embodiments, the blink detection module 316 can be configured to detect a blink when the position of the eye is about 1 cm to 30 cm (including 30 cm) from the image capture device. Those of ordinary skill in the art will understand that the distances and ranges provided are only for the purpose of illustrating examples and may be based on the resolution of the image capture device and / or the processing power of the processor.

[0094] In some embodiments, the blink detection module 316 can be implemented using a Yolov4-tiny ANN that is trained on a dataset of 3,000 or more manually labeled images of various individuals with open and / or closed eyes. For example, as will be appreciated by one of ordinary skill in the art, individuals may vary based on eye color, race, gender, and / or age. In some embodiments, the image resolution from a camera or image capture device is converted from a first size or resolution to a second size or resolution. For example, an image from a camera may be resized to have a resolution of 416×416, which may be a trade-off between image resolution and processing speed. It should be understood that other resolutions may be used, and the image from the camera may not be resized before further processing. The ANN may be converted from a first form to one or more second forms. For example, the ANN may be converted from a first form (e.g., a Darknet form) to a second or intermediate form (e.g., a Keras form), and then to a third or final form (e.g., CoreML) so that the ANN can run on the mobile device 100 and / or the computer 54.

[0095] A profile module 318 may also be provided and may be configured to track a region of interest of the eye from the profile view in order to obtain an image of the iris (or other feature) to facilitate detection of glaucoma. One of ordinary skill in the art will appreciate that glaucoma may result in fluid accumulation within the eye, which may affect the optic nerve at the back (e.g., posterior) of the eye. One method of detecting glaucoma is by viewing the profile of the patient's eye and determining whether the iris is pushed forward (e.g., in a forward direction) relative to a normal eye and / or whether fluid has accumulated in the front portion of the eye.

[0096] The contour tracking module 318 can be implemented in a variety of ways. In some embodiments, the contour tracking module 318 can be implemented by a Yolov-tiny ANN pre-trained on the ImageNet dataset, which is fine-tuned on the target dataset. For example, the model can be trained using approximately 2500 manually annotated region of interest contour images, with different scales and data augmentation. Here again, in order to eliminate or reduce bias, individuals may differ based on eye color, race, gender and / or age, as will be understood by those of ordinary skill in the art.

[0097] In some embodiments, the resolution of an image from a camera or image capture device is converted from a first size or resolution to a second size or resolution. For example, an image from a camera may be resized to have a resolution of 416×416, which may be a trade-off between quality and processing speed. It should be understood that other resolutions may be used and that the image from the camera may not be resized prior to further processing. The ANN may be converted from a first form to one or more second forms. For example, the ANN may be converted from a first form (e.g., a Darknet form) to a second or intermediate form (e.g., a Keras form) to a third or final form (e.g., CoreML) such that the ANN can run on the mobile device 100 and / or the computer 54.

[0098] In some embodiments, an illumination and focus detection module 320 may be provided. One of ordinary skill in the art will appreciate that the illumination and focus detection module 320 may be a separate module or may be integrated as part of another module (e.g., the image acquisition module 306), for example. In addition, the illumination and focus detection module 320 may be implemented separately, such that there may be a separate illumination detection and adjustment module and another separate focus detection and adjustment module. The illumination and focus detection module 320 may be configured to interface with the camera 12, for example, to determine and / or adjust one or more of an illumination (e.g., flash) setting and a focus setting.

[0099] A user direction or instruction module 322 may be provided to interface with one or more of the display module 132, the gaze detection module 314, the blink detection module 316, the eye contour module 318, and the illumination focus and detection module 320. The user detection module 322 may be configured with logic to receive one or more inputs from one or more other modules and determine one or more instructions to the user. For example, the user detection module 322 may be configured to receive outputs from one or more other modules and determine whether corrective action needs to be taken, thereby determining whether instructions should be provided to the user. In one possible example, if it is found that the detected object (e.g., iris, pupil, eye) is outside the desired position and / or the size is not appropriate (e.g., too large or too small), an instruction is generated and transmitted to the user to take corrective action. The instruction may be an auditory instruction transmitted to the user through the speaker 122, a message displayed on the display 106, and / or the mobile device may vibrate (e.g., through the oscillator 123).

[0100] In some embodiments, the auditory instructions may include commands or instructions for causing the user to move the camera in a specific direction to try to obtain an image in which an object (e.g., an eye, pupil, iris, or other desired object) is located in a suitable position in the image. For example, the command may instruct the user to move the camera left, right, up, and / or down to adjust the position of the user's face relative to the camera, thereby changing the position of the object in the image. In addition or alternatively, the command may instruct the user to move the camera closer, farther, and / or adjust the zoom setting to change the size of the object in the image. In some embodiments, the command may instruct the user to look in different directions. For example, the user may be instructed to "look up," "look down," "look left," and / or "look right." Other instructions may be provided to obtain a video for the purpose of evaluating the fit of contact lenses. For example, instructions may be provided to the user to look forward and blink one eye at a time (e.g., "Look forward and blink left eye. Continue to look forward and blink right eye."), then look up and blink one eye at a time (e.g., "Look up and blink left eye. Continue to look up and blink right eye.").

[0101] In some embodiments, instructions to move the camera relative to the user's eyes (or relative to the camera's eyes) can be displayed to the user, for example on a display. For example, the words "move camera left", "move camera right", "move camera up", "move camera down", "move camera away", and / or "move camera closer" can be presented to the user on the display. In some embodiments, the display can be the display of the mobile device 100, but the display can be an auxiliary display that is separate from the display of the mobile device but communicatively connected thereto. For example, the display can be an independent display that displays a mirror image of the display 106 of the user device. In some embodiments, a rear camera can be used, and the user can be in front of the mirror. Feedback can include displaying text on the display of the mobile device in an inverted manner so that the user can read the instructions / feedback in the mirror.

[0102] Additionally or alternatively, tactile feedback (e.g., instructions / commands) may be provided to the user. For example, one or a series of vibrations may be used to indicate where the camera should move relative to the user's eyes (or vice versa). In some embodiments, one or more short vibrations and / or one or more long vibrations may be used to indicate the direction the camera should move. In some embodiments, the tactile feedback is causing the mobile device to vibrate from a certain direction (e.g., from the left side of the mobile device) to indicate the direction the camera should move. Those of ordinary skill in the art will appreciate that various types of instructions or commands may be implemented and communicated to the user in various ways, including through a combination of auditory, visual, and / or tactile or haptic means.

[0103] Reference again Figure 3An image capture module 326 may be provided for selecting one or more images to present to an image quality assurance ("QA") module 328. In some embodiments, the image capture module 326 and the image QA module 328 may be combined into a single module, or they may be implemented as Figure 3 In some embodiments, the image capture module 326 is configured to receive one or more images from the user orientation module 324 and forward to the image QA module 328. For example, the user orientation module 324 may determine that no further instructions need to be provided to the user because the acquired image meets a set of preliminary criteria, such as suitable focus, lighting, and objects in the acquired image meet certain criteria, such as size, position, and / or gaze.

[0104] The image capture module 326 may then provide one or more of these images to the image QA module 328 to ultimately determine which one or more images should be stored in the session image and / or video database 330. In some embodiments, the image QA module 328 may select the "best" image from the plurality of images for storage. For example, an image that best meets one or more image requirements stored in the image / video requirements data store 302 may be selected from the plurality of images. The image QA module 328 may also provide feedback to one or more other modules (e.g., the image cropping module 310), such as Figure 3 shown.

[0105] Those of ordinary skill in the art will understand that Figure 3 The connections and general data flows shown are merely exemplary, and more and / or fewer connections and / or data flows may be implemented. For example, additional analysis modules may be provided, such as a module configured to provide a "digital stain" in an acquired image to alert a doctor or physician to a potential injury suffered by a patient and detected by the system. The digital stain may simulate fluorescein staining, which is used in many applications, including ophthalmic applications. For example, fluorescein is used as a diagnostic tool for diagnosing corneal abrasions, corneal ulcers, and herpetic corneal infections, as well as for rigid gas permeable contact lens fitting. Digital staining may be used in a similar manner by highlighting or identifying areas of interest or concern in an image.

[0106] In addition, as described above, the detection module can be configured to detect (and track) the edge of a contact lens (or other object of interest). In some embodiments, the edge of a contact lens can be tracked relative to one or more fixed landmarks (e.g., the boundary between the iris and sclera) to assist an eye care provider in determining the fit of the contact lens. For example, when a patient tries a new brand or prescription of contact lenses, the provider will typically check to ensure that the contact lenses fit the patient. Vertical movement of the lenses (e.g., after the patient blinks) can provide the provider with valuable information about whether the lenses fit the patient. However, because contact lenses are typically transparent, tracking the movement of contact lenses can be difficult. To assist the provider in performing such an assessment, in some embodiments, a separate module can be provided to detect such objects and / or provide an overlay or highlighting of the detected objects in the acquired image. For example, Fig. 6A One example of an image or frame of an eye 600 is shown having an iris 602 and a pupil 604. A contact lens 606 is shown disposed on the eye 600. Figure 6B One example of an image or frame of an eye 600 is shown, with the edge of an iris 602 identified by a first highlight 608 and the edge of a contact lens 606 identified by a second highlight 610 . Figure 6C An example of an image of an eye 600 is shown with the outline of a contact lens 606 highlighted 612 in different ways. One of ordinary skill in the art will appreciate that various ways of highlighting an object of interest may be used. In some embodiments, an annotated video of a patient's blink may be generated and stored for review by a provider to assist in assessing the fit of a contact lens. It should be appreciated that this processing may also be performed in real time and provided to a provider via a network connection to assess the fit of a contact lens in real time. In some embodiments, information about the movement of the edge of a contact lens may also be processed to automatically determine whether the eye is suitable for the user.

[0107] It should also be understood that Figure 3 The various modules of the processing pipeline shown can be executed in real time on the mobile device 100. As described herein, it should be understood that the processing can be implemented on devices with different processing capabilities, including high-performance desktop computers and / or in a distributed cloud environment. In the cloud computing example, the image captured by the image capture device or camera can be transmitted to one or more servers for processing, and the cloud and / or distributed processing system can provide one or more outputs from the processing to the mobile device 100 or computer 54.

[0108] Reference again Figure 1, the telemedicine network 20 can be accessed by patients and providers, so that patients and providers can obtain access to the content stored in the data storage unit 26 of the telemedicine network 20. In some embodiments, patients and / or providers can use computers 34, 54 and / or mobile devices 100 to obtain access to the content. For example, a patient can download an application (e.g., "app") from a network store (e.g., Google Play, Apple App Store and / or Microsoft Store) and install it on the mobile device 100 and / or computer 54. Once installed, the app can facilitate access to the telemedicine network 20. In addition or alternatively, a patient can use a web browser to obtain access to the telemedicine network through a web page or portal (e.g., a patient portal or interface 28). Access to the telemedicine network 20 may be protected, and the patient user may need to enter a username and password to obtain access. Those of ordinary skill in the art will understand that other ways of identifying patients can be used, such as through biometrics (e.g., facial recognition, fingerprints, eye scans). As described above, such identification and authentication can be performed by the user profile module 304. As described above, provider users (e.g., physicians, clinicians, and / or support staff) may also gain access to the telemedicine network 20 by downloading and installing an app or through a provider portal or interface 30. As described below, the content accessible to patients and / or providers may differ.

[0109] Mobile devices are typically configured with multiple cameras, one or more of which (e.g., camera 124-1) is located on the front of the device. Figure 3A As shown, one or more cameras (e.g., cameras 124-2, 124-3) are located on the back of the mobile device (i.e., the side opposite the screen of the mobile device), as shown Figure 3B The camera located on the back of the device (e.g. Figure 3B The cameras 124-2, 124-3) shown typically have a higher resolution than the camera or cameras located on the front of the mobile device.

[0110] The mobile device 100 and / or computer 54 can also be connected to a camera that is separate from the mobile device 100 or computer 54, such as a peripheral camera that is connected to the mobile device and / or computer 54 by a wireless or wired connection. For example, the camera can be a high-resolution camera that is connected to the mobile device 100 or computer 52 via a USB cable or via a network, such as a Wi-Fi network or a personal area network (such as Bluetooth). It may be desirable to use a camera with a higher available resolution in order to provide the most accurate and detailed eye image to the physician or other medical provider. However, given the reality that these high-resolution camera locations may be far from the display or screen, it may be difficult for the user to correctly position the camera relative to the eye. Each of the various modules of the app installed on the user device (such as the mobile device 100 and / or computer 54) or the web application accessible via the network is configured to provide guidance to the user to ensure that one or more images obtained using the camera are located at a desired location in the frame. In some embodiments, the desired location is the approximate center of the frame of the acquired image.

[0111] Reference again Figure 3A In some embodiments, one or more soft buttons 202, 204, 206, 208 may be displayed to the user on the display 106. In some embodiments, the soft generation and / or display of the soft buttons may be controlled by the user profile module 304 and / or the display module 312. The soft buttons may vary depending on whether the user is a patient or a provider. For example, a provider may be provided with one or more soft buttons that enable the provider to request that one or more patients perform one or more eye examinations using the mobile device 100 and / or the computer 54.

[0112] In some embodiments, a provider may be provided with the ability to request that a patient obtain images of the anterior segment of the eye and / or obtain one or more videos (e.g., a series of images) for use in evaluating the fit of one or more contact lenses. Obtaining one or more images of the anterior segment of the eye may include obtaining one or more images of one or more eyes, and one or more images of the eye when the eye is in different orientations and / or one or more images of the eye when the image capture device is located at different positions relative to the eye.

[0113] For example, Figure 4A An example of a plurality of selections 400 that a provider user can make through an app and / or web portal (eg, provider portal or interface 30) is shown. Figure 4AAs shown, the provider may be able to input a request for the patient-user to obtain one or more anterior images 402 or to obtain a video 430 of a contact lens fitting. If the provider user requests the patient-user to obtain one or more anterior images 402, the provider may be able to select the left eye 404 and / or the right eye 406. For each of the left eye and the right eye, the provider user may be able to select whether the patient-user should obtain one or more images 408, 412 with the eye wide open and / or obtain one or more images 410, 414 of the eye in different orientations (e.g., looking up 416, looking down 418, looking left 420, looking right 422) and / or use a camera oriented to obtain a profile image 424. If the provider user requests the patient-user to obtain one or more videos, the provider user may also be able to select whether the user should obtain a video of the left eye 432 and / or the right eye 434.

[0114] A person skilled in the art will understand that Figure 4A The optional selections shown in are merely exemplary, and fewer or more selections may be implemented. In addition, it should be understood that one or more GUIs may be presented to the provider user so that the provider user may make selections on a single screen or multiple screens presented on the display 106. In some embodiments, the provider user may make selections by selecting one or more soft buttons (e.g., soft buttons 202, 204, 206, 208) and / or by voice input.

[0115] The instructions entered by the provider user can be transmitted from the mobile device 100 and / or the computer 54 to the telemedicine network 20. In some embodiments, the provider instructions can be stored in a database or other data storage, such as one or more data storage units 26. The provider instructions can also be provided to the patient user. The patient user can be able to obtain the information provided by the provider user by logging into the app running on the mobile device 100 and / or the computer 54 and / or logging into the patient portal 28.

[0116] Once the patient user logs into the app and / or web interface, the patient user may be presented with a number of soft buttons, e.g. Figure 3AOne or more soft buttons 402, 404, 406, 408 shown in . In some embodiments, the soft buttons presented to the patient-user can be configured based on the instructions provided. For example, if the provider user only provides instructions to the patient to obtain a front image of the left eye in a wide-open orientation, a first set of soft buttons can be presented to the user on the display 106. If the provider user provides instructions to the patient to obtain a front image of the left eye and the right eye in a wide-open orientation and in all directions, a different set of soft buttons can be presented to the patient-user. In some embodiments, the soft buttons can be the same regardless of the provider instructions, but if the provider does not provide instructions to obtain certain images, certain soft buttons may be grayed out or invalid. Those of ordinary skill in the art will understand that the general architecture of the soft buttons and user interfaces and the manner of one or more GUIs may be configurable.

[0117] Figure 4B An example of a plurality of selections 450 available to a patient user is shown. Figure 4B As shown, the patient-user may be prompted to obtain an anterior image of the eye 452 and / or a video of the contact lens positioned in the eye 480. In some embodiments, the patient-user may be able to select the order in which to obtain one or more images and / or videos, such as by pressing one of the soft buttons or by providing voice input, as will be understood by one of ordinary skill in the art.

[0118] If the user selects the front image 452, the user may be given the option to obtain one or more images of the left eye 454 and / or the right eye 456. In some embodiments, the user may be instructed or prompted to obtain the images in some predetermined order (e.g., left first, then right). Similarly, the patient user may be able to select whether to obtain one or more images of the eyes in a wide open position 458, 462 and / or obtain one or more images of the eyes looking in one or more directions 462, 464 (e.g., looking up 466, looking down 468, looking to the left 470, looking to the right 472) or with the camera positioned to obtain a profile image 474. Alternatively, the user may be instructed or prompted to obtain the images in some predetermined order, which order may vary, as will be understood by one of ordinary skill in the art.

[0119] Figure 5 An example of a process flow that may be performed to obtain one or more images and / or videos according to some embodiments is shown. At block 502, the process begins. Figure 4A and 4BThe process may start in response to a user selecting one or more images and / or videos to be obtained, or the process may start automatically, where the user is instructed to obtain a specific image and / or video. In some embodiments, before instructing the user to obtain one or more images, the user may be prompted to register and / or screen, such as box 502A. Additionally or alternatively, a tutorial or other instructions may also be presented to the user at box 502B. In some embodiments, the processes performed at boxes 502, 502A, 502B may be implemented by the user profile module 304 and / or the image acquisition module 306.

[0120] At block 504, the camera 124 or other image capture device is initialized. The camera initialized may be a front-facing camera (e.g., Figure 3A ), a rear camera (e.g., Figure 3B In some embodiments, the image acquisition module 306 can initialize the image capture device.

[0121] At box 506, the user can be instructed to prepare to obtain one or more images. For example, the image acquisition module 306 and / or the user orientation module 324 can be configured to provide instructions to the user to position the image capture device relative to the user and / or position the user relative to the image capture device to obtain a first type of image. In one example, the user can be instructed to position the image capture device so that it is parallel to the frontal lobe or sagittal plane of the user, where the image capture device is aimed at the area of ​​interest (e.g., the face, head, eyes, or other body parts that may be of interest). The instructions can be provided to the user in an auditory, visual, and / or tactile manner, and can be based on a predetermined order of images and / or videos to be collected, such as an order determined according to instructions received from a provider or user input.

[0122] At box 508, a first set of one or more images is obtained, and an area or object of interest may be determined. In some embodiments, the first set of one or more images may be obtained in response to a user selection (e.g., a user pressing a physical button (e.g., input button 128) and / or a virtual or soft button). In some embodiments, the first set of one or more images may be obtained in response to a user speaking a command, which may be detected by microphone 128 in conjunction with an audio detection process performed by one or more processors 102, as will be understood by one of ordinary skill in the art. In some embodiments, the first set of one or more images may be obtained automatically after a predetermined amount of time has passed. For example, a countdown may be communicated to the user audibly, visually, and / or tactilely, and then the first set of one or more images may be acquired by the initialized camera. An audible countdown may be emitted from microphone 126, a visual countdown may be displayed on display 106, and a tactile countdown may be provided by oscillator 123. In some embodiments, the first set of one or more images may be obtained automatically without providing an indication to the user that the first set of one or more images is to be captured. For example, once the camera is initialized, the first set of one or more images may be acquired automatically by the camera.

[0123] In some embodiments, the image capture device 124 may obtain a series of images, which may be controlled by the image acquisition module 308, and the detection module 308 analyzes the series of images in real time. As described above, the detection module 308 may include an ANN configured to detect objects in the image, such as eyes or pupils. In addition, the illumination and focus detection module 320 may be configured to analyze the series of images and determine whether the illumination and / or focus of the obtained images are acceptable. Although these modules are described as operating on a series of images (e.g., two or more images), it should be understood that these modules can be configured to operate on a single image.

[0124] At block 510, a decision is made as to whether an object (e.g., eye(s), pupil(s), or other object to be imaged) is located in an image frame with adequate illumination and focus. In some embodiments, the analysis performed at blocks 508 and / or 510 also includes determining whether the location of the object is appropriate, including determining the location of the object in the frame. For example, the distance between an edge of the object and a reference point (e.g., an edge and / or center of the image) may be determined. If the distance is outside a predetermined range (e.g., too far from the center) or within a certain range (e.g., too close to the edge of the frame), it may be determined that the location of the object in the first set of one or more images is inappropriate. Additionally or alternatively, the size of the object may be determined and compared to a predetermined size or other criteria. In some embodiments, the size of the object may be based on the number of pixels in which the object is detected. One of ordinary skill in the art will appreciate that there may be other methods of determining the size of an object, such as determining the number of pixels in which the object is not detected. In some examples, the user may be prompted to provide a specific reference object (e.g., a credit card, a coin) of known size that is used to determine the size of other objects in the image. In some examples, assumptions may be made about the size of anatomical landmarks in the image. The assumption of the landmark (e.g., iris diameter) can be the average size of the landmark for all people, or the assumption can be based on the user (e.g., gender, age, race, or other attributes of the user). In some examples, a sensing camera (e.g., the "TrueDepth" camera available on the iPhone) can be used to measure the distance between the camera and the user. When combined with the focal length of the camera, this distance can be used to determine the conversion between image pixels and physical units (e.g., millimeters). Other examples of determining the size of an object are disclosed in U.S. Patent No. 9,532,709 and U.S. Patent No. 11,488,239, which are incorporated herein by reference in their entirety. The decision at box 510 can be made by one or more (e.g., a combination) of the detection module 308, the illumination and focus detection module 320, and the user direction module 324.

[0125] If it is determined that the first set of one or more images is not appropriate, for example, the object is not in the frame (or not in the expected area of ​​the frame), the object is too small or too large, the lighting is too low or too bright, the image is out of focus, the process can return to box 506, where the user direction module 324 and / or the lighting and focus correction module 320 can cause the user to take corrective action. For example, if the detected object is found to be outside the expected location, the size is not appropriate (e.g., too large or too small), the lighting is insufficient, and / or the focus is poor, an instruction is generated and transmitted to the user to take corrective action. The instruction can be an auditory instruction transmitted to the user through the speaker 122, a message displayed on the display 106, and / or the mobile device can vibrate (e.g., via the oscillator 123).

[0126] The auditory instructions may include commands or instructions for causing the user to move the camera in a particular direction in an attempt to obtain a second set of one or more images in which an object (e.g., an eye, pupil, iris, or other desired object) is located in a suitable position in the frame (or frames) of the (one or more) images. For example, the command may instruct the user to move the camera left, right, up, and / or down to adjust the position of the user's face relative to the camera, thereby changing the position of the object in the image. Additionally or alternatively, the command may instruct the user to move the camera closer, farther, and / or adjust the zoom setting to change the size of the object in the frame. In some embodiments, the command may instruct the user to look in different directions. For example, the user may be instructed to "look up," "look down," "look left," and / or "look right."

[0127] As described above, in some embodiments, instructions to move the camera relative to the user's eyes (or relative to the camera's eyes) may be displayed to the user, for example on a display. For example, the words "move camera left," "move camera right," "move camera up," "move camera down," "move camera away," and / or "move camera closer" may be presented to the user on a display. In some embodiments, the display may be the display of the mobile device 100, but the display may be an auxiliary display that is separate from the display of the mobile device but communicatively coupled thereto. For example, the display may be an independent display that displays a mirror image of the display 106 of the user device.

[0128] Additionally or alternatively, tactile feedback (e.g., instructions / commands) may be provided to the user. For example, one or a series of vibrations may be used to indicate the position to which the camera should move relative to the user's eyes (or vice versa). In some embodiments, one or more short vibrations and / or one or more long vibrations may be used to indicate the direction in which the camera should move. In some embodiments, the tactile feedback is to vibrate the mobile device from a certain direction (e.g., from the left side of the mobile device) to indicate the direction in which the camera should move. Those of ordinary skill in the art will appreciate that various types of instructions or commands may be implemented and communicated to the user in various ways, including through a combination of auditory, visual, and / or tactile or haptic means.

[0129] Then, the process can continue to blocks 508 and 510, where an additional set of one or more images can be obtained and analyzed as described above. In this way, the process can perform a continuous loop until an acceptable set of one or more images is obtained.

[0130] Once it is determined that a set of one or more images is suitable or acceptable, the process can move to block 512 where the acceptable set of one or more images is cropped to a first region of interest. In some embodiments, the image cropping module 310 can perform the cropping of the acceptable set of one or more images in real time (e.g., operating on the images as they are received from the camera, introducing additional delays beyond those due to processing). For example, the acceptable set of one or more images can be cropped to reduce the size of the overall image to a region that does not include extraneous features but includes a particular region of interest (e.g., an eye, pupil, or other feature or object).

[0131] At block 514, the user orientation module 320 may generate one or more instructions or prompts to cause the user to move the image capture device and / or the user's body to collect a second set of one or more images and / or videos. In some embodiments, the instructions provided to the user for obtaining a second set of one or more images and / or videos are based on the request for (one or more) images and / or (one or more) videos at block 502. The instructions generated by the user orientation module 320 at block 514 may be generated and / or transmitted to the user in a manner similar to the instructions described above with respect to block 506. In some embodiments, the user may already be properly positioned, and the instructions may request the user to remain still or maintain a certain position.

[0132] At box 516, the object is identified in the frame and the orientation is determined. For example, if the object of interest is an eye or pupil, the frame can be analyzed to ensure that the pupil or eye is within the frame and looking in the desired direction (e.g., looking straight, up, down, left, right, etc.). The analysis at box 516 can be performed by one or more of the detection module 308, the cropping module 310, and the gaze detection module 314. It should be understood that this process can be used to analyze other objects or body parts suitable for telemedicine imaging, including ears, noses, and throats, to name just a few possibilities. Thus, the object being detected can be a tongue, and the orientation can be an extended position (e.g., sticking out of the mouth). One of ordinary skill in the art will understand that other objects and orientations can be determined.

[0133] At box 518, a decision is made as to whether the object has been correctly detected and oriented correctly. In some embodiments, the determination is made based on the analysis performed by one or more of the detection module 308, the cropping module 310, and the gaze detection module 314 at box 516. If the object is not in the frame and / or the orientation is incorrect, the process can return to box 514, where the user direction module 320 can generate one or more instructions. As will be understood by those of ordinary skill in the art, the user direction module 320 can receive the output of the analysis performed by one or more of the detection module 308, the cropping module 310, and the gaze detection module 314 as input, and generate appropriate instructions to allow the user to correct the problem. The one or more instructions can be similar to the instructions described above with respect to box 506, and then the process can proceed to boxes 516 and 518. A repeated description of the instructions, analysis, and decisions made at boxes 506, 516, and 518 is not provided. In this way, the process can perform a continuous loop until the object is in the frame and in the desired orientation.

[0134] Once it is determined at block 518 that the object is in the frame and properly oriented, the process may move to block 520. At block 520, one or more images and / or videos are captured for further processing. For example, the image capture module 326 may select one or more images in which the object is in the frame and properly oriented for further analysis.

[0135] At block 522, the image QA module 328, which may also utilize one or more of the gaze detection module 314, the blink detection module 316, and the eye contour module 318, analyzes the one or more images selected at block 520. For example, the image QA module 328 may analyze the selected one or more images to determine whether the selected images meet a set of predetermined requirements at block 524. In some embodiments, the set of predetermined requirements may be stored in the image / video requirements database 302 described above. In some embodiments, the image QA module may consult the output of one or more of the gaze detection module 314, the blink detection module 316, and the eye contour module 318 to determine whether the selected images meet the requirements and identify one or more of the one or more selected images that best meet the requirements.

[0136] If it is determined at decision block 524 that none of the selected images meet the requirements, the process may return to block 514, where the user direction module 320 described above may generate one or more instructions. The one or more instructions may be similar to the instructions described above with respect to block 506, and the process may then continue to perform the processes described above with respect to blocks 516, 518, 520, 522. A repeated description of the instructions, analyses, and decisions made at blocks 506, 516, 518, 520, 522 is not provided. In this manner, the process may execute a continuous loop until one or more images or videos meet the requirements.

[0137] Once one or more images and / or videos are identified as satisfying the requirements, the one or more videos and / or images may be stored in the session image / video database 310 at box 526. As described above, the one or more images and / or videos may be stored locally on the mobile device 100 or computer 54. However, in some embodiments, the one or more images and / or videos may be stored in a remote computer-readable storage medium, such as a computer-readable storage medium located in the telemedicine network 20 (e.g., in the data storage unit 26). As will be appreciated by one of ordinary skill in the art, the image (or a copy of the image) may be transmitted from the mobile device 100 or computer 54 to the telemedicine network 20 for storage. The image may be and / or may be stored in an encrypted form in a database (e.g., a relational database) and / or other data store, such as a simple storage service (S3) object store. In some embodiments, the stored image may be associated with a unique identifier of the patient user to facilitate, for example, retrieval of the image by a physician or other practitioner.

[0138] At decision box 528, determine whether additional images and / or videos are needed. In some embodiments, this determination is made based on the type of eye examination selected by the user and / or prescribed by the physician, as described above. For example, a particular eye examination can be configured to collect multiple images of the eye, such as an image of the eye looking directly at the camera (e.g., looking straight ahead), an image of the eye looking up, an image of the eye looking down, an image of the eye looking to the left, and / or an image of the eye looking to the right. In some embodiments, an eye examination may require obtaining one or more images of both eyes of a person. In such an example, the program will track which images have been obtained and stored and which images have not yet been obtained. Therefore, if an image of an object in a particular orientation or position has not yet been obtained, control of the process will move to box 530.

[0139] At box 530, another set of parameters and instructions are generated to obtain another image and / or video. For example, if the first image successfully obtained and stored at box 526 is an image of the eyes looking straight ahead, and the user wants to obtain an image of the eyes looking in another direction, or when the camera is arranged to obtain a profile view of the eyes, the instructions can be loaded and provided to the user direction module, the user profile module, and / or the image acquisition module. The user can then be instructed to reposition the image capture device and / or the user to obtain the next desired image and / or video. In some embodiments, the process can move to box 512, where the image obtained by the image capture device is segmented to determine how the user should redirect the image capture device and / or the image capture device to obtain the next desired image or video. Then, the process can be performed through boxes 512-528 as described above.

[0140] If it is determined at decision block 528 that all images and / or videos have been acquired, the process can move to block 532. In some embodiments, the user can be presented with one or more images and / or videos captured by the user profile module 304 during the session. The user can be given the ability to submit replacement (e.g., recaptured) images of one or more acquired and stored images before submitting the images to the telemedicine network 20. If the user decides to recapture or replace the image, the process can return to block 504 or block 512. In some embodiments, the user can be prompted (e.g., using a GUI and / or soft button) to confirm submission to the telemedicine network 20. Once the user approves submission of the one or more images or videos, the process can move to block 534.

[0141] At box 534, one or more acquired images and / or videos may be stored in response to the user input at box 532. As described above, one or more acquired images and / or videos may be stored in a computer-readable storage medium located remotely from the mobile device 110 or the computer 54, such as a computer-readable storage medium located in the telemedicine network 20 (e.g., in the data storage unit 26). One of ordinary skill in the art will appreciate that the image (or a copy of the image) may be transmitted from the mobile device 100 or the computer 54 to the telemedicine network 20 for storage. The image may be and / or may be stored in an encrypted form in a database (e.g., a relational database) and / or other data store, such as a simple storage service (S3) object store. In some embodiments, the stored image may be associated with a unique identifier of the patient user to facilitate retrieval of the image by, for example, a physician or other practitioner.

[0142] The process ends at block 536. In some embodiments, once the images and / or video have been transmitted to the remote computer-readable storage medium (and receipt of such images and / or video has been acknowledged), the user may be automatically logged out of the session, or the user may be prompted to terminate the session, such as by pressing a soft button or by other means as will be appreciated by one skilled in the art.

[0143] The disclosed systems and methods may be embodied in the form of program code in a tangible medium, such as a floppy disk, CD-ROM, DVD-ROM, Blu-ray disc, hard drive, or any other machine-readable storage medium, wherein when the program code is loaded into and executed by a machine (e.g., a computer), the machine becomes an apparatus for practicing the method. The systems and methods may also be embodied in the form of program code, for example, whether stored in a storage medium, loaded into and / or executed by a machine, or transmitted over some transmission medium, such as by wire or cable, optical fiber, or by electromagnetic radiation, wherein when the program code is loaded into and executed by a machine (e.g., a computer), the machine becomes an apparatus for practicing the method. When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates similarly to a specific logic circuit.

[0144] Although the devices, systems, and methods have been described in the form of exemplary embodiments, they are not limited thereto. Rather, the appended claims should be interpreted broadly to include other variations and embodiments of the devices, systems, and methods that may be made by those skilled in the art without departing from the scope and range of equivalents of the devices, systems, and methods.

Claims

1. A system comprising: an image capture device configured to obtain one or more images; and a processor, the processor being communicatively coupled to the image capture device, the processor being configured to: receiving a first image obtained by the image capture device; determining whether a first object is in the first image; If it is determined that the first object is in the first image, determining whether the size and position of the first object in the first image meet a threshold; determining at least one corrective action to take if at least one of a size and a position of the first object in the image does not satisfy a threshold; as well as Instructions for taking the at least one corrective action are communicated to a user.

2. The system according to claim 1, wherein: The first object is at least one of an iris and a pupil of an eye.

3. The system according to claim 1, wherein: The first object is a contact lens.

4. The system according to claim 1, wherein: The at least one corrective action comprises moving the image capture device in at least one direction relative to at least one of a user and the object.

5. The system according to claim 4, wherein: The image capture device is a camera located on a side of the mobile device opposite to a side where the screen is located.

6. The system according to claim 1, wherein: The at least one corrective action comprises moving an eye of a user relative to the image capture device.

7. The system according to claim 1, wherein: The processor is configured to determine whether the first object is in the first image using a first neural network.

8. The system according to claim 7, wherein: The processor is configured to track the first object in at least one second image obtained from the image capture device using a second neural network different from the first neural network.

9. The system according to claim 1, wherein: The processor is configured to store the first image in a non-transitory machine-readable storage medium communicatively coupled to the processor if the size and position of the first object in the first image satisfy a threshold.

10. The system according to claim 1, wherein: The processor is configured to analyze a plurality of images received from the image capture device and determine an eye blink.

11. The system according to claim 1, wherein: Instructions for taking at least one corrective action are communicated to a user via an audio output device communicatively coupled to the processor.

12. The system of claim 1, wherein: Instructions for taking at least one corrective action are communicated to the user via tactile means.

13. The system of claim 1, wherein: Instructions for taking at least one corrective action are presented to a user on a display communicatively coupled to the processor.

14. The system of claim 1, wherein: The processor is configured to provide an indication that the size and position of the first object in the image satisfies a threshold.

15. The system of claim 14, wherein: The indication includes a tactile indication.

16. The system of claim 14, wherein: The indication comprises a visual indication.

17. The system of claim 14, wherein: The instructions include audible instructions.

18. The system of claim 14, wherein: The instructions include instructions to acquire a second image.

19. A method comprising: determining, by a first processing module, whether an object is present in a first image received from an image capture device; If it is determined that the object exists in the first image, determining by a second processing module whether the object satisfies at least one predetermined criterion; If the object does not meet the at least one predetermined criterion, determining, by a third processing module, at least one adjustment to be made; and transmitting an instruction to perform the at least one adjustment, The at least one predetermined criterion includes at least one of size and position in the image.

20. The method according to claim 19, wherein: The first processor, the second processor, and the third processor are the same processor.

21. The method according to claim 19, wherein: The first processor, the second processor, and the third processor are different processors.

22. The method according to claim 19, wherein: Instructions to perform the at least one adjustment are communicated to a user using an audio output device.

23. The method according to claim 19, wherein: Instructions to perform the at least one adjustment are communicated to the user via a tactile manner.

24. The method according to claim 19, wherein: Instructions are displayed to a user to perform the at least one adjustment.

25. The method of claim 19, further comprising providing an indication that the at least one predetermined criterion has been met.

26. The method according to claim 25, wherein: The indication comprises a tactile indication.

27. The method according to claim 25, wherein: The indication comprises a visual indication.

28. The method according to claim 25, wherein: The instructions include audible instructions.

29. The method according to claim 25, wherein: The instructions include instructions to acquire a second image.

30. A machine-readable storage medium storing executable code, which, when executed by a processor, causes the processor to perform a method, the method comprising: receiving a first image obtained by an image capture device; determining whether a first object is in the first image; If it is determined that the first object is in the first image, determining whether the size and position of the first object in the first image meet a threshold; determining at least one corrective action to take if at least one of a size and a position of the first object in the image does not satisfy a threshold; and Instructions for taking the at least one corrective action are communicated to a user.

31. The machine-readable storage medium of claim 30, wherein: The first object is at least one of an iris and a pupil of an eye.

32. The machine-readable storage medium of claim 30, wherein: The first object is a contact lens.

33. The machine-readable storage medium of claim 30, wherein: The at least one corrective action comprises moving the image capture device in at least one direction relative to at least one of an eye of a user and the object.

34. The machine-readable storage medium of claim 33, wherein: The image capture device is a camera located on a side of the mobile device opposite to a side where the screen is located.

35. The machine-readable storage medium of claim 30, wherein: The at least one corrective action comprises moving an eye of a user relative to the image capture device.

36. The machine-readable storage medium of claim 30, wherein: Determining whether the first object is in the first image includes using a first neural network.

37. The machine-readable storage medium of claim 36, wherein: The method includes tracking the first object in at least one second image obtained from the image capture device using a second neural network different from the first neural network.

38. The machine-readable storage medium of claim 30, wherein: The method includes storing the first image in a non-transitory machine-readable storage medium communicatively coupled to the processor if a size and a position of the first object in the first image satisfies a threshold.

39. The machine-readable storage medium of claim 30, wherein: The method includes analyzing a plurality of images received from the image capture device and determining an eye blink.

40. The machine-readable storage medium of claim 30, wherein: Instructions for taking at least one corrective action are communicated to the user via an audio output device.

41. The machine-readable storage medium of claim 30, wherein: Instructions for taking at least one corrective action are communicated to the user via tactile means.

42. The machine-readable storage medium of claim 30, wherein: Instructions are presented to the user on the display for taking at least one corrective action.

43. The machine-readable storage medium of claim 30, wherein: The method includes providing an indication that the at least one of a size and a position of the first object in the image satisfies a threshold.

44. The machine-readable storage medium of claim 43, wherein: The indication comprises a tactile indication.

45. The machine-readable storage medium of claim 43, wherein: The indication comprises a visual indication.

46. ​​The machine-readable storage medium of claim 43, wherein: The instructions include audible instructions.

47. The machine-readable storage medium of claim 43, wherein: The instructions include instructions to acquire a second image.

48. A system comprising: an image capture device configured to obtain one or more images; and a processor, the processor being communicatively coupled to the image capture device, the processor being configured to: receiving a first image obtained by the image capture device; determining whether a first object is in the first image; If it is determined that the first object is in the first image, determining whether the size and position of the first object in the first image meet a threshold; An indication that the first image satisfies a threshold is communicated to a user.

49. The system of claim 48, wherein: The processor is configured as follows: determining at least one corrective action to take if at least one of the size and position of the first object in the image does not satisfy a threshold; and Instructions for taking the at least one corrective action are communicated to a user.

50. A method comprising: determining, by a first processing module, whether an object is present in a first image received from an image capture device; If it is determined that the object exists in the first image, determining by a second processing module whether the object satisfies at least one predetermined criterion; providing an indication to a user if the object satisfies the at least one predetermined criterion, The at least one predetermined criterion includes at least one of size and position in the image.

51. The method of claim 50, further comprising: If the object does not meet the at least one predetermined criterion, determining, by a third processing module, at least one adjustment to be made; and An instruction is transmitted to perform the at least one adjustment.

52. A machine-readable storage medium storing executable code, which, when executed by a processor, causes the processor to perform a method, the method comprising: receiving a first image obtained by an image capture device; determining whether a first object is in the first image; If it is determined that the first object is in the first image, determining whether the size and position of the first object in the first image meet a threshold; and An indication that the first image satisfies the threshold is communicated to a user.

53. The machine-readable storage medium of claim 52, wherein: The method comprises: determining at least one corrective action to take if at least one of the size and position of the first object in the image does not satisfy a threshold; and Instructions for taking the at least one corrective action are communicated to a user.

Images