Sharing medical data using blockchain

Abstract

A system / method for transacting medical data between different stakeholders (remote users or subscribers) uses a blockchain computer network to record all data transactions and ensure the privacy of data owners.

Description

[Technical field]

[0001] The present invention relates generally to the use of blockchain for medical applications. [Background technology]

[0002] Data plays a key role in developing artificial intelligence solutions and analytics, as well as building smart devices and services. Sharing data among physicians, researchers, and commercial entities can advance new discoveries and provide better diagnoses and treatments in medicine. Specifically, sharing data among the ophthalmology community has the potential to enable better eye care for patients.

[0003] Leveraging patient data to improve clinical outcomes presents two major challenges: 1) technical access to data across various repositories, applications, and IT / security frameworks, and 2) enhanced global patient (consumer) privacy protections through an ever-changing, increasingly influential, and highly heterogeneous regulatory landscape.

[0004] There are obstacles that affect access to patient medical data, such as ophthalmology data, orthopedic data, cardiology data, dermatology data, radiology data, etc. For example, patient medical data is typically stored in electronic medical record systems that are located at each hospital, imaging center, or clinic where the data was originally acquired (e.g., the medical institution that collected the medical data) or, more recently, in some cases at virtual machine (VM) hosting sites (the cloud). Due to technical complexities in storage infrastructure, software / databases, local security infrastructure (e.g., firewalls, authentication, encryption, etc.), legal privacy issues, data size, administrative issues in sharing data with other physicians, reliability, and increasing patient concerns about their digital footprint, the cost of enabling data access generally exceeds the price of making the data accessible.

[0005] Other issues include the regulatory landscape. Today, accessible biometric data cannot be “legally used” without patient consent. Data collected at a time when global patient / consumer privacy frameworks were evolving and practice / physician consent was tightening has created challenges in using existing data collected and new data in the future. Even in the presence of informed patient consent, doctors, hospitals, and other healthcare organizations are still not motivated to share medical data because incentives are case-by-case and difficult to negotiate at scale. In addition, for “data consumers,” managing both the datasets and legal consent documents poses significant administrative / legal challenges. For example, questions such as “is consent created by another institution in a particular time period still legally valid today?”

[0006] Approaches to address one or more of the above problems have been proposed previously. Some examples include "Single-Sided Parallel Processing," in ... [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] "A Framework for Secure and Decentralized Sharing of Medical Imaging Data Via Blockchain Consensus", Health Informatics J, December 2019, Vol. 25, No. 4, 1398-1411, doi:10.1177 / 1460458218769699 [Non-Patent Document 2] A. Azaria, A. Ekblaw, T. Vieira, A. Lippman, "MedRec: Using Blockchain for Medical Data Access and Permission Management", 2nd International Conference on Open and Big Data (OBD), Vienna, 2016, pp. 25-30, doi:10.1109 / OBD.2016.11 [Non-Patent Document 3] Fan, K., Wang, S., Ren, Y. et al., "MedBlock: Efficient and Secure Medical Data Sharing via Blockchain," J Med Syst, Vol. 42, No. 136 (2018), doi: / 10.1007 / s10916-018-0993-7 Summary of the Invention [Problem to be solved by the invention]

[0008] It is an object of the present invention to provide a system and method for providing valid real-time patient / consumer consent to access patient medical data / records. Another object of the present invention is to provide a system and method that ensures real-time compliance with legal / regulatory requirements.

[0009] It is a further object of the present invention to provide methods and systems that enable and incentivize patients / consumers and (e.g., medical) institutions to share data and gain access to archived (e.g., historically collected) data. [Means for solving the problem]

[0010] As the instability of global regulatory frameworks is not expected to stabilize anytime soon, a solution is needed that can enable real-time patient / consumer confirmed consent. In addition to ensuring compliance with real-time legal requirements, a solution is needed that incentivizes patients / consumers to share data, opens up access to previously collected data, and incentivizes institutions to share data. The above objectives are met in a method / system that provides a framework based on the use of blockchain technology to achieve this solution.

[0011] In the present invention, a marketplace for data sharing is established using blockchain technology to enable patients to commercialize their medical data / records with market value within the medical community. As an example, the present invention is described as being applied to the ophthalmology medical field, but it should be understood that the present invention can be applied to other medical fields, or other more general data collection fields. The present invention also provides a platform for researchers and organizations to purchase the necessary data. At the same time, all stakeholders, such as patients, doctors, data hosts, medical institutions, and blockchain service providers (e.g., management entities or workplace platforms or marketplace platforms or blockchain administrators or data brokers), get a portion of the payment as an incentive.

[0012] For transactions taking place in the marketplace, multiple different transaction types (both monetary and non-monetary) can be defined to limit consent (by the owner of the medical data) and, if necessary, to provide different pricing tiers (e.g., different data access privileges). Such transaction types can include view only, time-limited rental / lease, permanent download, processing on the (workplace / marketplace) platform (e.g., an administrative entity controls / manages the processing of selected medical data remotely from the entities requesting access to the medical data), etc.

[0013] To ensure that the data shared matches the original data, a data validation mechanism is built into the network to validate that the data stored outside the blockchain always matches the metadata on the blockchain. For example, the metadata on the blockchain may include an electronic ledger (e.g., a description / summary of the stored medical data / records), and the medical data itself is stored by a separate data host (e.g., under the control of an administrative entity), and a hash algorithm (or other validation method) may be used to ensure that the metadata on the blockchain matches the separately stored medical data. Similarly, the administrative entity may use a validation mechanism such as a hash algorithm to check that the data made accessible to member users (e.g., remote users) is the same as the data when it was originally stored by the administrative entity.

[0014] The blockchain (computer) network can be directly integrated into medical data acquisition devices such as ophthalmic imaging devices, and the acquired medical data (i.e., measurements, images, etc.) can be automatically transmitted / transmitted by the medical device to a managing entity with consent from the patient (either before or after acquisition of the medical data). Thus, the medical data is seamlessly stored / recorded in the blockchain (and / or the managing entity) without impacting the current clinical workflow.

[0015] Based on this framework, doctor-to-doctor referrals through sharing of data between doctors can also be implemented through this blockchain configuration. Since the patient is at the center of the blockchain, the patient can access his / her medical data at any time. Since all members have access to the blockchain ledger, this network between blockchain members builds transparency and trust.

[0016] This novel approach provides a way to share imaging data (e.g., within the ophthalmology community or other medical community) using a marketplace that provides financial incentives to all stakeholders. The invention also provides different data access types / formats / methods, including data validation. This novel approach also allows ophthalmology patients to have full access to and control over their imaging data (or other medical data).

[0017] Thus, the present invention provides a method and system for conducting a data transaction (e.g., a transaction of medical data), where, with the consent of the owner of the medical data (e.g., a patient who is the subject of or described by the medical data), a managing entity (workplace platform / marketplace platform / blockchain service provider / blockchain administrator) receives the medical data for storage. The medical data may be received electronically (e.g., via a computer network such as the Internet using encryption technology) or on an electronic data storage medium (e.g., CD / DVD). The managing entity records a summary of the received medical data in a blockchain that maintains at least an electronic ledger of available medical records (e.g., medical records accessible via the managing entity). A remote user may then send a request for medical data to the managing entity, preferably specifying criteria describing the type of medical data requested. The managing entity may respond to an electronic request from a remote user for medical data that meets the user-specified criteria by providing the remote user with a list (e.g., count) of available qualifying medical records that meet the user-specified criteria, determined at least in part from the electronic ledger. The remote user may select from the list or simply specify some desired medical records. The managing entity may respond to the remote user's selection of one or more of the eligible medical records by authorizing the remote user to access the selected eligible medical records according to an access approval status for each eligible medical record granted by the eligible medical record owner, and recording the data transaction on the blockchain. The access approval status may be provided by the data owner prior to the remote user requesting the data. For example, the data owner may provide prior approval for selected physicians / laboratories, or for physician referrals, or for specific uses of the medical data.

[0018] Optionally, the managing entity may provide different types of access permissions to the stored medical data. In this case, the managing entity may receive an access type request from the remote user indicating the type of data access requested. The access type request may include one or more of data viewing (only), temporary data access with a time limit, permanent data access by data download, and data processing managed by the managing entity remotely from the remote user (e.g., data processing services provided by the managing entity). The option for permanent access by data download may include removing the accessed eligible medical records from the available medical records in the electronic ledger (e.g., the downloaded data will no longer be available on the blockchain). Optionally, each access type may have an associated access price to be paid by the remote user.

[0019] In some embodiments, the access options for data processing managed by the management entity may include generating a machine learning model using selected medical records and granting access to the generated machine learning model to a remote user. For example, the machine model option may include a deep learning model selected from one or more of an artificial neural network, a convolutional neural network, a U-Net, a recurrent neural network, a generative adversarial network, and a multi-layer perceptron. Alternatively or additionally, the machine model option may include a machine learning model based on one or more of a classification model, a regression model, clustering, and dimensionality reduction.

[0020] As described above, the medical data may include medical measurements or images obtained by a medical device (e.g., a radiology machine, a computed tomography machine, an optical coherence tomography machine, a fundus imaging machine, a visual field testing instrument (perimeter) for testing a subject's visual field, or a slit-scanning ophthalmic system), which, with consent from the subject, automatically transmits the medical data to a management entity.

[0021] The owner of the medical data may be identified by a public identifier based on a public key associated with a corresponding private key, the public identifier excluding any personally identifying information. In this way, the management entity keeps any personally identifying information of the owner of the medical data hidden. In other words, a remote user cannot personally identify the owner of the stored medical data.

[0022] Optionally, the list of available eligible medical records that meet the user-specified criteria may include a count of the eligible medical records. If that number is not sufficient for the remote user, the remote user may choose to modify the user-specified criteria, such as by reducing the number of user-specified criteria. The managing entity may then respond to the electronic modification of the user-specified criteria by providing the remote user with an updated list of available eligible medical records that meet the modified user-specified criteria that are higher than those previously presented.

[0023] The remote user may select individual medical records, but may also select or provide a number of desired medical records that may comprise a lot or group. In response to the remote user selecting one or more of the eligible medical records, the managing entity may check an existing access approval status for each of the selected eligible medical records (e.g., each of the eligible medical records in a lot / group of records), and for each selected eligible medical record that does not have an existing access approval status, the managing entity may send an approval request (such as a message to the application, or email, or text, etc.) to the owner of the eligible medical record and update the approval status of the eligible medical record according to the owner's approval response.

[0024] Similarly, in response to the remote user designating a desired number of eligible medical records, the management entity may check the existing access approval status for each of the eligible medical records. If there are a sufficient number of eligible medical records with existing access approval status, the management entity may select the desired number of eligible medical records from among those with existing access approval status. If there are not a sufficient number of eligible medical records with existing access approval status, the management entity may determine the number of additional eligible medical records required to meet the specified desired number and send an approval request to the owner of the additional required eligible medical records for each additional required eligible medical record. The management entity may then update the approval status of the additional required eligible medical records according to the owner's approval response.

[0025] Optionally, the managing entity may respond to an electronic request from a remote user for medical data that meets user-specified criteria by providing the remote user with a price list associated with a list of available eligible medical records. The remote user may then choose to accept or reject the offered price. The managing entity may respond to a remote user that selects one or more of the eligible medical records by collecting the price associated with the eligible medical records to which the remote user was granted access and distributing the payment to one or more of the owner of each selected eligible medical record, the medical institution that collected any of the eligible medical records to which the remote user was granted access, the data host that hosts any of the eligible medical records to which the remote user was granted access, and / or the managing entity itself.

[0026] The administrative entity may manage the blockchain described above, which may additionally or alternatively be a public ledger computer network. In addition, the received medical data may be stored at least partially on-chain (within the blockchain) and off-chain (outside the blockchain).

[0027] The managing entity may provide automated data access to physicians to whom patients (owners of the medical data) are referred. For example, if the remote user is a physician to whom a patient is referred and the user-specified criteria specifies only the patient's medical records, the managing entity may maintain (or automatically set) the access approval status of eligible medical records to indicate access approval.

[0028] Similarly, the managing entity may provide automated data access to medical record owners. For example, if the remote user is a medical data owner and the user-specified criteria specifies only the medical data owner's medical records, the managing entity may maintain (auto-set) the access approval status of eligible medical records to indicate access approval.

[0029] The managing entity may provide selected users with unfettered viewing access to the electronic ledger recorded on the blockchain for transparency purposes. For example, the managing entity may maintain a group of privileged remote users, each with unencumbered viewing access to the electronic ledger. Privileged users may be identified based on access criteria established by the managing entity, such as one or more of previously granted authorization, the number of medical records the privileged user has submitted for storage to the managing entity, and previous agreed-upon subscription periods or subscriptions paid for.

[0030] By way of example, the user-specified criteria submitted by the remote user may include one or more of a type of anatomical measurement, a physical function measurement, a data scan type, and an image type. For example, the anatomical measurement may include intraocular pressure, keratometry measurements, refractive error, or eye size, the physical function measurement includes an electrocardiogram, vital sign measurements (temperature, pulse rate, respiratory rate), the data scan type includes an A-scan, B-scan, or C-scan from an optical coherence tomography device, and the image type includes a fundus image, an en-face image, or an anterior segment image of the eye.

[0031] As mentioned above, the data transaction may optionally include some incentive to motivate data sharing participation. One example is monetizing medical data transactions. For example, a computer-based method / system may be provided to facilitate a transaction between a buyer and at least one seller, where the seller uses a private key to authorize the submission of medical information to a management entity (or data store). The management entity may identify the seller by a public key-based identifier that is unrelated to the seller's personal identity. The use of public key-private key transactions is known and within the scope of those skilled in the art. The buyer may submit a purchase offer to the management entity for access to a specified type of medical information (e.g., meeting user-specified criteria), and the management entity may identify a list of potential medical records (or sellers with submitted medical information) that match the type of medical information specified in the purchase offer. In response to the seller accepting the purchase offer using the seller's private key (e.g., providing access approval status), the managing entity may complete the transaction, including providing the buyer with the requested access to the seller's medical information that matches the type of medical information specified in the purchase offer, and providing payment to the seller and / or other stakeholders. The managing entity may maintain at least one of the description of the medical information submitted by the seller and the seller's public key using a blockchain computer network. The current transaction is then recorded on the blockchain. Optionally, the purchase offer may include an asking price, but the managing entity may provide the buyer with an asking price for the specified type of medical information. The buyer may accept the asking price or submit another (higher or lower) purchase offer. Similar to the above example, the seller may pre-approve the asking price before the seller submits the purchase offer. In this case, the seller may pre-approve different buying prices for different types of medical information (e.g., ophthalmic vs. orthopedic, or image vs. non-image, etc.) found within the medical information submitted by the seller.

[0032] Optionally, the management entity may provide a machine model creation service, and the buyer's purchase offer may include a desired number of data samples of a specified type of medical information required to create the machine model. In this case, in response to collecting enough acceptances to the purchase offer from the seller to satisfy the desired number of data samples, the management entity creates a buyer-specified machine model using the data samples and provides the created machine model to the buyer. When selecting the machine model creation service, the buyer's requested access may exclude viewing access to the seller's medical information. By way of example, the machine model may include a deep learning model selected from one or more of an artificial neural network, a convolutional neural network, a U-Net, a recurrent neural network, a generative adversarial network, and a multi-layer perceptron. Optionally, the machine model includes a machine learning model based on one or more of a classification model, a regression model, clustering, and dimensionality reduction.

[0033] In the example transaction described above, medical information (eg, medical data / records) access provided to a remote user or buyer may exclude information that personally identifies the data owner or seller.

[0034] More generally, the present invention provides a method / system for sharing medical (e.g., ophthalmology) data. The method includes storing / storing the medical data on a blockchain. A remote user then requests access to the medical data stored / specified on the blockchain with a payment offer. The access request is then approved or denied (e.g., by the data owner). The access request decision is then recorded on the blockchain, and the remote user either gains access to the medical data or receives a notification that the request has been denied based on the access request decision. Payment may then be made to the stakeholder once the access request is approved. The stakeholder may include the owner of the data, which may be the patient from whom the medical data was collected, the doctor or medical facility that collected the data from the patient, the data host that stores the data, and / or the blockchain service provider (which may include an administrative entity). Optionally, the payment may be made based on cryptocurrency. Furthermore, storing the data on the blockchain may include storing both on-chain and off-chain data. For example, the on-chain data may include at least one of a data owner identifier for contacting the owner of the (e.g., ophthalmic) data and storage location information indicating the location of the data stored off-chain. Preferably, the approval or denial of the access request is made by the patient who owns the data. For example, the approval or denial of the access request may be received via an electronic (wired or wireless) communication device from the patient or the like whose medical data was collected. As described above, access to the data may have multiple types, such as view only, rental with a time limit, permanent data download, and processing within the network platform (e.g., process only). In process-only access, the requested data is accessed and processed within a computer network platform that is remote from the remote user who requested the data access, and view access by the remote user may be excluded.

[0035] The present invention also provides a method / system for referring a patient within a medical (e.g., ophthalmology) community. For example, the method may include the steps of storing a patient's medical data in a blockchain network, viewing the medical data set by a first physician and making a referral decision, sending a referral link to the second physician to be referred according to the referral decision, informing the patient that it is desirable for the patient's medical data to be shared with the second physician, the patient approving or denying the data sharing for this referral, recording the approval decision in the blockchain, and receiving a notification that the second physician has access to the data or that the request is denied. Typically, the referral is made with automatic access approval, but as explained above, in this embodiment, the patient can optionally stop the data sharing. In this embodiment, storing the data in the blockchain may include storing both on-chain and off-chain data. The referral link may be sent via email, text message, electronic messaging, etc. As previously mentioned, the approval or denial of the access request is made by the patient who owns the data.

[0036] The present invention also provides a method / system for sharing medical (e.g., ophthalmology) data with patients. The method / system includes storing the data in a blockchain network, where the blockchain can authenticate users who access the blockchain-stored data online. Storing the data in the blockchain can include storing both on-chain and off-chain data. Online access to the blockchain-stored data can be via a web browser, a mobile device app, or a computer application.

[0037] The present invention further provides a method / system for building transparency and trust in medical (e.g., ophthalmology) data sharing. The method includes storing data in a blockchain network, storing all transactions on the blockchain (e.g., a networked public (transaction) ledger), and allowing members to access / view the public transaction ledger. Optionally, access to the networked public transaction ledger by members requires certain criteria, such as paid-up subscription.

[0038] Other objects and achievements of the present invention, together with a fuller understanding of the invention, will become apparent and appreciated by reference to the following description and claims taken in conjunction with the accompanying drawings.

[0039] To facilitate the understanding of the present invention, several publications are cited or referenced herein. All publications cited or referenced herein are incorporated by reference in their entirety.

[0040] The embodiments disclosed herein are merely examples, and the scope of the disclosure is not limited thereto. Any feature of an embodiment described in one claim category, e.g., a system, may also be claimed in another claim category, e.g., a method. Dependencies or back references in the appended claims are selected for formality reasons only. However, any subject matter resulting from a careful back reference to a previous claim may also be claimed, thereby disclosing any combination of the claims and their features and may be claimed regardless of the dependencies selected in the appended claims. [Brief description of the drawings]

[0041] In the drawings, like reference symbols / letters refer to like elements. [Figure 1] FIG. 1 illustrates an exemplary blockchain computer network. [Diagram 2]FIG. 1 provides a general overview of an exemplary system in accordance with the present invention. [Diagram 3] FIG. 1 illustrates some example user-specified criteria for identifying desired medical data. [Figure 4] FIG. 1 illustrates an example of a data count list, data access types, and associated access prices. [Diagram 5] FIG. 1 illustrates an exemplary data access transaction process between a remote user (e.g., a data customer or researcher) and a subject management entity (e.g., a marketplace / workplace platform or blockchain administrator / service provider). [Figure 6] FIG. 1 illustrates the process for storing (moving) new patient data to the managing entity (e.g., marketplace / workplace platform or blockchain administrator / service provider). [Figure 7] FIG. 1 illustrates a workflow for physician referral using the present system / platform. [Figure 8] FIG. 1 is a diagram showing an example of a visual field testing device (perimeter) for testing the visual field of a subject. [Figure 9] FIG. 1 illustrates an example of a slit-scanning ophthalmic system for imaging the fundus. [Figure 10] FIG. 1 illustrates a generalized frequency-domain optical coherence tomography system that may be used to collect 3D image data of the eye suitable for use in the present invention. [Figure 11] FIG. 1 shows an exemplary OCT B-scan image of a normal retina of a human eye, illustratively identifying various normal retinal layers and boundaries. [Figure 12] FIG. 1 shows an example of an en face vascular image. [Figure 13] FIG. 1 shows an exemplary B-scan vasculature (OCTA) image. [Figure 14] FIG. 1 illustrates an example of a multi-layer perceptron (MLP) neural network. [Figure 15]FIG. 1 illustrates a simplified neural network consisting of an input layer, a hidden layer, and an output layer. [Figure 16] FIG. 1 illustrates an exemplary convolutional neural network architecture. [Figure 17] FIG. 1 illustrates an exemplary U-Net architecture. [Figure 18] FIG. 1 illustrates an exemplary computer system (or computing device or computer). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] A typical process for sharing (e.g., medical) data among stakeholders (e.g., participants in a data transaction / exchange) within the ophthalmology community (or other medical community) comprises the following steps: 1. creating a (e.g., medical) research proposal outlining a protocol (e.g., collection of (medical) data, administration of tests, selection of subjects, etc.); 2. Obtaining approval from an IRB (e.g., an Institutional Review Board or Ethics Committee); 3. Obtaining consent and generating legal rights (e.g., use of the patient's medical data) for each subject (e.g., patient in a medical research study); 4. Obtaining the necessary data based on the protocol and storing the data and consent in a database; 5. Sharing all obtained data / consent (e.g., with stakeholders) via online sharing tools, external portable data storage drives, etc.

[0043] However, this current approach has several limitations: 1) Obtaining consent is difficult, not scalable, and generally cannot be revoked. Generally, obtaining consent involves explaining technical details and clinical benefits to patients, which can reduce the efficiency of clinical workflow. Some patients may also refuse to consent given the limited time to understand the terms, limitations, and purpose of consent. Furthermore, if data is obtained under one specific use case based on consent, it may not be extended to other / future uses not covered by the original consent. Also, once consent is given, it can be very difficult to withdraw consent if the patient later changes their mind.

[0044] 2) It is difficult to establish data sharing agreements with physicians. In many cases, physicians tend to limit data sharing to close collaborators. For small companies and new researchers without well-established collaborators, it is difficult to obtain research proposals or research agreements with physicians.

[0045] 3) Patients and physicians do not have adequate financial incentives to share data. Today, innovative solutions based on data can generate substantial economic revenue. However, patients, who are the owners of (medical) data, generally do not obtain adequate financial payment (or other compensation) for their data. This is in part because the current data sharing framework does not provide any mechanism for this to happen. Even physicians who share data have difficulty correctly estimating the value of the data due to the lack of a free marketplace valuation. This poses further challenges when negotiating research proposals / research agreements.

[0046] 4) Once all the research proposals and consents are in place, the data sharing part itself is largely manual, which can be very difficult to scale. This is especially true when shipping physical hard drives (or CDs, or DVDs, or flash drives, or other physical storage media) across countries or around the world. Such shipping poses the risk of data loss during transport.

[0047] 5) Under the current data sharing framework, patients, who are the owners of the data, usually cannot easily access their own data. The present invention uses blockchain technology to address the above problems. The configuration of a typical blockchain computer network is well known in the art, but for purposes of illustration, FIG. 1 shows an exemplary blockchain computer network. In general, a blockchain 11 is a distributed ledger with a growing list of records or blocks [e.g., 11a, 11b, ..., 11(n-1)] that are linked together using cryptographic techniques. The first block in a blockchain is usually called the genesis block. Each block may contain a cryptographic hash of itself and the hash of the previous block (which identifies its location in the blockchain), a timestamp, and transaction data, although additional types of data may be stored within a block. For example, a new block 11n would contain the data to be stored, the hash of block 11n, and the hash of the previous block 11(n-1). The data stored within a block depends on the type of blockchain (e.g., the purpose of the blockchain). As is known in the art, the hash may be determined using a hashing algorithm and provides a unique identifier that identifies the block and all of the block's contents. If the block's contents are changed, a new hash is generated to reflect this change. Thus, the hash can be used to determine when a block has been modified. Furthermore, the new hash of the modified block effectively identifies the modified block as a different block. Additionally, the timestamp can be used to ensure that the stored transaction data was present when the block was published and added to the blockchain. Because each block identifies (e.g., contains information about) its previous block, they form a chain, with each additional block reinforcing its previous block. Blockchains are resistant to data modification because, once recorded, the data in any given block cannot be retroactively changed without also changing all subsequent blocks in the blockchain to reflect the change.

[0048] Previous ophthalmology blockchain applications have not targeted image data sharing. They have focused primarily on aspects of electronic medical records (EMRs), which are largely textual data, and textual data has limited value compared to diagnostic imaging data. Blockchain applications in other domains that allow image data to be shared have not provided a marketplace-based commercialization system for image data. In addition to other features of the present invention, the economic aspects of the present invention have the potential to have a large impact in the industry, creating strong incentives to make data sharing substantive, organic, and patient-centric. Furthermore, the method provides multiple data access types, optionally with different pricing tiers, which provides flexibility to benefit different user groups. Data validation between on-chain data (data stored on the blockchain) and off-chain data (data stored outside the blockchain, but the blockchain may optionally identify where the data is stored) may be provided to ensure that the present blockchain-based solution meets any regulatory requirements and provides real-world benefits. Transactions that occur within the blockchain are commonly referred to as on-chain transactions, and transactions that occur outside the blockchain network are commonly known as off-chain transactions. Also, regulatory requirements may be addressed using digital (or "smart") contracts in on-chain and / or off-chain transactions. Additionally, in the present system, referrals and access to patient imaging data (at all times) may be handled differently than traditional approaches.

[0049] FIG. 2 provides a general overview of an exemplary system according to the present invention. In this example, a patient 21 visits a private doctor or medical institution (e.g., a clinic, hospital, research institute, etc.) that uses a medical device 22 to collect medical data from the patient 21. The collected data may include responses to a patient medical questionnaire, biological measurements of the patient, and / or medical images. Examples of different types of medical devices include, but are not limited to, visual field testing devices (perimeters) for testing the patient's visual field, slit-scanning ophthalmic systems or other cameras for imaging the anterior or posterior segments of the eye, optical coherence tomography (OCT) systems, and OCT angiography (OCTA) systems. A more detailed description of some of these types of devices is provided below. Examples of collected biological measurements and medical images include, but are not limited to, retinal thickness maps, retinal layers, OCT / OCTA B-scans, OCT / OCTA A-scans, OCT / OCTA C-scans, OCT / OCTA en face images, images of especially targeted biological / geographical features such as retina, macula, fovea, optic disc or nerve, posterior pole, especially targeted vascular structures, pupil, cornea, iris, etc. In this example, the patient may grant permission (e.g., provide consent) before or after the consultation for the doctor or medical institution to transmit all or part of the collected data to the management entity (workplace platform or marketplace platform or blockchain service provider or blockchain administrator or data broker) 23. Optionally, the medical equipment / device capturing the medical measurements or medical images automatically transmits / transmits the medical data to the management entity 23 with consent from the patient so as not to disrupt the examination workflow. The data may be transmitted electronically via email, via the Internet (e.g., web portal), computer application, portable device app, or on a physical data storage medium (e.g., hard drive, CD, DVD, USB flash drive, etc.).Thus, with the consent of the owner of the medical data, the managing entity 23 is authorized to receive the medical data for storage in the data store / host 25 or otherwise access the medical data from the remote data store / host 28. For example, the remote data store may be operated by the doctor or medical institution that collected the medical data. Optionally, the owner of the collected medical data, for example the patient in this example, may communicate directly with the managing entity 23 and, optionally, submit his / her medical data directly to the managing entity for storage. Optionally, the data submission agreement with the managing entity 23 may allow the managing entity 23 unfettered research access to the received medical data. Optionally, this research access may exclude the patient's personal identification information.

[0050] The managing entity 23 manages / monitors / governs and records transactions for the exchange of data (e.g. medical data / data records, which may include monetary transactions of data) between different stakeholders (e.g. patients, data owners, doctors, researchers, institutions, etc.). The managing entity 23 may hold / retain a copy of the data, as exemplified by an internal data store 25, or may access medical data stored remotely, as exemplified by a data store 27 that may be under the control of the managing entity 23, or by an independent data store 28. For example, the independent data store 28 may be managed by a doctor or institution that provided the managing entity with the data, or at least a description of the data including instructions for gaining access to the stored data.

[0051] The managing entity 23 may keep the personal identification information of the data owners (and other stakeholders) hidden, e.g., privately, before, during, and after the transaction, but also maintains an electronic ledger (e.g., electronic record) of all transactions, including the public identifiers of the stakeholders involved. As explained above, the managing entity 23 achieves this by using one or more blockchains. Thus, the managing entity 23 may record summaries of all transactions (e.g., medical data received) in the blockchain, which may maintain an electronic ledger of available medical records. This managing entity 23 may host its own private blockchain 24a and / or maintain / manage the public blockchain 24b. In either case, the managing entity 23 may control who is allowed to access the blockchain 24a / 24b. Optionally, stakeholders may communicate with the private blockchain 24a via the managing entity 23, as indicated by the solid arrows, or may be given permission to communicate directly with the public blockchain 24b, as indicated by the dotted arrows. Blockchain can function as a distributed ledger, similar to a database that is shared and synchronized by consensus across multiple sites, institutions, or regions and accessible by multiple people, and blockchain can allow transactions to have public "witnesses," thus enhancing transparency. Participants at each node of the distributed public ledger network can be granted access to recorded transactions that are shared across the network. This allows any changes or additions made to the ledger (e.g., transactions) to be reflected and verified by various stakeholders. The administrative entity 23 can use decentralized identifiers (DIDs) associated with personally identifiable information (PII) and other private data.A DID may be a public domain identity, but its associated PII is kept hidden. Only the owner of a DID can access its associated PII. In this way, the owner can choose with whom to share private (e.g., medical) data, and when, where, and how to share it. This puts access control of personal data on the blockchain back to the owner of the personal data. Optionally, to speed up blockchain transactions, the administrative entity 23 may execute a set of multiple transactions between the same stakeholders and upon completion of the set of transactions, store a record of the transactions on the blockchain. Transactions that occur outside the blockchain network, such as storing or accessing data in a data store outside the blockchain, are commonly known as off-chain transactions, although records of off-chain transactions may also be maintained on the blockchain.

[0052] The blockchain 24a / 24b may use a public key cryptosystem, which may be based on asymmetric encryption. In a public key cryptosystem, the network (e.g., the administrative entity 23 and / or the blockchain 24a / 24b) allows users to generate and use public-private key pairs, where a public key and a uniquely corresponding private key are generated together. The public key can be freely shared with anyone, while the private key acts as a secure passcode to unlock transactions belonging to the public key of that particular pair. Thus, the private key is generally kept secret. In this way, the public-private key pair enables secure transactions, such as communication and data exchange. Transactions are encrypted using the public key, and the corresponding private key of the public key is required for decryption. For example, if a first user wants to receive a message from a second user, the first user shares its public key with the second user, and the second user encrypts the message using the first user's public key. Upon receiving the encrypted message, the first user may decrypt the message using his or her private key. In all transactions, the user's personally identifiable information remains hidden from public view.

[0053] Because public keys can be long strings of alphanumeric characters, they can be unwieldy and cumbersome to use. Optionally, a modified representation of the public key may be used in place of the public key. This modified representation is preferably shorter and easier to use than the original public key it represents. For example, the modified representation of the public key in a public-private key pair may take the form of a "public address" or public identifier. In certain embodiments, the public address / public identifier may take the form of a common email address, making it easier for users (e.g., stakeholders) of the blockchain to identify each other because public addresses may be freely shared. For example, a public address may be created using a hash algorithm on the public key it represents, which effectively adds an additional layer of encryption. Thus, a public address is typically easier to use than the public key it represents, but it is virtually impossible to reverse engineer the private key corresponding to the public address. In this example, when one wishes to access the blockchain, one presents one's public address (or public key), but may also be required to verify one's identity by providing the corresponding private key of the public address (e.g., of the public key represented). If the private key is incorrect, access to the blockchain is denied. In this way, it can be ensured that only holders of specific public addresses (or public keys) are allowed access.

[0054] Thus, owners of medical data stored / managed by the management entity 23 may be identified by a public identifier, which may be based on a public key, and the public identifier is associated with corresponding personally identifiable information, which may be based on a private key. The public identifier preferably excludes any personally identifiable information, thereby allowing the management entity 23 to keep any personally identifiable information of the owners of the medical data hidden.

[0055] A plurality of remote users 29-1 to 29-i may submit electronic requests for medical data that meet user-specified criteria. Optionally, the remote users 29-1 to 29-i may check a ledger of available medical data (provided by the managing entity 23 and / or the blockchain 24a / 24b) before submitting the electronic request. After validating the request, as described above, the managing entity may respond to the electronic request by providing the remote users with a list of available eligible medical records that meet the user-specified criteria, determined at least in part from the electronic ledger.

[0056] With reference to FIG. 3, examples of user-specified criteria may be divided by category. For example, remote users 29-1 to 29-i may specify specific medical measurements or data types 31 and / or may specify patient-specific descriptors / requirements 32. Medical measurements or data types 31 may include anatomical measurements (such as intraocular pressure, keratometry measurements, refractive error, or eye size), body function measurements (e.g., electrocardiogram or vital sign measurements (temperature, pulse rate, respiratory rate)), data scan type (e.g., A-scan, B-scan, or C-scan from an optical coherence tomography device), image type (e.g., fundus image, en-face image, or anterior / posterior segment image of the eye), etc. Patient-specific requirements may include age group, gender, socioeconomic status, geographic location (e.g., country, state, town, geographic region, etc.), ethnicity, pre-existing medical conditions (e.g., previously diagnosed illness), current treatments, etc.

[0057] 4, optionally, the list of available eligible medical records that meet the user-specified criteria may include a count of available (eligible) medical records 41. If the count is not sufficient for the remote user's purposes, the remote user may choose to modify the user-specified criteria (see FIG. 3). The management entity 23 may respond to this electronic modification of the user-specified criteria by automatically providing the remote user with an updated list of available eligible medical records that meet the modified user-specified criteria, including an update availability count 41.

[0058] The remote user may then select individual records and / or submit a general number of desired medical data records. In response to the remote user selecting one or more of the eligible medical records, the management entity 23 checks the existing access authorization status for each selected (eligible) medical record. That is, the owner (e.g., patient) of the individual medical data / records may submit existing authorization for access to their data according to certain criteria (e.g., type of access, intended use, payment amount, specific user requesting the data, etc.). For each selected eligible medical record that does not have an existing access authorization status from the data owner, the management entity sends a request for authorization (which may include a price offer) to the data owner and updates the authorization status of the eligible medical record according to the data owner's approval response.

[0059] Optionally, the remote user may submit a desired number of eligible medical records 42 that is less than the available record count 41. The management entity 23 may respond by checking the existing access approval status for each of the eligible medical records, and if the number of eligible medical records with existing access approval status is sufficient, the management entity 23 may freely select the desired number of eligible medical records from among those with existing access approval status. However, if the number of eligible medical records with existing access approval status is insufficient, the management entity 23 may determine the number of additional eligible medical records required to meet the specified desired number, send an approval request to the owner of the additional required eligible medical record for each additional required eligible medical record, and update the approval status of the additional required eligible medical record according to the data owner's approval response.

[0060] In some cases, the management entity 23 may assign an access approval status to selected medical data according to existing consent from the data owner. For example, if a first doctor is referring a patient to a second doctor, the management entity 23 may allow the second doctor to access the patient's associated medical records without requiring a new access approval response from the patient. For example, if the remote user is the second doctor to whom the patient is referred and the user-specified criteria specify only the patient's medical records, the management entity 23 may maintain or assign an automatic access approval status to the eligible medical records. In another example, if the patient (or other data owner) wishes to access his or her own records, the request for a new access approval response from the patient may be omitted. In other words, if the remote user who submitted the request for medical data is the medical data owner and all user-specified criteria specify only the medical records of the medical data owner, the management entity 23 may maintain or assign an automatic access approval status to the requested medical data.

[0061] As described above, the system may provide financial incentives for stakeholders to participate in the system. For example, the managing entity 23 may provide a remote user requesting medical data with a price list (e.g., per data item or per medical record) associated with the list of available eligible medical records. The price list may vary based on various criteria, such as the type of data requested (e.g., measurement data, image data, questionnaire data, etc.), the geographic location where the data was collected (e.g., country of origin, etc.), the type of data access requested, etc. In any event, if the remote user selects to access any of the eligible medical records, the managing entity 23 may collect the price associated with the eligible medical records from the remote user and distribute the payment to one or more of the owners of the selected medical records, the medical institutions that collected any of the selected medical records, and each of the data stores that host any of the selected medical records, and to the managing entity itself. If the remote user does not accept the proposed price list, the remote user may submit a counteroffer (e.g., offer a different price) for the desired data that is higher or lower than that shown in the proposed price list. The administrative entity 23 and / or the data owner may review the counteroffer and choose to accept or reject it. Future price lists may be based, at least in part, on previously accepted counteroffers.

[0062] Regardless of whether the managing entity 23 collects and distributes payments or provides some other type of incentive benefit, once the remote user selects one or more of the eligible medical records for access, the managing entity 23 responds by granting the remote user access to the selected eligible medical records in accordance with the access approval status for each selected (eligible) medical record granted by the owner of the selected (eligible) medical record, and records the data transaction on the blockchain 24a / 24b.

[0063] As mentioned above, the management entity 23 may provide different types of data access to the remote user. For example, referring to FIG. 4, the management entity 23 may receive from the remote user an access type request selected from a list 43 of available access types for the requested data. For example, the access type list 43 may include data viewing only 43a, temporary (full) data access with a time limit 43b, permanent data access by data download 43c, and remote data processing performed by (or under the control of) the management entity 23 remote from the remote user 43d. Multiple types of access requests may be selected. For example, the remote data processing option 43d may not include viewing access. If the remote user desires the option to view the data, the remote user may further select the viewing only option 43a. Optionally, the permanent download option 43b (e.g., permanent access by data download) may include deleting the accessed eligible medical data from a list of available medical records in the electronic ledger. If financial incentives are implemented, each access type may have an associated access price 44a-44d (eg, per medical record value) payable by the remote user.

[0064] In the remote data processing option 43d, the remote user further selects which type of data processing is desired. The management entity 23 then performs or prepares the desired processing for execution at the remote site. For example, the desired data processing may include using selected medical data to generate a machine learning model (using selected medical records) to address a specified objective, such as locating the fovea in a fundus image. The remote user is then granted full access to the generated machine learning model. Examples of types of machine models include deep learning models selected from one or more of artificial neural networks, convolutional neural networks, U-Net, recurrent neural networks, generative adversarial networks, and multi-layer perceptrons. Examples of these types of machine models are provided below. More generally, the machine model option may additionally or alternatively include machine learning models based on one or more of classification models, regression models, clustering, and dimensionality reduction.

[0065] As described above, the management entity 23 may grant access to the electronic ledger to selected (privileged) remote users for viewing, thereby providing a record of available medical data for access. These privileged remote users may be granted unhindered viewing access to the electronic ledger. Privileged users may be identified based on access criteria established by the management entity 23, including one or more of previously granted authorizations, the number of medical records the privileged user has submitted for storage to the management entity 23, and / or a pre-agreed subscription price / term for access.

[0066] 5 and 6 show a first use case of the present invention. This embodiment illustrates a workflow for sharing (medical) data between blockchain network members (e.g., remote users or stakeholders), for example, through a marketplace. FIG. 5 shows data access (e.g., data exchange / purchase / lease) by (remote) users (e.g., data customers or researchers), and FIG. 6 shows a process for storing (moving) new patient (medical) data to the marketplace platform (e.g., management entity / blockchain). With reference to FIG. 5, first, a (remote) user of the blockchain makes a request for a specific (medical) data type or profile (block 51). Based on the request, a catalog (e.g., list) of available potential datasets that fulfill the request is aggregated (block 52), for example, by the platform / management entity. This aggregation can be done within the blockchain or by a data broker (e.g., management entity) that hosts the datasets. Based on this catalog, a corresponding request for each dataset is issued to the individual patients who own the requested data, as one or more electronic notifications with an offer (e.g., price) based on the data access type (block 53). Data access types may include view only, rental / lease with expiration date (time period), permanent download, data handling within a marketplace platform (e.g., management entity), etc. As shown in FIG. 4, different price tiers may be configured for each data access type such that the offer(s) vary depending on the data access type.

[0067] The owner of the requested data (e.g., the patient) then chooses to approve or deny the data access (block 54). If the patient does not approve the access, a corresponding response (e.g., a notice of access denial) is sent to the (remote) user who submitted the access request (block 55). If the patient approves the access, this means that the patient has explicitly consented to sharing the patient's data, and this explicit consent is recorded in the blockchain and stored as a permanent record (block 57). The user who requested the access (e.g., the data customer) is given access to the data in the requested format (data access type) (block 58). As mentioned above, a validation mechanism is also built into the marketplace platform, so that the data accessed by the (remote) user is validated against the state / record when it was accessed / retrieved and, optionally, against the state / record when it was stored / stored in the platform / management entity. If the validation is passed, this data access is considered successful. Once the access is completed, a payment transaction is made and all stakeholders get their appropriate share of the payment (block 59). The appropriate percentage of payment can be agreed upon before the transaction or determined in real-time based on current supply and demand value / percentage trends. For example, the patient who owns the data, the doctor / clinic who obtained the data, the hospital / imaging center / device company that hosts the data, and the blockchain service all receive payments. At any time, the patient can withdraw consent and the pro rata payment will be refunded. After consent is withdrawn, the (previously) permanently downloaded data can no longer be used for research or commercial purposes.

[0068] The present marketplace platform may include various tools / services for purchased, leased, or otherwise accessed data within the platform itself. These tools may be used to assist in the manipulation, analysis, or other processing of the accessed data. For example, if the processing of the accessed data is performed at least in part within the marketplace platform, the marketplace platform may enable users (data customers) to run one or more applications / tools / services (apps) directly on the purchased / leased data available via the blockchain. For example, the marketplace platform may provide the data customer access to one or more automated machine learning (ML) apps that assist in the analysis of the purchased / leased data. Various examples of ML architectures are provided below. When a researcher (user) purchases / leases data and chooses to process the purchased / leased data within the platform, an automated ML app may be purchased (or leased) and used on the purchased / leased data to generate a trained ML model, optionally according to training instructions / training settings provided by the researcher. In another example, an algorithm or analytical app may be purchased / leased and run on the purchased / leased data in the blockchain to generate business insights.

[0069] Referring to FIG. 6, the saving (storage) of new patient data to the blockchain (managing entity) can be performed in the background and is fully compatible with existing clinical workflow. For example, after a medical test is completed (block 61), based on the patient's preference / permission, the patient's data is saved / sent to the blockchain (managing entity) or not saved / sent (block 62). If the patient prefers not to opt-in to the marketplace platform (block 62=no), the patient data is not moved to the workplace platform (block 64) according to the (doctor / clinic's) normal workflow. If the patient's preference is to opt-in to the marketplace platform (block 62=yes), the patient's test data is saved to the blockchain (managing entity / platform) and the patient receives a notification after the data saving is successful (block 63). At the same time, this patient data (e.g., image dataset) is added to the data sharing marketplace database (e.g., electronic ledger / blockchain) as indicated by block 63.

[0070] Similar to sharing patient / medical data, algorithms / applications can also be shared using the blockchain (governing entity / platform). In this case, the algorithms / applications are processed in a similar way to how "data" is processed and can be tested / licensed / purchased on a marketplace platform using the blockchain. In this way, all approvals and transaction records are automatically stored on the blockchain.

[0071] FIG. 7 illustrates a workflow for physician referral (e.g., using blockchain) using the present managing entity / platform. In this use case, a first physician views patient data (block 71) and then decides whether to refer the patient to a second physician (block 72). If the referral decision is no (block 72=no), the process ends (block 73). If the referral decision is yes (block 72=yes), a referral link (e.g., email, electronic message, URL, etc.) is sent to the second physician with instructions on how to join the blockchain network (e.g., how to register as a user with the managing entity) (block 74). At the same time, the patient who owns the data may receive a notification indicating that their medical data / records will be shared with the second physician, e.g., as part of a physician referral (block 74). In one embodiment, the patient may then approve or disapprove the sharing of their data (block 75). If the patient does not approve (block 75=no), both physicians receive a response indicating the patient's refusal to share the data with the second physician (block 76). This is equivalent to the patient refusing the referral. If the patient approves the sharing of their data (block 75=yes), the second physician gains access to the patient's data (block 77). Alternatively, this patient approval process can be integrated into the clinical workflow as part of the patient consent to allow the patient's data to be shared (automatically) with other physicians.

[0072] Optionally, patients whose data are stored on the blockchain network can access their data at any time through a web portal (e.g., a web browser) or app, as described above. Once the patient's credentials are authenticated, the patient has, by default, full consent to access their data.

[0073] Optimally, the marketplace platform (administrative entity) may provide members of the blockchain network who meet certain requirements (e.g., paid subscription, minimum number of data sets shared) access to a blockchain ledger summary with all recorded block information. This provides transparency in the blockchain network and builds trust by having everyone have access to a single record of truth.

[0074] Various hardware and architectures suitable for the present invention are described below. Visual Field Testing System The improvements described herein can be used in combination with any kind of visual field tester / system (e.g., perimeter). One such system is a "bowl" visual field tester VF0, as shown in FIG. 8. A subject (e.g., a patient) VF1 is shown looking at a generally bowl-shaped hemispherical projection screen (or other kind of display) VF2, hence the name tester VF0. Typically, the subject is instructed to gaze at a point at the center of the hemispherical screen VF3. The subject places his / her head on a patient support, which may include a chin rest VF12 and / or a forehead rest VF14. For example, the subject places his / her head on the chin rest VF12 and his / her forehead on the forehead rest VF14. Optionally, the chin rest VF12 and the forehead rest VF14 can be moved together or independently of each other to properly fix / position the patient's eyes, for example, relative to a trial lens holder VF9, which may hold a lens through which the subject can see the screen VF2. For example, the chin rest and head rest can move independently vertically to accommodate different patient head sizes, and move together horizontally and / or vertically to properly position the head, however this is not limiting and other positions / movements can be envisioned by one skilled in the art.

[0075] A projector or other image forming device VF4 under the control of a processor VF5 displays a series of test stimuli (e.g., test points of any shape) VF6 on a screen VF2. The subject VF1 indicates that he / she has seen the stimuli VF6 by activating a user input VF7 (e.g., pressing an input button). This subject's response can be recorded by the processor VF5. The processor VF5 can function to evaluate the field of view of the eye based on the subject's response, for example, to determine the size, position, and / or intensity of the test stimuli VF6 that are no longer visible by the subject VF1, thereby determining the (visibility) threshold of the test stimuli VF6. A camera VF8 can be used to capture the patient's gaze (e.g., gaze direction) throughout the test. The gaze direction can be used to confirm the patient's alignment and / or compliance with the proper test procedure. In this example, camera VF8 is positioned on the Z-axis relative to the patient's eye (e.g., relative to trial lens holder VF9) and behind the bowl (of screen VF2) to capture live image(s) or video of the patient's eye. In other embodiments, this camera may be positioned away from this Z-axis. Images from gaze camera VF8 can be optionally displayed on a second display VF10 to a clinician (who may be interchangeably referred to herein as a technician) to assist with patient alignment or verification of the test. Camera VF8 can record and store one or more images of the eye during each stimulus presentation. This can allow tens to hundreds of images to be collected in a single visual field test, depending on the test conditions. Alternatively, camera VF8 can record and store full-length videos during the test and provide timestamps indicating when each stimulus was presented. Additionally, images can be collected during the presentation of stimuli to provide details of the subject's overall attention during the VF test.

[0076] A trial lens holder VF9 may be placed in front of the patient's eye to correct the refractive error of the eye. Optionally, the lens holder VF9 may carry or hold a liquid trial lens that may be utilized to provide variable refractive correction to the patient VF1 (see, for example, U.S. Patent No. 8,668,338, which is incorporated herein by reference in its entirety). However, it should be noted that the present invention is not limited to using liquid trial lenses for refractive correction, and other conventional / standard trial lenses known in the art may also be used.

[0077] In some embodiments, one or more light sources (not shown) can be placed in front of the subject's eye VF1 to generate a reflection from the surface of the eye, such as the cornea. In one variation, the light source can be a light emitting diode (LED).

[0078] Although FIG. 8 shows a projected visual field tester VF0, the invention described herein can be used with other types of devices (visual field testers), including devices that generate images via liquid crystal displays (LCDs) or other electronic displays (see, for example, U.S. Pat. No. 8,132,916, which is incorporated herein by reference). Other types of visual field testers include, for example, flat screen testers, miniature testers, and binocular visual field testers. Examples of these types of testers can be found in U.S. Pat. Nos. 8,371,696, 5,912,723, 8,931,905, and U.S. Design Registration No. D472,637, each of which is incorporated herein by reference in its entirety.

[0079] The visual field tester VF0 may incorporate an instrument control system (e.g., executing an algorithm, which may be software, code, and / or routine) that uses hardware signals and a motorized positioning system to automatically position the patient's eye at a desired position (e.g., the center of the refractive lens in the lens holder VF9). For example, stepper motors may move the chin rest VF12 and forehead rest VF14 under software control. Rocker switches may be provided to allow the technician to adjust the position of the patient's head by activating the chin rest and forehead stepper motors. A manually movable refractive lens may also be placed in front of the patient's eye on the lens holder VF9 as close to the patient's eye as possible without adversely affecting the patient's comfort. Optionally, the instrument control algorithm may pause the execution of the visual field test while the chin rest and / or forehead motor movement is in progress if such movement would interfere with the execution of the test.

[0080] Fundus Imaging System Two categories of imaging systems used to image the fundus are flood-illumination imaging systems (or flood-illumination imagers) and scanning-illumination imaging systems (or scanning imagers). A flood-illumination imager simultaneously floods the entire field of view (FOV) of interest of the object with light, such as by using a flash lamp, and captures a full-frame image of the object (e.g., fundus) with a full-frame camera (e.g., a camera having a two-dimensional (2D) photosensor array sized collectively to capture the desired FOV). For example, a flood-illumination fundus imager floods the fundus of the eye with light and captures a full-frame image of the fundus in a single image capture sequence of the camera. A scanning imager provides a scanning beam that is scanned across the object, e.g., the eye, and the scanning beam is imaged at different scanning positions as the scanning beam is scanned across the object to create a series of image segments that can be reconfigured, e.g., combined, to create a composite image of the desired FOV. The scanning beam can be a point, a line, or a two-dimensional region such as a slit or a wide line. Examples of fundus imaging devices are provided in US Pat. Nos. 8,967,806 and 8,998,411.

[0081] FIG. 9 illustrates an example of a slit-scanning ophthalmic system SLO-1 for imaging the fundus F, which is the inner surface of the eye E opposite the eye lens (or crystalline lens) CL and may include the retina, optic disc, macula, fovea, and posterior pole. In this example, the imaging system is in a so-called "scan-descan" configuration, in which the scanning line beam SB traverses the optical components of the eye E (including the cornea Crn, iris Irs, pupil Ppl, and crystalline lens) so as to be scanned across the entire fundus F. In the case of a floodlight fundus imaging device, no scanner is required and light is irradiated over the entire desired field of view (FOV) at one time. Other scanning configurations are known in the art, and the specific scanning configuration is not critical to the present invention. As shown, the imaging system includes one or more light sources LtSrc, preferably a multi-color LED system or a laser system with a suitably adjusted etendue. An optional slit Slt (adjustable or stationary) may be positioned in front of the light source LtSrc and used to adjust the width of the scanning line beam SB. Additionally, the slit Slt can remain stationary during imaging or can be adjusted to different widths to allow different confocal levels and different applications for a particular scan or during scanning used for reflection suppression. An optional objective lens ObjL can be placed in front of the slit Slt. The objective lens ObjL can be any one of the state-of-the-art lenses, including but not limited to refractive, diffractive, reflective, or hybrid lenses / systems. The light from the slit Slt passes through a pupil splitting mirror SM and is directed to the scanner LnScn. It is desirable to bring the scanning plane and the pupil plane as close as possible to reduce vignetting of the system. An optional optical system DL can be included to manipulate the optical distance between the images of the two components. The pupil splitting mirror SM can pass the illumination beam from the light source LtSrc to the scanner LnScn and reflect the detection beam from the scanner LnScn (e.g., the reflected light back from the eye E) towards the camera Cmr. The task of the pupil splitting mirror SM is to split the illumination beam and the detection beam and to help suppress system reflections.Scanner LnScn can be a rotating galvo scanner or other type of scanner (e.g., piezoelectric or voice coil, microelectromechanical system (MEMS) scanner, electro-optic deflector, and / or rotating polygon scanner). Depending on whether pupil splitting occurs before or after scanner LnScn, the scan can be split into two steps with one scanner in the illumination path and a separate scanner in the detection path. Particular pupil splitting arrangements are described in detail in U.S. Pat. No. 9,456,746, the entirety of which is incorporated herein by reference.

[0082] From the scanner LnScn, the illumination beam passes through one or more optical systems, in this case a scan lens SL and an ophthalmic or ocular lens OL, that allow the pupil of the eye E to be imaged into the image pupil of the system. In general, the scan lens SL receives the scanning illumination beam from the scanner LnScn at any of a number of scan angles (angles of incidence) and generates a scan line beam SB with a substantially planar focal plane (e.g., a collimated optical path). The ophthalmic lens OL can then focus the scan line beam SB onto the object to be imaged. In this example, the ophthalmic lens OL focuses the scan line beam SB onto the fundus F (or retina) of the eye E to image the fundus. In this way, the scan line beam SB creates a transverse scan line that moves across the fundus F. One possible configuration of these optical systems is a Keplerian telescope, where the distance between the two lenses is selected to generate an intermediate fundus image that is approximately telecentric (4-f configuration). The ophthalmic lens OL can be a single lens, an achromatic lens, or an arrangement of different lenses. All lenses can be refractive, diffractive, reflective, or hybrid, as known to those skilled in the art. The size and / or shape of the ophthalmic lens OL, the focal length of the scanning lens SL, the pupil splitting mirror SM, and the scanner LnScn can vary depending on the desired field of view (FOV), so that arrangements can be envisioned in which multiple components can be switched in and out of the beam path, for example, by using a flip of the optical system, a motorized wheel, or removable optical elements, depending on the field of view. The pupil splitting can also be changed to accommodate the change in FOV, since a change in field of view results in a different beam size on the pupil. For example, a field of view of 45° to 60° is a typical or standard FOV for a fundus camera. A higher field of view, for example a wide field of view FOV of 60° to 120° or more, may also be feasible. A wide field of view FOV may be desirable for the combination of a wide line fundus imager (BLFI) with another imaging modality, such as an optical coherence tomography (OCT). The upper limit of the field of view may be determined by the accessible working distance combined with the physiological conditions around the human eye.Since a typical human retina has an FOV of 140° horizontally and 80°-100° vertically, it may be desirable to have an asymmetric field of view with respect to the FVO as high as possible on the system.

[0083] The scanning beam SB passes through the pupil Ppl of the eye E and is directed to the retina or fundus, i.e. surface F. The scanner LnScn1 adjusts the position of the light on the retina or fundus F so that a range of lateral positions of the eye E is illuminated. The reflected or scattered light (or emitted light in case of fluorescence imaging) is directed along a similar path as the illumination, defining a focused beam CB on a detection path to the camera Cmr.

[0084] In the "scan-descan" configuration of the exemplary slit-scanning ophthalmic system SLO-1 of the present invention, the light returning from the eye E is "descanned" by the scanner LnScn on the way to the pupil splitting mirror SM. That is, the scanner LnScn scans the illumination beam SB from the pupil splitting mirror SM to define a scanning illumination beam SB across the eye E, but since the scanner LnScn also receives the returning light from the eye E at the same scanning position, it has the effect of descanning the returning light (e.g., canceling the scanning motion) to define a non-scanning (e.g., stationary) focused beam from the scanner LnScn to the pupil splitting mirror SM, which folds the focused beam towards the camera Cmr. At the pupil splitting mirror SM, the reflected light (or emitted light in the case of fluorescence imaging) is separated from the illumination light onto a detection path that is directed to the camera Cmr, which may be a digital camera with a photosensor to capture an image. An imaging (e.g., objective lens) lens ImgL may be positioned in the detection path such that the fundus is imaged onto the camera Cmr. As in the case of the objective lens ObjL, the imaging lens ImgL can be any type of lens known in the art (e.g., refractive, diffractive, reflective or hybrid lens). Additional operational details, in particular methods for reducing artifacts in images, are described in WO 2016 / 124644, the entire contents of which are incorporated herein by reference. The camera Cmr captures the received images and, for example, creates an image file, which can be further processed by one or more (electronic) processors or computing devices (e.g., the computer system shown in FIG. 18). Thus, focused beams (returned from all scanning positions of the scanning line beam SB) are collected by the camera Cmr, and a full frame image Img can be constructed, such as by montage, from a combination of the individually captured focused beams. However, other scanning configurations are also envisaged, including those in which the illumination beam is scanned across the eye E and the focused beam is scanned across the optical sensor array of the camera.WO 2012 / 059236 and U.S. Patent Application Publication No. 2015 / 0131050, which are incorporated herein by reference, describe several embodiments of scanning slit ophthalmoscopes, including various designs in which the returning light is swept across the camera's photosensor array, designs in which the returning light is not swept across the camera's photosensor array, and the like.

[0085] In this example, the camera Cmr is connected to a processor (e.g., processing module) Proc and a display (e.g., display module, computer screen, electronic screen, etc.) Dspl, both of which may be part of the imaging system itself or may be part of separate, dedicated processing and / or display units, such as a computer system, where data is passed from the camera Cmr to the computer system via a cable or computer network, including a wireless network. The display and processor may be an integrated unit. The display may be a traditional electronic display / screen or may be of the touch screen type and may include a user interface for displaying information to and receiving information from the equipment operator or user. The user may interact with the display using any type of user input device as known in the art, including but not limited to a mouse, knob, button, pointer, and touch screen.

[0086] It may be desirable for the patient's gaze to remain fixed while imaging is performed. One way to achieve gaze fixation is to provide a fixation target to which the patient can be instructed to look. The fixation target can be internal or external to the device depending on which area of ​​the eye is imaged. One embodiment of an internal fixation target is shown in FIG. 9. In addition to the primary light source LtSrc used for imaging, an optional second light source FxLtSrc, such as one or more LEDs, can be positioned such that a light pattern is imaged onto the retina using a lens FxL, a scanning element FxScn and a reflector / mirror FxM. The fixation scanner FxScn can move the position of the light pattern, and the reflector FxM directs the light pattern from the fixation scanner FxScn to the fundus F of the eye E. Preferably, the fixation scanner FxScn is positioned such that it is located in the pupil plane of the system so that the light pattern on the retina / fundus can be moved according to the desired fixation position.

[0087] The slit-scanning ophthalmology system can operate in different imaging modes depending on the light source and wavelength-selective filtering elements used. True color reflectance imaging (similar to that observed by clinicians when examining the eye using a handheld or slit-lamp ophthalmoscope) can be achieved when imaging the eye with a series of colored LEDs (red, blue, green). Images of each color can be built up stepwise with each LED turned on at each scanning position, or each color image can be taken completely separately. The three color images can be combined to display a true color image, or displayed individually to highlight different features of the retina. The red channel best highlights the choroid, the green channel highlights the retina, and the blue channel highlights the pre-retinal layers. Additionally, specific frequencies of light (e.g., individual colored LEDs or lasers) can be used to excite different fluorophores (e.g., autofluorescence) in the eye, and the resulting fluorescence can be detected by filtering out the excitation wavelengths.

[0088] Fundus imaging systems can also provide infrared reflectance images, such as by using an infrared laser (or other infrared light source). Infrared (IR) mode is advantageous in that the eye is not sensitive to IR wavelengths. This infrared (IR) mode may allow the user to take images continuously without disturbing the eye (e.g., in preview / alignment mode) to assist the user during alignment of the instrument. IR wavelengths may also have high penetration through tissue and provide improved visualization of choroidal structures. In addition, fluorescein angiography (FA) and indocyanine green (ICG) angiography imaging can be accomplished by collecting images after a fluorescent dye is injected into the subject's bloodstream. For example, with FA (and / or ICG), a series of time-lapse images may be captured after a photoreactive dye (e.g., a fluorescent dye) is injected into the subject's bloodstream. It should be noted that caution is advised as fluorescent dyes can cause life-threatening allergic reactions in some individuals. High-contrast grayscale images are captured using specific light frequencies selected to excite the dye. As the dye flows through the eye, different parts of the eye glow brightly (e.g., fluoresce), allowing the viewer to see how the dye, and therefore blood, is progressing through the eye.

[0089] Optical coherence tomography system In general, optical coherence tomography (OCT) uses low-coherence light to generate two-dimensional (2D) and three-dimensional (3D) internal views of biological tissues. OCT allows in vivo imaging of retinal structures. OCT angiography (OCTA) generates flow information such as vascular flow from within the retina. Examples of OCT systems are provided in U.S. Pat. Nos. 6,741,359 and 9,706,915, and examples of OCTA systems include U.S. Pat. Nos. 9,700,206 and 9,759,544, all of which are incorporated herein by reference in their entireties. Exemplary OCT / OCTA systems are provided herein.

[0090] FIG. 10 illustrates a generalized frequency domain optical coherence tomography (FD-OCT) system for ocular 3D image data collection suitable for use with the present invention. The FD-OCT system OCT_1 includes a light source LtSrc1. Typical light sources include, but are not limited to, a broadband light source with a short temporal coherence length, or a swept laser source. A beam of light from the light source LtScr1 is typically guided by an optical fiber Fbr1 to illuminate a sample, for example, an eye E, a typical sample being human intraocular tissue. The light source LrSrc1 can be, for example, a broadband light source with a short temporal coherence length in the case of spectral domain OCT (SD-OCT) or a tunable laser source in the case of swept-source OCT (SS-OCT). The light can typically be scanned using a scanner Scnr1 between the output of the optical fiber Fbr1 and the sample E, so that the beam of light (dashed line Bm) is scanned laterally over the area of ​​the sample to be imaged. The light beam from the scanner Scnr1 can pass through a scanning lens SL and an ophthalmic lens OL and be focused on a sample E to be imaged. The scanning lens SL can receive the light beam from the scanner Scnr1 at multiple angles of incidence and generate substantially collimated light, which the ophthalmic lens OL can then focus on the sample. This example shows a scanning beam that needs to be scanned in two lateral directions (e.g., x-direction and y-direction on a Cartesian plane) to scan a desired field of view (FOV). This example is a point-field OCT that uses a point-field beam to scan across the sample. Thus, the scanner Scnr1 is exemplarily shown to include two sub-scanners, a first sub-scanner Xscn for scanning the point-field beam over the sample in a first direction (e.g., horizontal x-direction) and a second sub-scanner Yscn for scanning the point-field beam on the sample in an intersecting second direction (e.g., vertical y-direction). If the scanning beam is a line field beam (e.g., line field OCT) and can sample an entire line portion of the sample at one time, only one scanner may be required to scan the line field beam across the sample to span the desired FOV.If the scanning beam is a full-field beam (eg, full-field OCT), a scanner may not be required and the full-field light beam may be illuminated across the entire desired FOV at once.

[0091] Regardless of the type of beam used, light scattered from the sample (e.g., sample light) is collected. In this embodiment, scattered light returning from the sample is collected in the same optical fiber Fbr1 used to route light for illumination. The reference light originating from the same light source LtSrc1 travels along a separate path, which in this case includes an optical fiber Fbr2 and a retroreflector RR1 with an adjustable optical delay. As will be appreciated by those skilled in the art, a transmissive reference path can also be used, and an adjustable delay can be placed in the sample or reference arm of the interferometer. The collected sample light is combined with the reference light, for example, in a fiber coupler Cplr1, to form optical interference in an OCT photodetector Dtctr1 (e.g., a photodetector array, digital camera, etc.). Although one fiber port is shown reaching the detector Dtctr1, as will be appreciated by those skilled in the art, various designs of interferometers can be used for balanced or unbalanced detection of the interference signal. The output from the detector Dtctr1 is fed to a processor (e.g., an internal or external computing device) Cmp1, which converts the observed interference into sample depth information. The depth information may be stored in a memory associated with the processor Cmp1 and / or displayed on a display (e.g., computer / electronic display / screen) Scn1. The processing and storage functions may be localized within the OCT device, or the functions may be offloaded to (e.g., executed on) an external processor (e.g., an external computer system) to which the collected data is transferred. An example of a computing device (or computer system) is shown in FIG. 18. This unit may be dedicated to data processing or may perform other tasks that are quite general and not dedicated to the OCT device.The processor (computing device) Cmp1 may include, for example, a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a system on a chip (SoC), a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or combinations thereof, that may perform some or all of the processing steps in a serial and / or parallel manner with one or more host processors and / or one or more external computing devices.

[0092] The sample and reference arms in the interferometer can be constructed of bulk optics, fiber optics, or hybrid bulk optics systems and can have different architectures, such as Michelson, Mach-Zehnder, or common path designs, as known to those skilled in the art. Light beam, as used herein, should be interpreted as any carefully directed optical path. Instead of mechanically scanning the beam, a light field can illuminate a one- or two-dimensional area of ​​the retina to generate OCT data (see, e.g., U.S. Pat. No. 9,332,902; D. Hillmann et al., "Holoscopy-holographic optical coherence tomography," Optics Letters, vol. 36(13), p. 2290, 2011; Y. Nakamura et al., "High-Speed ​​three dimensional human retinal imaging by line field spectral domain optical coherence tomography," Optics Express, vol. 20(1), pp. 111-115, 2012). Express, 15(12), 7103, 2007; Blazkiewicz et al., Signal-to-noise ratio study of full-field Fourier-domain optical coherence tomography, Applied Optics, 44(36), 7722, 2005. In time-domain systems, the reference arm needs to have an adjustable optical delay to create interference. Balanced detection systems are typically used in TD-OCT and SS-OCT systems, and a spectrometer is used at the detection port for SD-OCT systems. The invention described herein can be applied to either type of OCT system.Various aspects of the present invention may be applied to any type of OCT system or to other types of ophthalmic diagnostic systems and / or to multiple ophthalmic diagnostic systems, including but not limited to fundus imaging systems, visual field testing instruments, and scanning laser polarimeters.

[0093] In Fourier Domain Optical Coherence Tomography (FD-OCT), each measurement is a real-valued spectrally controlled interferogram (Sj(k)). The real-valued spectral data typically undergoes several post-processing steps, including background removal, dispersion correction, etc. A Fourier transform of the processed interferogram gives the complex OCT signal output Aj(z)=|Aj|eiφ. The absolute value of this complex OCT signal, |Aj|, ​​reveals the scattering intensity at different path lengths and thus the scattering profile with respect to depth (z-direction) within the sample. Similarly, the phase φj can also be extracted from the complex OCT signal. The profile of the scattering with respect to depth is called an axial scan (A-scan). A collection of A-scans measured at adjacent locations within the sample produces a cross-sectional image (tomogram or B-scan) of the sample. A collection of B-scans collected at different lateral locations on the sample constitutes a data volume or cube. For a particular data volume, the fast axis refers to the scan direction along which one B-scan is collected, and the slow axis refers to the axis along which multiple B-scans are collected. The term "cluster scan" may refer to one unit or block of data generated by repeated acquisition at the same (or substantially the same) location (or region) to analyze motion contrast that may be used to identify blood flow. A cluster scan may consist of multiple A-scans or B-scans collected at approximately the same location on the sample at a relatively short time interval. Because the scans in a cluster scan are of the same region, motion contrast between scans that meets predetermined criteria may be identified as blood flow, while stationary structures remain relatively unchanged between scans in the cluster scan.

[0094] Various methods for generating B-scans are known in the art, including, but not limited to, along the horizontal or x direction, along the vertical or y direction, along the x and y diagonals, or in a circular or spiral pattern. The B-scan may be in the xz dimension, but may also be any cross-sectional image including the z dimension. An exemplary OCT B-scan image of a normal retina of a human eye is shown in FIG. 11. An OCT B-scan of the retina provides a view of the structure of the retinal tissue. For illustrative purposes, FIG. 11 identifies various normal retinal layers and layer boundaries. The identified retinal boundary layers include (from top to bottom) inner limiting membrane (ILM) layer 1, retinal nerve fiber layer (RNFL or NFL) layer 2, ganglion cell layer (GCL) layer 3, inner plexiform layer (IPL) layer 4, inner nuclear layer (INL) layer 5, outer plexiform layer (OPL) layer 6, outer nuclear layer (ONL) layer 7, the junction between the outer segments (OS) and inner segments (IS) of photoreceptors (indicated by reference numeral 8), external limiting membrane (ELM or OLM) layer 9, retinal pigment epithelium (RPE) layer 10, and Bruch's membrane (BM) layer 11.

[0095] In OCT angiography or functional OCT, analysis algorithms may be applied to OCT data collected at different times (e.g., cluster scans) at the same or nearly the same sample location on the sample to analyze motion or flow (see, e.g., U.S. Patent Application Publication Nos. 2005 / 0171438, 2012 / 0307014, 2010 / 0027857, 2012 / 0277579, and U.S. Patent No. 6,549,801, all of which are incorporated herein by reference in their entireties). The OCT system may use any one of a number of OCT angiography processing algorithms (e.g., motion contrast algorithms) to identify blood flow. For example, motion contrast algorithms may be applied to intensity information derived from the image data (intensity-based algorithms), phase information from the image data (phase-based algorithms), or complex image data (complex-based algorithms). An en face image is a 2D projection of the 3D OCT data (e.g., by averaging the intensity of each individual A-scan, whereby each A-scan defines a pixel in the 2D projection). Similarly, an en face vascular image is an image that displays a motion contrast signal in which a data dimension that corresponds to depth (e.g., the z-direction along the A-scan) is displayed as one representative value (e.g., a pixel in the 2D projection image), typically by summing or integrating all or isolated portions of the data (see, e.g., U.S. Pat. No. 7,301,644, which is incorporated herein by reference in its entirety). OCT systems that provide angiography capabilities may be referred to as OCT angiography (OCTA) systems.

[0096] FIG. 12 shows an example of an en face vasculature image. After processing the data and highlighting the motion contrast using any of the motion contrast methods known in the art, an en face (e.g., front view) image of the vasculature may be generated by summing pixel ranges corresponding to a tissue depth from the surface of the retina's internal limiting membrane (ILM). FIG. 13 shows an example B-scan of a vasculature (OCTA) image. As shown, structural information may be less clear because blood flow traverses multiple retinal layers, which may obscure the multiple retinal layers more than in a structural OCT B-scan such as that shown in FIG. 11. Nevertheless, OCTA provides a non-invasive technique for imaging the retinal and choroidal microvasculature, which may be important for diagnosing and / or monitoring various pathologies. For example, OCTA may be used to identify diabetic retinopathy by identifying microaneurysms, neovascular complexes, and quantifying the foveal avascular zone and nonperfused areas. Moreover, OCTA has been shown to be in good agreement with fluorescein angiography (FA), a more traditional but more evasive technique that requires the injection of dye to observe vascular flow in the retina. Furthermore, in dry age-related macular degeneration, OCTA has been used to monitor the generalized reduction in choriocapillaris flow. Similarly, in exudative age-related macular degeneration, OCTA can provide qualitative and quantitative analysis of choroidal neovascular membranes. OCTA has also been used to study vascular occlusion, for example, for the evaluation of nonperfused areas as well as the integrity of the superficial and deep plexuses.

[0097] Neural Networks As mentioned above, the present invention may use a neural network (NN) machine learning (ML) model. For completeness, neural networks are generally described herein. The invention may use any of the following neural network architectures, alone or in combination: A neural network, or neural net, is a network of interconnected neurons (via nodes), with each neuron representing a node in the network. A collection of neurons may be arranged in layers, with the output of one layer being fed forward to the next layer in a multi-layer perceptron (MLP) arrangement. An MLP may be understood as a feed-forward neural network that maps a set of input data to a set of output data.

[0098] FIG. 14 illustrates an example of a multi-layer perceptron (MLP) neural network. The structure may include multiple hidden (e.g., inner) layers HL1-HLn, which map an input layer InL (receiving a set of inputs (or vector inputs) in_1-in_3) to an output layer OutL, which generates a set of outputs (or vector outputs), e.g., out_1 and out_2. Each layer may have any number of nodes, which are shown here for illustrative purposes as circles within each layer. In this example, the first hidden layer HL1 has two nodes, and hidden layers HL2, HL3, and HLn each have three nodes. In general, the deeper the MLP (e.g., the more hidden layers there are in the MLP), the greater its learning capacity. The input layer InL may receive vector inputs (shown for illustrative purposes as three-dimensional vectors consisting of in_1, in_2, and in_3) and feed the received vector inputs to the first hidden layer HL1 in the sequence of hidden layers. The output layer OutL receives the output from the last hidden layer in the multi-layer model, say HLn, and produces a vector output result (shown for illustration purposes as a two-dimensional vector consisting of out_1 and out_2).

[0099] Typically, each neuron (i.e., node) generates one output that is fed forward to the neurons in the immediately following layer. However, each neuron in a hidden layer may receive multiple inputs, either from the input layer or from the outputs of neurons in the immediately preceding hidden layer. In general, each node may apply a function to its inputs to generate an output for that node. Nodes in a hidden layer (e.g., the learning layer) may apply the same function to each of their inputs to generate each of their outputs. However, some nodes, e.g., nodes in the input layer InL, may only receive one input and be passive, meaning that they simply relay the value of that one input to their output, e.g., they provide a copy of that input to their output, which is indicated by dashed arrows in the nodes of the input layer InL for illustration purposes.

[0100] For illustrative purposes, FIG. 15 shows a simplified neural network consisting of an input layer InL′, a hidden layer HL1′, and an output layer OutL′. The input layer InL′ is shown to have two input nodes i1 and i2, which receive inputs Input_1 and Input_2, respectively (e.g., the input nodes of layer InL′ receive two-dimensional input vectors). The input layer InL′ feeds forward into one hidden layer HL1′ with two nodes h1 and h2, which in turn feeds forward into an output layer OutL′ with two nodes o1 and o2. The interconnections, or links, between neurons (shown as solid arrows for illustrative purposes) have weights w1 through w8. Typically, except for the input layer, a node (neuron) may receive as input the output of a node in the layer immediately preceding it. Each node may calculate its output by multiplying each of its inputs by each input's corresponding interconnection weight, adding the products of the inputs, adding (or multiplying) a constant defined by other weights or biases that may be associated with that particular node (e.g., node weights w9, w10, w11, w12 corresponding to nodes h1, h2, o1, and o2, respectively), and then applying a nonlinear or logarithmic function to the result. The nonlinear function may be referred to as an activation function or a transfer function. Multiple activation functions are known in the art, and the selection of a particular activation function is not important to this discussion. However, it should be noted that the operation of an ML model, the behavior of a neural net depends on the values ​​of the weights, which the neural network may be trained to provide a desired output for a given input.

[0101] During a training, or learning, phase, a neural net learns (e.g., is trained to identify) appropriate weight values ​​to achieve a desired output for a given input. Before a neural net is trained, each weight may be individually assigned an initial (e.g., random, optionally non-zero) value, e.g., a random number seed. Various methods for assigning initial weights are known in the art. The weights are then trained (optimized) so that, for a given training vector input, the neural network produces an output that is close to a desired (predetermined) training vector output. For example, the weights may be gradually adjusted over thousands of iterative cycles by a method called backpropagation. In each backpropagation cycle, a training input (e.g., a vector input or training input image / sample) is passed forward through the neural network to provide its actual output (e.g., a vector output). An error for each output neuron, or output node, is then calculated based on the actual neuron's output and the supervised training output for that neuron (e.g., a training output image / sample that corresponds to the current training input image / sample). It then propagates backwards through the neural network (from the output layer back to the input layer) and updates the weights based on how much influence each weight has on the overall error, so that the output of the neural network approaches the desired training output. This cycle is then repeated until the actual output of the neural network is within an acceptable error range of the desired training output for that training input. As will be appreciated, each training input may require many backpropagation iterations to achieve the desired error range. Typically, an epoch refers to one backpropagation iteration (e.g., one forward pass and one backward pass) of all training samples, and many epochs may be required to train a neural network. In general, the larger the training set, the better the performance of the trained ML model, so various data augmentation methods may be used to increase the size of the training set.For example, if the training set includes pairs of corresponding training input images and training output images, the training images may be divided into multiple corresponding image segments (or patches). Corresponding patches from the training input images and the training output images may be paired to define multiple training patch pairs from one input / output image pair, thereby expanding the training set. However, training a large training set increases the demands on computer resources, such as memory and data processing resources. The computational demands may be reduced by dividing the large training set into multiple mini-batches, the size of which determines the number of training samples in one forward / backward pass. In this case, and one epoch may include multiple mini-batches. Another problem is the possibility that the NN may overfit the training set, reducing its ability to generalize from a particular input to different inputs. The problem of overfitting may be reduced by creating an ensemble of neural networks or by randomly dropping out nodes in the neural network during training, which effectively removes the dropped leads from the neural network. Various dropout adjustment methods are known in the art, such as inverse dropout.

[0102] It should be noted that the computation of the trained NN machine model is not a simple algorithm of computation / analysis steps. Indeed, when the trained NN machine model receives an input, the input is not analyzed in the traditional sense. Rather, regardless of the subject or nature of the input (e.g., vectors defining a live image / scan, or vectors defining any other entity such as a demographic description or activity record), the input is subject to the same architectural construction of the trained neural network (e.g., the same node / layer arrangement, trained weights and bias values, predetermined convolution / deconvolution operations, activation functions, pooling operations, etc.), and it may not be obvious how the architectural construction of the trained network generates its output. Furthermore, the values ​​of the trained weights and biases are not deterministic and depend on many factors, such as the amount of time given to the neural network for training (e.g., the number of epochs in training), the random starting values ​​of the weights before training begins, the computer architecture of the machine on which the NN is trained, the choice of training samples, the distribution of the training samples among multiple mini-batches, the choice of activation function, the choice of error function that modifies the weights, and even whether training is interrupted on one machine (e.g., with a first computer architecture) and completed on another machine (e.g., with a different computer architecture). The point is that the reason why a trained ML model arrived at a particular output is not obvious, and much research is currently being done to identify the factors on which the ML model bases its output. Thus, the processing of neural networks on live data cannot be reduced to a simple algorithm of steps. Rather, the operation depends on the training architecture, the training sample set, the training sequence, and various circumstances in the training of the ML model.

[0103] In summary, the construction of a NN machine learning model may include a learning (or training) stage and a classification (or computation) stage. In the learning stage, a neural network may be trained for a specific purpose and may be provided with a set of training examples, including training (sample) inputs and training (sample) outputs, and optionally a set of validation examples to test the progress of the training. During this learning process, various weights associated with the nodes and node interconnections in the neural network are gradually adjusted to reduce the error between the actual output of the neural network and the desired training output. In this way, a multi-layer feedforward neural network (such as those described above) may be able to approximate any measurable function to any desired accuracy. The result of the learning stage is a learned (e.g., trained) (neural network) machine learning (ML). In the computation stage, a set of test inputs (or live inputs) may be provided to the learned (trained) ML model, which may apply what it has learned to generate output predictions based on the test inputs.

[0104] Similar to the regular neural networks of Fig. 14 and Fig. 15, convolutional neural networks (CNNs) are also composed of neurons with learnable weights and biases. Each neuron receives an input and performs an operation (e.g., a dot product), optionally followed by a nonlinear transformation. However, CNNs receive raw image pixels at one end (e.g., the input end) and provide classification (or class) scores at the other end (e.g., the output end). Because CNNs expect images as input, they are optimized to handle volumes (e.g., the pixel height and width of the image, and the depth of the image, e.g., color depth, such as RGB depth defined by three colors, red, green, and blue). For example, layers of CNNs may be optimized for neurons arranged in three dimensions. Neurons in a CNN layer may be connected to a small region of the previous layer, rather than all of the neurons of a fully connected NN. The final output layer of a CNN may reduce the full image to one vector (classification) arranged along the depth dimension.

[0105] FIG. 16 provides an exemplary convolutional neural network architecture. A convolutional neural network may be defined as a sequence of two or more layers (e.g., layer 1 to layer N), where a layer may include a (image) convolution step, a (result) weighted sum step, and a nonlinear function step. The convolution may be performed on the input data by applying a filter (or kernel) on, for example, a moving window over the input data to generate a feature map. Each layer and layer component may have different predefined filters (from a filter bank), weights (or weighting parameters), and / or function parameters. In this example, the input data is an image of a certain pixel height and width, and may be the raw pixel values ​​of this image. In this example, the input image is depicted to have a depth of three color channels RGB (red, green, blue). Optionally, various preprocessing may be performed on the input image, and the results of the preprocessing may be input instead of or in addition to the raw image data. Some examples of image processing may include retinal vessel map segmentation, color space conversion, adaptive histogram equalization, connected component generation, etc. Within a layer, a dot product may be calculated between certain weights and the small regions to which they are connected in the input volume. While many methods for constructing CNNs are known in the art, by way of example, layers may be constructed to apply element-wise activation functions, such as a max(0,x) threshold at zero. A pooling function may be performed (e.g., along the xy direction) to downsample the volume. A fully connected layer may be used to identify classification outputs and generate one-dimensional output vectors, which have proven useful for image recognition and classification. However, for image segmentation, a CNN needs to classify each pixel. Since each CNN layer tends to reduce the resolution of the input image, another stage is needed to upsample the image to its original resolution. This may be achieved by application of a transposed convolution (or deconvolution) stage TC, which typically does not use any predefined interpolation method, but instead has learnable parameters.

[0106] Convolutional neural networks have been successfully applied to many problems in computer vision. As mentioned above, training a CNN generally requires a large training dataset. The U-Net architecture is based on a CNN and can generally be trained with a smaller training dataset than a traditional CNN.

[0107] FIG. 17 illustrates an exemplary U-Net architecture. This exemplary U-Net includes an input module (or input layer or stage), which receives an input U-in (e.g., an input image or image patch) of any size. For convenience, the image size at any stage or layer is indicated within a box representing the image, e.g., in the input module, the numbers "128x128" are enclosed to indicate that the input image U-in is composed of 128x128 pixels. The input image may be a fundus image, an OCT / OCTA en face, a B-scan image, etc. However, it should be understood that the input may be of any size or dimension. For example, the input image may be an RGB color image, a monochrome image, a volumetric image, etc. The input image passes through a series of processing layers, each of which is illustrated with exemplary sizes, but these sizes are for illustrative purposes only and will depend on, for example, the size of the image, the convolution filters, and / or the pooling stages. The architecture consists of a convergent path (herein illustratively including four encoding modules) followed by an augmented path (herein illustratively including four decoding modules) with copy-and-crop links (e.g., CC1-CC4) between the corresponding modules / stages that copy the output of one encoding module in the convergent path and combine (e.g., append) it to the upconverted input of the corresponding decoding module in the augmented path. The result is a characteristic U-shape from which the architecture is named. Optionally, for computational considerations or the like, a "bottleneck" module / stage (BN) can be placed between the convergent path and the augmented path. The bottleneck BN may consist of two convolutional layers (with batch normalization and optional dropout).

[0108] Convergent paths are similar to encoders, and typically use feature maps to capture context (or feature) information. In this example, each encoding module in the convergent path includes two or more convolutional layers, indicated by an asterisk symbol "*", which may be followed by a max pooling layer (e.g., a downsampling layer). For example, an input image U-in is shown going through two convolutional layers, each with 32 feature maps. It can be understood that each convolutional kernel produces a feature map (e.g., the output from a convolution operation with a given kernel is an image, commonly referred to as a "feature map"). For example, the input U-in goes through a first convolution that applies 32 convolutional kernels (not shown), producing an output consisting of 32 individual feature maps. However, as is known in the art, the number of feature maps produced by a convolution operation can be adjusted (upwards or downwards). For example, the number of feature maps can be reduced by averaging groups of feature maps, removing some feature maps, or other known methods of reducing feature maps. In this example, this first convolution is followed by a second convolution whose output is limited to 32 feature maps. Another way to envision the feature maps is to consider the output of the convolution layer as a 3D image whose 2D dimensions are given by the described XY plane pixel dimensions (e.g., 128×128 pixels) and whose depth is given by the number of feature maps (e.g., the depth of the 32 plane images). Following this illustration, the output of the second convolution (e.g., the output of the first encoding module of the convergence path) can be described as a 128×128×32 image. The output from the second convolution is then subjected to a pooling operation, which reduces the 2D dimensions of each feature map (e.g., the X and Y dimensions can each be reduced by half). The pooling operation can be embodied within a downsampling process, as indicated by the downward arrow. Several pooling methods, such as max pooling, are known in the art, and the particular pooling method is not critical to the present invention.The number of feature maps doubles with each pooling: 32 feature maps in the first encoding module (or block), 64 feature maps in the second encoding module, etc. Thus, the convergent path forms a convolutional network composed of multiple encoding modules (or stages or blocks). As is typical for convolutional networks, each encoding module may provide at least one convolution stage followed by an activation function (e.g., a rectified linear unit (ReLU) or sigmoid layer) (not shown), and a max-pooling operation. In general, the activation function introduces nonlinearity in the layer (e.g., to avoid overfitting problems), receives the results of the layer, and decides whether to "activate" the output (e.g., decides whether the value in a particular node meets a predefined criterion to forward the output to the next layer / node). In summary, the convergent path generally reduces spatial information and increases feature information.

[0109] The extension path is similar to the decoder, notably providing localization, and spatial information to the results of the convergence path, despite the downsampling and any max pooling performed in the contraction stage. The extension path includes multiple decoding modules, each of which combines its current upconverted input with the output of the corresponding encoding module. Thus, features and spatial information are combined in the extension path through a series of upconvolutions (e.g., upsampling or transposed convolutions, i.e., deconvolutions) and combinations (e.g., via CC1-CC4) with high-resolution features from the convergence path. Thus, the output of the deconvolution layer is combined with the corresponding (optionally cropped) feature map from the convergence path, followed by two convolution layers and activation functions (optionally batch normalized).

[0110] The output from the last augmentation module in the augmentation path may be fed to other processing / training blocks or layers, such as a classifier block, which may be trained together with the U-Net architecture. Alternatively, or additionally, the output of the last upsampling block (at the end of the augmentation path) may be subjected to another convolution (e.g., output convolution) operation, as indicated by the dotted arrow, before generating its output U-out. The kernel size of the output convolution may be selected to reduce the dimensions of the last upsampling block to a desired size. For example, a neural network may have multiple features per pixel just before reaching the output convolution, which may provide a 1×1 convolution operation that combines these multiple features into a single output value per pixel at the per-pixel level.

[0111] Computing Devices / Systems FIG. 18 illustrates an exemplary computer system (or computing device or computer device). In some embodiments, one or more computer systems may provide functionality described or illustrated herein and / or perform one or more steps of one or more methods described or illustrated herein. A computer system may take any suitable physical form. For example, a computer system may be an embedded computer system, a system on a chip (SOC), or a single board computer system (SBC) (e.g., a computer on module (COM) or a system on module (SOM)), a desktop computer system, a laptop or notebook computer system, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these. Where appropriate, a computer system may be in a cloud, which may include one or more cloud components in one or more networks.

[0112] In some embodiments, the computer system may include a processor Cpnt1, a memory Cpnt2, a storage Cpnt3, an input / output (I / O) interface Cpnt4, a communication interface Cpnt5, and a bus Cpnt6. The computer system may also optionally include a display Cpnt7, such as a computer monitor or screen.

[0113] The processor Cpnt1 includes hardware for executing instructions, such as those that constitute a computer program. For example, the processor Cpnt1 may be a central processing unit (CPU) or a general-purpose computing-on-graphics processing unit (GPGPU). The processor Cpnt1 may read (or fetch) instructions from an internal register, an internal cache, a memory Cpnt2, or a storage Cpnt3, decode and execute the instructions, and write one or more results to the internal register, the internal cache, the memory Cpnt2, or the storage Cpnt3. In a particular embodiment, the processor Cpnt1 may include one or more internal caches for data, instructions, or addresses. The processor Cpnt1 may include one or more instruction caches, one or more data caches, for example to hold data tables. Instructions in the instruction caches may be copies of instructions in the memory Cpnt2 or the storage Cpnt3, and the instruction caches may speed up the retrieval of these instructions by the processor Cpnt1. The processor Cpnt1 may include any suitable number of internal registers and may include one or more arithmetic logic units (ALUs). The processor Cpnt1 may be a multi-core processor or may include one or more processors Cpnt1. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0114] The memory Cpnt2 may include a main memory that stores instructions for the processor Cpnt1 to execute processing or to hold intermediate data during processing. For example, the computer system may load instructions or data (e.g., data tables) from the storage Cpnt3 or from other sources (e.g., other computer systems) into the memory Cpnt2. The processor Cpnt1 may load instructions and data from the memory Cpnt2 into one or more internal registers or internal caches. To execute an instruction, the processor Cpnt1 may read and decode an instruction from the internal register or internal cache. During or after the execution of an instruction, the processor Cpnt1 may write one or more results (which may be intermediate or final results) to the internal registers, the internal cache, the memory Cpnt2, or the storage Cpnt3. The bus Cpnt6 may include one or more memory buses (which may each include an Azures bus and a data bus) and may couple the processor Cpnt1 to the memory Cpnt2 and / or the storage Cpnt3. Optionally, one or more memory management units (MMUs) facilitate data transfer between the processor Cpnt1 and the memory Cpnt2. The memory Cpnt2 (which may be a high-speed volatile memory) may include a random access memory (RAM), such as a dynamic RAM (DRAM) or a static RAM (SRAM). The storage Cpnt3 may include a long-term or large-capacity storage for data or instructions. The storage Cpnt3 may be internal or external to the computer system and may include one or more of a disk drive (e.g., a hard disk drive HDD, or a solid-state drive SSD), a flash memory, a ROM, an EPROM, an optical disk, a magnetic optical disk, a magnetic tape, a universal serial bus (USB)-accessible drive, or other types of non-volatile memory.

[0115] The I / O interface Cpnt4 may be software, hardware, or a combination of both, and may include one or more interfaces (e.g., serial or parallel communication ports) for communicating with I / O devices, which may enable communication with a human (e.g., a user). For example, the I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, table, touch screen, trackball, video camera, other suitable I / O devices, or a combination of two or more thereof.

[0116] The communication interface Cpnt5 may provide a network interface for communicating with other systems or networks. The communication interface Cpnt5 may include a Bluetooth interface or other types of packet-based communication. For example, the communication interface Cpnt5 may include a network interface controller (NIC) and / or a wireless NIC or wireless adapter for communication with a wireless network. The communication interface Cpnt5 may provide communication with a WI-FI network, an ad-hoc network, a personal area network (PAN), a wireless PAN (e.g., Bluetooth WPAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a mobile phone network (e.g., a Global System for Mobile Communications (GSM) network, etc.), the Internet, or a combination of two or more of these.

[0117] The bus Cpnt6 may provide a communication link between the above-mentioned components of the computing system. For example, bus Cpnt6 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand bus, a low-pin-count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or any other suitable bus, or a combination of two or more thereof.

[0118] Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0119] As used herein, a computer-readable non-transitory storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage medium, or any suitable combination of two or more thereof, as appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, as appropriate.

[0120] While the present invention has been described in conjunction with several specific embodiments, as will be apparent to those skilled in the art in light of the foregoing description, many other alternatives, modifications, and variations will be apparent. Accordingly, the invention described herein is intended to embrace all such alternatives, modifications, applications, and variations that may fall within the spirit and scope of the appended claims.

Claims

Claim 1 A method for performing a transaction of medical data, comprising: obtaining consent from the owner of the medical data, and a management entity receiving the medical data for storage and recording a summary of the received medical data in a blockchain computer network that maintains an electronic ledger of available medical records; the management entity responding to an electronic request from a remote user for medical data meeting user-specified criteria by providing the remote user with a list of available eligible medical records meeting the user-specified criteria, at least partially determined from the electronic ledger; the management entity responding to the selection by the remote user of one or more of the eligible medical records by permitting the remote user to access the selected eligible medical records according to the access approval status for each eligible medical record granted by the owner of the eligible medical record, and recording the data transaction in the blockchain computer network. Claim 2 The management entity further receives from the remote user an access type request indicating the type of data access requested, wherein the access type request includes one or more of data viewing, temporary data access with an expiration date, permanent data access by data download, and one or more of data processing managed by the management entity remotely from the remote user, the method according to claim 1. Claim 3 The data processing managed by the management entity includes generating a machine learning model using the selected medical records and permitting the remote user to access the generated machine learning model, the method according to claim 2. Claim 4 The machine model includes a deep learning model selected from one or more of an artificial neural network, a convolutional neural network, a U-Net, a recurrent neural network, a generative adversarial network, and a multi-layer perceptron, and / or the machine model includes a machine learning model based on one or more of a classification model, a regression model, clustering, and dimensionality reduction, the method according to claim 3. Claim 5 The method according to claim 1, wherein the owner of the medical data is the patient described by the medical data.

6. The method according to claim 1, wherein the medical data includes medical measurement values or medical images acquired by a medical device, and the medical device automatically transmits the medical data to the management entity after obtaining consent from the patient.

7. The method according to claim 1, wherein the owner of the medical data is identified by a public identifier based on a public key associated with a corresponding private key, and the public identifier excludes personal identification information, whereby the management entity maintains the personal identification information of the owner of the medical data in a concealed state.

8. The management entity in response to a remote user selecting one or more of the eligible medical records, checks the existing access approval status for each of the selected eligible medical records, for each selected eligible medical record that does not have an existing access approval status, sends a request for approval to the owner of the eligible medical record, and updates the approval status of the eligible medical record according to the owner's approval response. The method according to claim 1.

9. The management entity in response to a remote user specifying a desired number of eligible medical records, checks the existing access approval status for each of the eligible medical records, in response to a sufficient number of eligible medical records having an existing access approval status, selects the desired number of eligible medical records from among those having an existing access approval status, in response to the number of eligible medical records having an existing access approval status being insufficient, determines the number of additional eligible medical records required to meet the specified desired number, and for each of the additionally required eligible medical records, sends a request for approval to the owner of the additionally required eligible medical record, and updates the approval status of the additionally required eligible medical record according to the owner's approval response. The method according to claim 1.

10. The method according to claim 1, wherein the blockchain computer network is a public ledger computer network.

11. The method according to claim 1, wherein the received medical data is stored at least partially on-chain within the blockchain computer network and off-chain outside the blockchain computer network.

12. The remote user is a doctor to whom a patient is referred, and in response to the user specification criteria specifying only the medical records of the patient, the management entity maintains an automatic access approval status for the eligible medical records, according to the method of claim 1.

13. The method according to claim 1, wherein the management entity maintains a group of privileged remote users each having unobstructed viewing access to the electronic ledger.

14. The user specification criteria submitted by a remote user includes one or more of the type of anatomical measurement, body function measurement, data scan type, and image type, according to the method of claim 1.

15. The anatomical measurement includes one or more of intraocular pressure, corneal curvature measurement, refractive error, or eye size, The body function measurement includes one or more of electrocardiogram, body temperature, pulse rate, or respiratory rate, The data scan type includes one or more of an A-scan, B-scan, or C-scan from an optical coherence tomography device, The method according to claim 14, wherein the image type includes one or more of a fundus image, an en-face image, or an anterior segment image.

Images