Methods and systems for annotating anatomical images

Abstract

Systems and methods that enable radiological image analysis to be performed without installing complex software in hospital systems and without transferring sensitive patient data beyond secure networks are provided. Annotating one or more anatomical images of a subject at an endpoint device includes receiving a query from a user defining the subject, retrieving the anatomical images from a first datasource comprised in a secured environment also comprising the endpoint device, retrieving image-related data comprising a set of findings from a second datasource, wherein the image-related data is free of protected health information, displaying the anatomical images overlaid with the findings, providing an interface for the user to edit and / or validate the findings, thereby updating the image-related data, and sending the updated image-related data to the second datasource.

Description

METHODS AND SYSTEMS FOR ANNOTATING ANATOMICAL IMAGESCROSS-REFERENCE TO RELATED APPLICATION

[0000] This application claims the benefit of, and priority to, United States Provisional Patent Application No. 63 / 689,417, filed August 30, 2024, and entitled “Methods and Systems for annotating anatomical Images”, the disclosure of which is hereby incorporated by reference in its entirety.TECHNICAL FIELD

[0001] The technical field relates to medical imaging, and more specifically to systems and methods for annotating anatomical images of a subject.BACKGROUND

[0002] Artificial intelligence offers great opportunity in radiology to improve workflows. However adoption has been very slow. Current technical frameworks to deploy solutions in clinical settings such as hospitals have important implications in terms of information technology infrastructure.

[0003] Traditionally, third-party vendors deploy software solutions on premises and either provide their own hardware or reuse available server infrastructure. This type of deployment can be complex and costly. For instance, information technology services may need to be involved in configuring networking, cybersecurity audits may be called for, and vendors may require remote or physical access to perform maintenance. Additionally, the hospital may get the burden of provisioning and managing hardware, including radiology data storage, which is expensive. Furthermore, this approach is hostile to small vendors and application providers.

[0004] Although cloud solutions can be offered, they involve sending medical images, carefully stripped of protected health information, to a vendor cloud infrastructure to process images. Analysis result can then be sent back in the form of DICOM™ reports and annotations. These solutions also have severaldownsides. Transferring images can be slow, since series can amount to several hundreds of megabytes, and medical images needing to leave the hospital premises can have important legal implications. Several hospital centres are not ready to take this leap.SUMMARY

[0005] There remains a need for solutions that allow for the deployment of artificial intelligence tools in radiology that do not imply clogging limited information technology resources of health providers, yet do not involve sensitive data such as protected health information being sent outside their secure environment.

[0006] Systems and methods that enable radiological image analysis to be performed without installing complex software in hospital systems and without transferring sensitive patient data beyond secure networks are provided. The systems and methods can allow findings generated by artificial intelligence tools to be separated from protected health information, so that results can be stored and accessed externally while patient data can remain safely within the hospital environment. Accordingly, the disclosed systems and methods make it possible to store data on two distinct datasources. As an example, a first datasource can be hosted in the secure environment of a health provider and used to store sensitive data, including for instance protected health information and medical images, and a distinct second datasource can be hosted outside the environment, for instance in the cloud, and used to store image-related data, including for instance findings, e.g., contours of lesions identified in the medical images, without storing the medical images. This architecture reduces the technical burden on healthcare providers, improves sensitive data protection, and facilitates broader adoption of artificial intelligence solutions in radiology.

[0007] In accordance with an aspect, a system for annotating one or more anatomical images of a subject is provided. The system includes at least one input device and at least one output device, at least one communication device configured for retrieving the anatomical images from a first datasource included ina secured environment also including the system and sending and / or retrieving image-related data to and / or from a second datasource, the second datasource being distinct from the first datasource, wherein the image-related data includes at least a set of findings, and wherein the image-related data is free of protected health information, at least one memory configured to store the anatomical images, the image-related data and instructions implementing a viewer application, and at least one processor configured to run the viewer application, causing the system to receive from a user via the input device a query defining the subject, display on the output device the anatomical images overlaid with indications of the findings and allow the user to edit and / or validate the findings, thereby updating the image- related data.

[0008] In some embodiments, the anatomical images are radiographic images of an organ of the subject.

[0009] In some embodiments, the anatomical images are radiographic images of the brain of the subject.

[0010] In some embodiments, the communication device is a network interface.

[0011] In some embodiments, the first datasource is a picture archiving and communication system.

[0012] In some embodiments, the communication device is configured to connect to the first datasource using a DICOMweb™ service.

[0013] In some embodiments, the second datasource is hosted in a cloud.

[0014] In some embodiments, the second datasource is outside of the secure environment.

[0015] In some embodiments, the second datasource is protected by an authentication mechanism and / or by an authorization mechanism.

[0016] In some embodiments, each finding corresponds to a lesion detected in at least one of the anatomical images.

[0017] In some embodiments, each finding correspond to a tumour.

[0018] In some embodiments, each finding corresponds to a metastasis.

[0019] In some embodiments, each finding is defined by a contour.

[0020] In some embodiments, the system further includes an inference module configured, in response to the set of findings being empty, to detect findings and to update the set of findings with the detected findings.

[0021] In some embodiments, the system further includes a neural network trained for detecting and contouring structures of interest in anatomical images.

[0022] In some embodiments, the one or more anatomical images includes a plurality of subsets of images, wherein each subset of images includes anatomical images of a generally equivalent area of the subject captured at different times.

[0023] In some embodiments, the system further includes a transform defining a mapping of pixels or voxels of a reference subset of images to corresponding pixels or voxels to other subsets of images.

[0024] In some embodiments, the transform is an affine and / or rigid transform.

[0025] In some embodiments, the system further includes a registration module configured to compute the transform.

[0026] In some embodiments, the image-related data includes the transform.

[0027] In some embodiments, the findings are assigned finding identifiers to link corresponding findings detected across two or more of the subsets of images, allowing for tracking the findings between the subsets of images.

[0028] In some embodiments, the system further includes a longitudinal module configured to assign the finding identifiers.

[0029] In some embodiments, the image-related data includes the finding identifiers.

[0030] In some embodiments, the viewer application causes the system to allow the user to rename, modify and / or delete a finding from the set of findings.

[0031] In some embodiments, the viewer application causes the system to allow the user to create a new finding and insert the new finding in the set of findings.

[0032] In some embodiments, the viewer application is configured, in response to being shut down, to remove the anatomical images and / or the image-related data from the memory.

[0033] In some embodiments, the viewer application is configured to run in a web browser.

[0034] In some embodiments, the viewer application is a WebAssembly application.

[0035] In accordance with another aspect, a computer-implemented method for annotating one or more anatomical images of a subject at an endpoint device is provided. The method includes receiving a query from a user defining the subject, retrieving the anatomical images from a first datasource included in a secured environment also including the endpoint device, retrieving image-related data including a set of findings from a second datasource, the second datasource being distinct from the first datasource, wherein the image-related data is free of protected health information, displaying the anatomical images overlaid with the findings, providing an interface for the user to edit and / or validate the findings, thereby updating the image-related data, and sending the updated image-related data to the second datasource.

[0036] In some embodiments, the anatomical images are radiographic images of an organ of the subject.

[0037] In some embodiments, the anatomical images are radiographic images of the brain of the subject.

[0038] In some embodiments, the first datasource is a picture archiving and communication system.

[0039] In some embodiments, retrieving the anatomical images from a first datasource includes connecting to the first datasource using a DICOMweb™ service.

[0040] In some embodiments, the second datasource is hosted in a cloud.

[0041] In some embodiments, the second datasource is outside of the secure environment.

[0042] In some embodiments, the second datasource is protected by an authentication mechanism and / or by an authorization mechanism.

[0043] In some embodiments, each finding corresponds to a lesion detected in at least one of the anatomical images.

[0044] In some embodiments, each finding correspond to a tumour.

[0045] In some embodiments, each finding corresponds to a metastasis.

[0046] In some embodiments, each finding is defined by a contour.

[0047] In some embodiments, the method further includes, in response to the set of findings being empty, detecting findings and updating the set of findings with the detected findings.

[0048] In some embodiments, detecting findings includes using a neural network trained for detecting and contouring structures of interest in anatomical images.

[0049] In some embodiments, the one or more anatomical images includes a plurality of subsets of images, wherein each subset of images includes anatomical images of a generally equivalent area of the subject captured at different times.

[0050] In some embodiments, the method further includes computing a transform defining a mapping of pixels or voxels of a reference subset of images to corresponding pixels or voxels to other subsets of images.

[0051] In some embodiments, the transform is an affine and / or rigid transform.

[0052] In some embodiments, the image-related data includes the transform.

[0053] In some embodiments, the method further includes assigning finding identifiers to the findings to link corresponding findings detected across two or more of the subsets of images, allowing for tracking the findings between the subsets of images.

[0054] In some embodiments, the image-related data includes the finding identifiers.

[0055] In some embodiments, the interface further allows the user to rename, modify and / or delete a finding from the set of findings.

[0056] In some embodiments, the interface further allows the user to create a new finding and insert the new finding in the set of findings.

[0057] In some embodiments, the method further includes, in response to the interface being shut down, removing the anatomical images and / or the image- related data from memory of the endpoint device.

[0058] In some embodiments, the interface is a web interface.

[0059] In some embodiments, the web interface is implemented as a WebAssembly application.

[0060] In accordance with a further aspect, a computer-readable memory having recorded thereon instructions, the instructions, when executed by at least one processor of an endpoint device, causing the at least one processor to receive for a query from a user defining the subject, retrieve the anatomical images from a first datasource included in a secured environment also including the endpoint device, retrieve image-related data including a set of findings from a second datasource, the second datasource being distinct from the first datasource, wherein the image-related data is free of protected health information, display the anatomical images overlaid with the findings, provide an interface for the user to edit and / or validate the findings, thereby updating the image-related data, and send the updated image-related data to the second datasource.

[0061] In some embodiments, the anatomical images are radiographic images of an organ of the subject.

[0062] In some embodiments, the anatomical images are radiographic images of the brain of the subject.

[0063] In some embodiments, the first datasource is a picture archiving and communication system.

[0064] In some embodiments, retrieving the anatomical images from a first datasource includes connecting to the first datasource using a DICOMweb™ service.

[0065] In some embodiments, the second datasource is hosted in a cloud.

[0066] In some embodiments, the second datasource is outside of the secure environment.

[0067] In some embodiments, the second datasource is protected by an authentication mechanism and / or by an authorization mechanism.

[0068] In some embodiments, each finding corresponds to a lesion detected in at least one of the anatomical images.

[0069] In some embodiments, each finding correspond to a tumour.

[0070] In some embodiments, each finding corresponds to a metastasis.

[0071] In some embodiments, each finding is defined by a contour.

[0072] In some embodiments, the instructions further cause the at least one processor to, in response to the set of findings being empty, detecting findings and updating the set of findings with the detected findings.

[0073] In some embodiments, detecting findings includes using a neural network trained for detecting and contouring structures of interest in anatomical images.

[0074] In some embodiments, the one or more anatomical images includes a plurality of subsets of images, wherein each subset of images includes anatomical images of a generally equivalent area of the subject captured at different times.

[0075] In some embodiments, the instructions further cause the at least one processor to compute a transform defining a mapping of pixels or voxels of a reference subset of images to corresponding pixels or voxels to other subsets of images.

[0076] In some embodiments, the transform is an affine and / or rigid transform.

[0077] In some embodiments, the image-related data includes the transform.

[0078] In some embodiments, the instructions further cause the at least one processor to assign finding identifiers to the findings to link corresponding findings detected across two or more of the subsets of images, allowing for tracking the findings between the subsets of images.

[0079] In some embodiments, the image-related data includes the finding identifiers.

[0080] In some embodiments, the interface further allows the user to rename, modify and / or delete a finding from the set of findings.

[0081] In some embodiments, the interface further allows the user to create a new finding and insert the new finding in the set of findings.

[0082] In some embodiments, the instructions further cause the at least one processor to, in response to the interface being shut down, remove the anatomical images and / or the image-related data from memory of the endpoint device.

[0083] In some embodiments, the interface is a web interface.

[0084] In some embodiments, the web interface is implemented as a WebAssembly application.BRIEF DESCRIPTION OF THE DRAWINGS

[0085] For a better understanding of the embodiments described herein and to show more clearly how they may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings which show at least one exemplary embodiment.

[0086] Figures 1A, 1 B and 1 C are schematic of systems for annotating anatomical images of a subject, in accordance with three embodiments.

[0087] Figure 2 is a flowchart of a method for annotating anatomical images of a subject, in accordance with an embodiment.DETAILED DESCRIPTION

[0088] It will be appreciated that, for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements or steps. In addition, numerous specific details are set forth in order to provide a thorough understanding of the exemplary embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practised without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure theembodiments described herein. Furthermore, this description is not to be considered as limiting the scope of the embodiments described herein in any way but rather as merely describing the implementation of the various embodiments described herein.

[0089] One or more systems and methods described herein may be implemented in computer program(s) executed on processing device(s), each comprising at least one processor, a data storage system (including volatile and / or non-volatile memory and / or storage elements), and optionally at least one input and / or output device. “Processing devices” encompass computers, servers and / or specialized electronic devices which receive, process and / or transmit data. As an example, “processing devices” can include processing means, such as microcontrollers, microprocessors, and / or CPUs, or be implemented on FPGAs. For example, and without limitation, a processing device may be a programmable logic unit, a mainframe computer, a server, a personal computer, a cloud-based program or system, a laptop, a personal data assistant, a cellular telephone, a smartphone, a wearable device, a tablet, a video game console or a portable video game device.

[0090] Each program is preferably implemented in a high-level programming and / or scripting language, for instance an imperative e.g., procedural or object- oriented, or a declarative e.g., functional or logic, language, to communicate with a computer system. However, a program can be implemented in assembly or machine language if desired. In any case, the language may be a compiled or an interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein. In some embodiments, the system may be embedded within an operating system running on the programmable computer.

[0091] Furthermore, the system, processes and methods of the described embodiments are capable of being distributed in a computer program productcomprising a computer readable medium that bears computer-usable instructions for one or more processors. The computer-usable instructions may also be in various forms, including compiled and non-compiled code.

[0092] The processor(s) are used in combination with storage medium, also referred to as “memory” or “storage means”. Storage medium can store instructions, algorithms, rules and / or trading data to be processed. Storage medium encompasses volatile or non-volatile / persistent memory, such as registers, cache, RAM, flash memory, ROM, diskettes, compact disks, tapes, chips, as examples only. The type of memory is, of course, chosen according to the desired use, whether it should retain instructions, or temporarily store, retain or update data. Steps of the proposed method are implemented as software instructions and algorithms, stored in computer memory and executed by processors.

[0093] In the present disclosure, the term “set” is intended to be used in its traditional, set-theoretical meaning. In other words, a set of elements can contain any number of elements of the same categories. In particular, a set can contain no element, in which case it can be designated as an “empty set”, a set can contain one element, in which can it can be designated as a “singleton”, and a set can contain a plurality of elements. A set can further be modified by the insertion of an element or by the union with a second set of elements of the same category, or by the removal of an element or by the subtraction of a third set of elements of the same category.

[0094] In the present disclosure, the expression “protected health information” (PHI) is to be understood in a broad and non-limitative sense, encompassing any data that relates to the health status, provision of healthcare, or payment for healthcare that can be linked, directly and / or indirectly, to an individual. PHI can include, without limitation, identifiers such as names, addresses, dates, telephone numbers, email addresses, social security or insurance numbers, medical record numbers, biometric identifiers, photographic images and / or any uniquecharacteristics or codes that could permit identification. PHI can further encompass clinical information such as radiological images themselves, laboratory results, diagnoses, treatment plans, prescriptions, billing information and / or metadata associated with the foregoing. In some embodiments, the definition of PHI is aligned with applicable legislation, including for example the Health Insurance Portability and Accountability Act of 1996 (HIPAA) in the United States, the General Data Protection Regulation (GDPR) in the European Union, the Personal Information Protection and Electronic Documents Act (PIPEDA) in Canada, and / or any equivalent or comparable privacy or health information protection statutes. For the purpose of the present disclosure, PHI can be construed as including any data element that would reasonably be considered sensitive in a healthcare context and that, alone and / or in combination with other data, could permit re-identification of a patient.

[0095] With reference to figure 1 A, an exemplary environment 1 a of a system 100 for annotating anatomical images of a subject is shown. Anatomical images can correspond to medical images. Broadly described, environment 1 a includes the system 100 running a viewer which accesses datasources 10 and 20.

[0096] Environment 1 a includes an annotation system 100, for instance corresponding to an endpoint device, inside a secure environment 40, for instance the secure environment IT (information technology) environment of a health service provider such as a hospital. The system 100 includes a viewer configured by one or more datasources. The system 100 can leverage recent technological developments that open the door to delivering artificial intelligence (Al) applications to radiology department with minimal burden to IT services and resources, and minimal cybersecurity risks.

[0097] In some embodiments, the system 100 implements DICOMweb™, a standard allowing Digital Imaging and Communications in Medicine (DICOM™) images to be transferred using general-purpose data transfer protocols such as an Hypertext Transfer Protocol (HTTP) protocol, instead of, for instance, the legacy,TCP-based, DICOM™ Message Service Element (DIMSE) protocol. Using a modem, general-purpose data transfer protocol such as HTTP allows the viewer to implement secure data transfers, for instance using the Transport Layer Security (TLS), the Datagram Transport Layer Security (DTLS) and / or the Secure Sockets Layer (SSL) cryptographic protocols. Further, it makes it possible for the system 100 to securely access remote resources, rather than unsecured resources located within secure environment 40.

[0098] Environment 1 a can include and / or implement a Picture Archiving and Communication System (PACS), i.e. , a medical imaging platform that can be used to securely store, retrieve, manage, distribute and / or present radiological images and related information in digital form. In some embodiments (not illustrated), environment 1 a and the system 100 implement cloud-based PACS solutions. Modem medicine increasingly relies on radiology, which has put an increasing burden on the IT departments of health service providers, which are having a hard time facing the demand. An alternative to maintaining infrastructure internally is to migrate to a cloud provider. Therefore, in some embodiments, anatomical images are stored partly or completely remotely, and accessed by a local viewer such as viewer 100. As examples, companies such as Google™ and Amazon™ provide cloud-based PACS offerings.

[0099] In some embodiments, the system 100 implements WebAssembly (Wasm). Wasm defines a portable, low-level binary instruction format, which makes it possible for applications such as web browsers to execute binaries instead of merely interpreting scripts such as JavaScript™ and / or ECMAScript™ scripts, drastically improving computational performance. Several system-level programming languages such as C++ and Rust can be compiled to Wasm, effectively porting existing image processing libraries to the browser. Furthermore, Wasm makes it possible to implement machine learning (ML) model deployment platforms, such as ONNX™ WebRuntime, which makes it possible to execute trained models, for instance deep learning models, in web browsers, providing efficient in-browser inference.

[0100] In some embodiments, the viewer of the system 100 implements existing solutions that provide for rendering anatomical images through Hypertext Markup Language 5 (HTML5). The combination of Wasm, DicomWeb™ and / or cloudbased PACS has led to the development of zero-footprint HTML5 medical image viewers, which can be implemented in the viewer. One such image viewer is the Open Health Imaging Foundation™ (OHIF) Viewer, an open-source project that allows the development of custom plugins, especially for Al applications, also commercialized as FlexView™. Other solutions include for instance the vtk.js™ Visualization Toolkit, and the Stone Web™ Viewer.

[0101] Different embodiments of environment 1a can implement different datasources. In some embodiments, a first data source corresponds to a PACS 10 configured to store the anatomical images in secure environment 40, which the system 100 can access through a local area network (LAN). In some embodiments, a second data source corresponds to a hosting infrastructure such as a cloud hosting infrastructure 20 configured to persist image-related data, e.g., in a database 22, which the system 100 accesses through a wide area network (WAN) such as the Internet 50. The combination of the hosting infrastructure 20 and its database 22 can also be named an “Al results server”.

[0102] Environment 1 a makes it possible to deploy Al-enhanced imaging software through a viewer of the system 100, which can be a web application executed by a web browser, by leveraging some or all of the technologies described above. Advantageously, because the software is web-based, it does not require installation or maintenance. In some embodiments, the system 100 can use DICOMweb™ to access images. In some embodiments, the system 100 can use a bridge application that provides RESTful application programming interface (API) to access to PACS resources, such as the Orthanc™ server. In some embodiments, the system 100 can use a locally executed, portable application, which serves as a datasource by loading local files from the endpoint device. The portable application can expose a local, DICOMWeb™-compatible interface to the browser. In some embodiments, a portable application can act as a proxy betweena legacy PACS. The portable application can be configured to accept incoming DICOMWeb™ requests and process them using, e.g., the DIMSE protocol. It can be appreciated that comparable solutions, that can be independent of the DICOM™ and / or DICOMWeb™ standards, are also suitable for storing and / or managing images for the operation of system 100. As examples only, the ANALYZE™, the Neuroimaging Informatics Technology Initiative (NlfTI) and / or the Metalmage Header (MHD) formats can be used in addition to or as an alternative to the DICOM™ format, and XDS-I™ and / or HL7™ FHIR™ ImagingStudy™ can be used in addition to or as an alternative to the DICOM™-defined protocols.

[0103] By packaging all image-processing modules required for detection and tracking directly within the web application executed by the endpoint device, all Al results are generated on-site, within the browser. While all DICOM™ image data can be discarded after processing, Al results, free of protected health information (PHI) can be persisted remotely to a remotely hosted server 22. In some embodiments, environment 1a is advantageously implemented such that medical images do not leave secure environment 40, e.g., do not leave the premises of the health service provider. When the viewer is invoked with a given DICOM™ series, the remote database 22 can be queried for any previously computed results, which are then downloaded and redisplayed within the viewer. The anatomical images can be fetched by the system 100 by connecting to a DICOMweb™ enabled PACS, which can be cloud-based and / or accessible via the LAN, or by directly loading files located at the endpoint device 100. In some embodiments, the main component of the software solution consists of a single page web application (SPA).

[0104] With reference to figure 1 B, another exemplary environment 1 b including a system 100 for annotating anatomical images of a subject is shown. Broadly describes, environment 1 b includes system 100, which can be implemented in an endpoint device such as an IT workstation, a first datasource 10 and a second datasource 20.

[0105] Environment 1 b includes a first datasource 10. The first datasource 10 includes a database 12 configured to store anatomical images 14. The anatomical images 14 can correspond to bi- or tridimensional images acquired via an image acquisition device. For example, the anatomical images 14 can be a medical images, for instance radiographic images of one or more organs of a subject, such as a magnetic resonance imaging (MRI) brain scan, although it is appreciated that other images are possible such as those acquired via electronic microscopy (EM), industrial computed tomography (CT), or other techniques for acquiring images of biological or non-biological specimens. The images can be received in various different formats, such as the DICOM™ format in the case of medical images. In some embodiments, a 3D image can be received as a plurality of 2D slices, defining an image series, and the 3D image can be reconstructed therefrom. In some embodiments, images or image series of a subject captures at different point in times can be combined as longitudinal series in database 12. The database 12 can correspond to a medical imaging platorm such as a PACS. Database 12 can be configured to allow image acquisition devices and / or endpoint devices such as endpoint device 100 to upload and / or download images and image-related data such as annotations, using any suitable protocol such as DIMSE™ or HTTP. The database 12 can be configured to store images and / or series of images in association with an identifier that uniquely identifies each image and / or series of images.

[0106] In some embodiments, the first datasource 10 is included in a secured environment 40 that also includes system 100. The secured environment can for instance include a LAN and / or a virtual private network (VPN), such that all communications between system 100 and datasource 10 are routed either through a network controlled by and / or located in the premises of an institution such as a health service provider, or are routed through secured communication links, e.g., implemented via a tunnelling protocol such as the Secure Socket Tunneling Protocol (SSTP), Generic Routing Encapsulation™ (GRE), or provided by Internet Protocol Security (IPsec). In some embodiments, secured environment 40 provides compliance with physical and technical safeguards mandated by relevantlaws and regulations such as the Health Insurance Portability and Accountability Act and the General Data Protection Regulation.

[0107] Environment 1 b includes a second datasource 20. The second datasource 20 includes a database 22 configured to store image-related data 24.

[0108] Database 22 can for instance be implemented using a graph database such as an RDF store that can be manipulated or searched using a query language such as SPARQL, using a relational database management system (RDBMS) that can be manipulated or searched using a query language such as SQL, or using an object-oriented database that can be manipulated or searched using a query language such as OQL.

[0109] Image-related data 24 are related to corresponding images stored by the first datasource 10, but do not include PHI. Generally, image-related data 24 include data generated by applying ML models to images 14. Therefore, second datasource 20 can also be qualified as the Al results server. In some embodiments, image-related data 24 include the results of segmenting one or more anatomical images 14, stored for instance as a mask or as a set of contours. The results of segmenting a medical image can for instance include a set of findings, each finding corresponding to an object detected in the image that is of medical interest, such as an abnormality or an irregularity in tissue, organs, or physiological structures, for instance lesions, edema, cysts and herniations. In some embodiments, the objects are more specifically lesions such benign and / or malignant tumours, including as examples gliomata, glioblastomata, meningiomata, astrocytomata, oligodendrogliomata, medulloblastomata, and other types of intra- and extracranial tumours, and / or metastases thereof, for instance leptomeningeal metastases, shwannomas, vascular anomalies such as cavernomas or microbleeds, strokes, and / or multiple sclerosis (MS) lesions. In some embodiments, image-related data 24 include geometric and / or qualitative information about findings detected in images, including for instance length, width, height, volume measurements, histological type, grade, histopathological features, growth patterns, necrosisinformation, and / or angiogenesis information. In some embodiments, image- related data 24 include other ancillary features, including for instance enhancement patterns such as ring-like enhancement patterns, tridimensional morphometric features such as spiculated and / or sphericity features, and / or radiomic features such as quantities characterizing textures. In some embodiments, image-related data 24 include a transform that can be applied to a first image of an organ of a subject to place it in the frame of reference of a second image of the organ of the subject, as further explained below. In some embodiments, image-related data 24 include an identifier associated with each finding, as further explained below. In some embodiments, image-related data 24 include treatment-related information, including for instance information about treatments received by the patient associated with the images, such as a treatment date and a received radiotherapy dose, for instance a dose expressed in gray (Gy). In some embodiments, image-related data 24 include anatomical location of at least one object, such as brain lobe and / or gyrus. In some embodiments, image- related data 24 include a label associated with an anatomical location of at least one detected lesion, e.g. “superior left temporal lobe”. The labels can be based on standard atlases such as the one described in Desikan, Rahul S., et al.; “An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest”; Neuroimage 31.3 (2006): 968-980, the disclosure of which is hereby incorporated by reference in its entirety. This can advantageously help health professionals such as radiation oncologists assess which brain functionality is likely to be affected by a treatment, e.g., motor and / or language functions, and communicate to a patient which areas of the brain is affected.

[0110] Every image-related data 24 element can be associated with an identifier of a corresponding patient, image 14 or image series, including for instance a DICOM™ Patient ID and / or Unique Identifier (UID). In some embodiments, to provide for greater data security, the database 22 is configured to associate image- related data 24 elements with a value irreversibly derived from the identifier such as a string obtained by applying a configurable hash function, such as SHA-3,SHA-256 and / or BLAKE3 to the identifier. In some embodiments, the identifier can be salted before being hashed, for instance using a configurable salt that is not available outside secured environment 40. In some embodiments, each health service provider could be associated with a given salt. In some embodiments, each user of system 100 could be associated with a given salt. In some embodiments, each patient could be associated with a given salt. In some embodiments, each instance of any type of identifier could be associated with distinct salt, for instance a randomly generated salt, such that the salt is stored in the secured environment and the hashed identifier is securely stored in datasource 20. In some embodiments, the database 22 is configured to associate image-related data 24 elements with a cryptogram, for instance a cryptogram created by symmetric-key cryptographic technique such as Advanced Encryption Standard (AES), Data Encryption Standard (DES) or Triple DES (3DES), i.e. , a cryptogram created using a single private key. In some embodiments, the cryptogram is created using a private key that is not available outside secured environment 40, such that the cryptogram can be decrypted in system 100 but not in datasource 20. In some embodiments, the cryptogram created using a private key in the secured environment 40 can itself be encrypted using a second private key by datasource 20, for instance a private key that is not available outside of datasource 20, such that a leaked cryptogram would require both the private key held in datasource 20 and the private key held in the secured environment 40 for decryption. Using symmetric encryption advantageously allows for recovering a list of patients processed in datasource 20 by system 100 without having to obtain a table including all the possible hashed identifiers to figure out which ones were processed. In some embodiments, certain identifiers are stored securely in datasource 20 and certain other identifiers are stored plainly. As an example only, in one embodiment, image-related data 24 can be index by both Patient ID and UID, with the Patient ID being hashed and / or encrypted and the UID being stored plainly. In some embodiments, some or all identifiers can be both hashed and encrypted, once or more than once, in a configurable order and at configurable locations between the secured environment 40 and the datasource 20. In someembodiments, before being initially hashed and / or encrypted, some or all identifiers can be further obfuscated by replacing the identifier with a different identifier stored in a dictionary data structure accessible only within the secured environment 40.

[0111] In some embodiments, one or both datasources 10 and 20 include an authentication and / or an authorization mechanism. In particular, it can be appreciated that users of system 100 can access the Al results server from different hospital sites. To prevent unauthorized access of a site user to another site’s data, datasource 20 and / or database 22 can implement an authorization scheme where each analysis is assigned a site identifier, such that only users from the corresponding site can access their data. In some embodiments, datasource 20 includes an authentication mechanism through which users identify themselves using authentication credentials such as a password, a personal identification number, an authentication token and / or biometric data, and an authorization mechanism through which datasource 20 and / or database 22 determine which image-related data 24 elements are accessible to the user. In some embodiments, a permission-based access control mechanism assigns access permissions on a per-user basis and / or based on attributes of the user, including for instance a role, an employer, a site and / or a health service provider associated with the user.

[0112] In some embodiments, datasource 20 is included in the secure environment 40. In some embodiments, datasource 20 can advantageously be outside of the secure environment 40 since it does not include any PHI. In some embodiments, datasource 20 is accessible via a WAN 50 such as the Internet. In some embodiments, datasource 20 is hosted in a cloud, e.g., in a network of remote servers providing scalable computing resources over the Internet, advantageously offering flexible, on-demand access, storage, and computing power without the need for local hardware or dedicated servers.

[0113] Environment 1 b includes at least one annotation system 100 implemented in an endpoint device such as a workstation, a laptop, a tablet, a smartphone, a server, a desktop computer, a gaming console or a smart television.

[0114] The annotation system 100 includes at least one output device 110 and at least one input device 115. The output device 110 presents information processed by processor 140 is a format suitable for human consumption. It can include for instance a monitor and / or a printer suitable at least for displaying anatomical images 14 and for conveying image-related data 24. The input device 115 captures user data or commands and transmits it to the system for processing. It can include for instance a keyboard and / or a pointing device suitable at least for allowing a system user to provide a query to obtain images 14 and corresponding data 24, and / or for allowing the system user to edit and / or validate image-related data 24.

[0115] The annotation system 100 includes at least one communication device 120 configured to allow for the exchange of data between the system 100 and external networks or other devices such as datasources 10 and 20. In particular, communication device 120 can allow the system to retrieve anatomical images 14 from datasource 10 and to both retrieve and send image-related data 24 from and to datasource 20. Communication device 120 enables connectivity and data transmission through any suitable communication protocols and technologies. The communication device 120 can for instance include one or more network interfaces, such as an Ethernet and / or a Wi-Fi adapter, radio transceivers, such as a Bluetooth interface, and / or modems.

[0116] The annotation system 100 includes at least one memory 130 configured to store at least data and instructions implementing applications. Memory 130 can include volatile storage suitable for storing images 14 and image-related data 24 received from the communication device 120 to allow for processing via applications and for conveying through the output device 110. Memory 130 can further include non-volatile storage suitable for persisting images 14 and image- related data 24 for faster subsequent access. Memory 130 further includes non-volatile and volatile storage suitable for storing computer code representing instructions associated with software applications and modules for executing these applications and modules.

[0117] The annotation system 100 includes at least one processor 140 configured for executing instructions and performing calculations necessary for the system operation. It interprets and processes data from software applications and hardware components, coordinating tasks and managing system functions. It is responsible for executing applications and modules, and in particular a viewer application 150, an inference module 160, a registration module 170 and / or a longitudinal module 180.

[0118] The annotation system 100 includes a viewer application 150. In some embodiments, the viewer 150 is a web application, e.g., a software program accessed via a web browser, executing code on a server or client side. In some embodiments, the viewer 150 is a Wasm application.

[0119] The viewer 150 provides an interface allowing for the user of the system to perform a query and to convey the result of the query. The query provides an indication of an image 14 or a series of images that the user wants to consult, including for instance an indication of a subject of which the user wants to consult images. The query can for instance include information about the subject of the images 14, such as a subject name, medical file number or a unique patient identifier, for instance corresponding to the DICOM™ Patient ID attribute, or information about the images 14 themselves, such as a DICOM™ UID attribute. In some embodiments, the viewer 150 implements a study browser, configured to assist the user in retrieving the desired images 14 from datasource 10, and in particular the images of the desired study, corresponding to a number of acquisitions typically acquired on a same date, for instance within the framework of a defined protocol such as an Alzheimer MRI protocol, and including one or more series, each corresponding to a specific sequence of images, for instance T1w images with contrast or T2-FLAIR images, each series including one or moreimage “instances”, for instance a single image slice in a tridimensional series, or a single X-ray image. The study browser can be configured to assist the user in building a query for the datasource 10. This query can parameters such as filtering results according to an attribute value, eg. patient ID and / or patient name, ordering results according to an attribute, e.g., by date and / or patient name, and pagination attributes for displaying the results, for instance displaying 20 studies per page and displaying controls that allow the user to cause the display of a subsequent page.

[0120] Based on the query and / or user interactions with the study browser, the viewer 150 determines which study the user wishes to view and causes the retrieval of the available series for this study, allowing the user to select a given series. Once a series is selected, a new query can be performed to retrieve a list of images 14 included in the series, followed by requests causing the retrieval of the images 14 either from non-volatile memory if they are persisted locally, or from the first datasource 10 via the communication device 120, and the storage of the images 14 in volatile and / or non-volatile memory 130. In some embodiments, retrieving the images 14 include retrieving the DICOM™ files which actually contain the pixel values, in addition to metadata. In some embodiments, the viewer 150 can further cause the retrieval of unique identifier(s) associated with the retrieved images, for instance the UID of the image series, in particular if the information is not included in the query. In some embodiments, the query process can include using an index file, for instance to indicate where images, e.g., DICOM™ files, are stored, making it possible to use cloud-based storage solutions such as AWS™ or Google™ Cloud storage.

[0121] In some embodiments, the viewer 150 and / or the browser implementing the viewer 150 are configured to prevent exfiltration of data, for instance PHI, towards an external server. As an example, the browser can be configured to implement Cross-Origin Resource Sharing (CORS). Servers can be configured to only accept requests made from a website served from the same origin. As an example, a PACS 12a located within the secured environment 40 can be configured to reject requests made from a web application served from a distantapplication server. In some embodiments, the PACS 12a can be configured to accept external requests, or the requests can be proxied through a server within the secured environment 40 as to appear to have the same origin.

[0122] Whenever the identifiers used to index the image-related data 24 has been retrieved, the viewer 150 also causes the retrieval or attempted retrieval of image- related data 24 associated with the images 14 either from non-volatile memory if they are persisted locally, or from the second datasource 20 via the communication device 120, and the storage of the data 24 in volatile and / or non-volatile memory 130. The viewer 150 can cause the image-related data 24 to be retrieved by querying the second datasource 20 using unique identifiers associated with the patient, the images 14 and / or series of images retrieved from the first data source 10. As an example, the identifiers can include a unique patient identifier such as a DICOM™ Patient ID, and / or one or more UID. In embodiments in which the database 22 storing the image-related data 24 indexes the data 24 using values irreversibly derived from the identifiers, the unique identifiers can include the derived value, which can be computed by processor 140 based on the unique identifiers, for instance by applying the hash function.

[0123] The retrieved image-related data 24 includes at least a set of findings. It can be appreciated that a set can include zero, one or a plurality of elements. As an example, if no findings were identified in images 14, the set of findings included in the image-related data 24 can be empty set. As another example, if the images 14 were not yet segmented, the set of findings included in the image-related data 24 can be empty set. In some embodiments, datasource 20 is configured to return an empty set if it is queried for image-related data 24 related to images 14 about which database 22 contains no information. In some embodiments, datasource 20 is configured to return an indication such as an error message if it is queried for image-related data 24 related to images 14 about which database 22 contains no information. If an empty set or an error message is received, the viewer 150 can cause the storage of an empty set in memory 130, or the storage of a suitable indication that the set of findings is empty.

[0124] The viewer 150 is configured, once images 14 and image-related data 24 have been retrieved, to cause at least one of the images 14 to be displayed on an output device 110 and to cause data related to the displayed image to be conveyed on the same output device 110. In some embodiments, conveying findings includes overlaying the image with a suitable indication of the findings at the proper location over the image. As an example, if the location of the findings is stored as a mask, the mask can be converted to a colour map, to apply colour tinting to portions of the images where findings were detected. As another example, if the location of the findings is stored as a set of contours, the contours can be drawn over the image in a contrasting colour. In some embodiments, different colours and / or colour gradients used for instance in colour tinting and / or contour drawing can be used to convey geometric and / or qualitative information. In some embodiments, parts of the image-related data 24 not otherwise conveyed can be printed over or next to the displayed image. As an example, each identifier of a finding can be overlaid onto the image in a suitable position inside, overlapping or near the associated finding, or can be printed next to the image, for instance using a leader line. In some embodiments, a user interface element can be provided to list all the findings. In embodiments where database 12 stored image series and / or longitudinal series, the viewer 150 can provide one or more user interface elements including controls allowing the user to consult the different images, for instance allowing navigating through images and / or juxtaposing images.

[0125] In some embodiments, the viewer 150 includes a review mode allowing the user to add, edit, remove, validate and / or reject findings. In review mode, the user can activate finding list items, for instance by clicking, to navigate the viewer to contour location. If appropriate, the user can approve or reject the finding. The user can rename findings, modify the contours of findings, add missing findings, delete created findings, and / or add, edit or remove additional information about findings such as qualitative information. The viewer 150 is configured to use these edits to update image-related data 24, and to cause the updated image-related data to be persisted at the second datasource 20. The updated image-related data 24 can be sent to the second data source 20 along with one or more uniqueidentifiers (including derived value) associated with the images and / or series of images to which the image-related data 24 pertains.

[0126] In some embodiments, when the viewer application 150 is terminated, for instance because the browser or browser tab was closed or the user activated a suitable user interface element, images 14 and / or image-related data 24 can be discarded from memory 130. In some embodiments, when the viewer application 150 is terminated, all the information that was retrieved or generated by the application is discarded from memory 130. In some embodiments, when the viewer application 150 is terminated, authentication tokens are retained in memory indefinitely, for a configurable amount of time and / or until a configurable event such as the user logging out of the annotation system 100 occurs. When reloading the web application, the remote Al results server can first be queried to verify if there is an existing analysis for the subject. If there is one, the data 24 can be restored in memory 130, allowing continuing where the user left off.

[0127] In some embodiments, the annotation system 100 includes an inference module 160 configured to perform image segmentation and / or to detect findings in images. In some embodiments, the inference module 160 is provided as part of the web application also implementing the viewer 150. When the viewer application 150 causes an image or an image series 14 to be loaded, if no associated findings are retrieved from datasource 20 or if an empty set of findings is retrieved, the inference module 160 can be triggered to process the image or image series 14. In some embodiments, image-related data can include a parameter to convey that an image series 14 has already been processed, and that no findings were found, so that the inference module 160 needs not be triggered every time the series is loaded. In some embodiments and in some contexts, such as a longitudinal context, one or more image series 14 associated with a patient may have been processed already, and a new image series 14 may have been acquired and require processing by the inference module 160. This process can result in sets of contours delineating findings, for instance, delineating brain metastases in a brain MRI. Each set of contours represents an individual finding from the model. In someembodiments, the inference module 160 includes one or more trained machine learning models, such as neural network models. As an example, the inference module 160 can include a first neural network model configured to detect objects in one or more images and a second neural network model configured to estimate the contours of the detected objects. In some embodiments, the inference module 160 is configured to use one or more techniques described in International Patent Application Publications Nos. WO 2022 / 073100 and / or WO 2023 / 205896, the complete disclosures of which are hereby incorporated in their entirety. Once created, findings and contours are persisted remotely in the Al results server 20.

[0128] In some embodiments, the annotation system 100 includes a registration module 170 configured to compute a transform between the different intra-patient series, i.e., within a longitudinal series. In some embodiments, the registration module 170 is provided as part of the web application also implementing the viewer 150. The transform can for instance be a rigid and / or an affine transform. The transform can define a transformation matrix corresponding to a mapping from the pixels of one given image and / or the voxels of one given image series in the longitudinal series, which can be named the reference image(s), to corresponding pixels and / or voxels of one or more other image or image series. The transform can advantageously make it easier to track changes or progress in the subject’s condition over time, by allowing the synchronization of the slicing between the different images series of the longitudinal series, making it possible to match findings. Furthermore, the transform can be used to synchronize viewports when multiple series are displayed in viewer 150. In some embodiments, the transform is persisted in the Al results server 20. Advantageously, tracking certain features of image-related data 24, such as metastases size and / or enhancement, can help assess a risk of recurrence.

[0129] It can be appreciated that running the inference module 160 and the registration module 170 can require significant computer resources. When used, it could disrupt the functioning of the viewer 150 and the overall user experience expected from a graphical user interface. In some embodiments, results can beprecomputed, for example, by a technician on a separate workstation. In some embodiments, the functions of the inference module 160 and / or registration module 170 can be externalized, for instance, to a “headless” daemon process running continually on a workstation within the secured environment. In some embodiments, the daemon process can be configured to process images and persist results remotely as they are acquired or received at datasource 10. In embodiments where datasource 10 is cloud-based, secure means of processing data remotely can be made available, transposing the functions of the inference and / or registration modules to a cloud, serverless solution.

[0130] In some embodiments, the annotation system 100 includes a longitudinal module 180 configured to match findings from different image series associated with a subject and taken at different times corresponding to the same object, such that they can be tracked longitudinally. In some embodiments, the longitudinal module 180 is provided as part of the web application also implementing the viewer 150. Using an assignment algorithm and the precomputed registration transforms, a unique identifier can be assigned to each object, defining a surjective mapping from findings to objects. In some embodiments, the viewer 150 is configured to allow the user to manually reassign findings identifiers if needed. The assignment is persisted on the results server 20.

[0131] With reference to figure 1 C, a further exemplary environment 1 c for annotating anatomical images of a subject using a browser 100 in the context of a health service provider such as a hospital is shown. Broadly described, environment 1 c includes the browser 100 running a viewer web application which accesses a datasource 10 and a remote Al results server 20.

[0132] The browser 100 is a software application that runs on an endpoint device located in on the hospital site 40. It interprets and displays web pages, which can be written in languages such as HTML, CSS, and JavaScript™. It also serves as an execution environment for web applications, allowing them to run directly within the browser interface. This includes Wasm applications, which are executed atnear-native speed, providing high-performance computing capabilities within the browser.

[0133] The browser 100 of environment 1 c is more particularly configured to execute a web application that allows annotating anatomical images of a subject. In the description of figure 1 C, the functionalities of the browser, including its native capabilities and those provided by the annotation application executed within the browser, are presented collectively and without distinction. The term “browser” encompasses both the intrinsic features of the browser itself and the extended functionalities enabled through the web application, as it operates seamlessly within the browser environment.

[0134] The browser 100 has access to a datasource 10, which can communicate with a PACS 12a hosted within the hospital site 40 and / or a cloud-based PACS 12b securely hosted outside the hospital site 40 using a suitable interface such as the one provided by the DICOMweb™ standard. The PACS 12a and / or 12b allow the browser 100 to retrieve images and / or RT objects using an import-export module 125 implementing the DICOM™ RT standard. The images can then be processed and / or displayed by the browser 100. In some embodiments, the browser 100 can cause retrieved images to be stored in persistent storage of the endpoint device, so that retrieving an image series can be performed through a file API simply by retrieving local files 135 of the endpoint device.

[0135] The browser 100 also has access to an Al results server 20, used to persist results of image processing and to restore pre-existing results. Advantageously, the results do not include any PHI. Therefore, the Al results server 20 can be hosted outside of the hospital site. The Al results server 20 includes a database to store results. For instance, this can include an object storage 22a and / or a relational database management system 22b (RDBMS). While in can be appreciated that all types of image-related data can be stored by a RDBMS 22b, object storage such as provided by AWS™ S3 or Google™ Cloud, can be moresuitable for storing certain types of image-related data, including for instance contours, which can be represented by polyline coordinates, and / or binary data.

[0136] The results stored by server 20 can include findings 24a detected in images, contours 24b of the detected findings, transforms 24c making it possible to map different images or image series belonging to a longitudinal series, i.e., corresponding to the same anatomical feature of the same subject acquired over a period of time to track the evolution of lesions, lesions 24d, each lesion corresponding to one finding in at least a subset of the images of the longitudinal series, and analysis data 24e, including for instance metadata such as creation and / or modification date, and / or identification of authors and / or editors. In some embodiments, analysis data 24e can be used to characterize different sets of data related to the same patient. This can advantageously allow for tests, training, repeatability studies, use by different professionals with distinct readings, approaches or interpretation, and / or use in different contexts. In a longitudinal series of radiological images, each finding 24a corresponds to a detected lesions and is unique to an image. However, findings 24a are used to track consistent lesions 24d across multiple images, enabling longitudinal assessment and monitoring of the same lesions over time. In other words, there exists a surjection such that each finding 24a can be mapped to one lesion 24d, but one lesion 24d can be mapped to multiple findings 24a. If the lesion tracking is successful, it should follow that the surjection is non-injective, i.e., that more than one finding 24a map to one lesion 24d.

[0137] The Al result server 20 can include an authentication mechanism 26 configured to identify a user or the hospital of the user connecting to the server 20 working in conjunction with an authorization mechanism 28 configured to grant access permissions to data 24a-d stored in databases 22a-b, for instance on a per- hospital basis. The mechanisms can advantageously be configured so that a user can only retrieve data generated by users of the same hospital. In some embodiments, the authentication mechanism 26 is configured to implement authentication by cryptographically signed token using a suitable standard such asJSON Web Token (JWT). In some embodiments, the authorization mechanism 28 is configured to implement row-level security (RLS), making it possible to define which user has access to which rows of database 22b. Therefore, user permissions can be configured at the database level. As an example, when server 20 receives a request, a corresponding query on the database is performed as the specific, authenticated user, such that only results which are allowed by the RLS policy are retrieved by the query. In some embodiments, existing cloud-based storage solutions and / or Backend-as-a-Service (BaaS) solutions are used to implement Al result server 20.

[0138] The browser 100 provides a viewer 150, configured to display the images and the corresponding data 24a-d. The viewer 150, furthermore, provides a review workflow 155, allowing the user to review, edit and / or validate findings 24a and / or contours 24b. For instance, the review workflow 155 can make it possible for the user to add or remove findings 24a and to edit contours 24b. Modified findings 24a and / or contours 24b can be persisted to the Al result server 20. In some embodiments, validated findings 24a can further be persisted as RT objects to the PACS 12a and / or 12b by the import export module 125. In some embodiments, the import export module 125 is configured to generate a DICOM-RT file containing the definition of a StructureSet. The StructureSet can define radiotherapy contours for a set of target structures to be treated, e.g., by stereotactic surgery. It can contain contours, e.g., set of tridimensional coordinates, and / or other metadata such as the generation algorithm, a description, and / or a type. They can represent an important part of the treatment planning step in radiotherapy. As an example, they would be useful to initialize a treatment plan and compare treated lesions. In some embodiments, if the patient has already has a treatment plan, the import export 125 module can be configured to parse the DICOM-RT file and import preexisting contours. Overlapping structures stored as image-related data 24a-b and / or obtained from the inference module 160 discussed below can be discarded. As an example, the retrieved data can be used to infer a treatment date per lesion, and / or to serve as a reference to assess treatment response.

[0139] The browser 100 can further provide an inference module 160 configured to compute findings 24a, a registration module 170 configured to compute the transforms 24c, and / or a longitudinal workflow 180 configured to associate findings with lesions 24d by assigning a lesion identifier to each finding. In some embodiments, the browser 100 further provides a report module 190, configured to generate reports 194 summarizing the findings.

[0140] With reference to figure 2, an exemplary method for annotating anatomical images of a subject using an endpoint device is provided.

[0141] In an initial step 210, a query can be received from a user, defining the subject directly by using personally identifiable information and / or indirectly by indicating which anatomical images should be retrieved. The query can be received, for example at an endpoint device.

[0142] A subsequent step 220 includes retrieving the images associated with the patient, for instance, from a first datasource hosted in a secured environment that also includes the endpoint device. The datasource can for instance correspond to a PACS hosted on the same site as the endpoint device, and / or to a cloud-based PACS accessible through a secured communication channel, for instance via a VPN. Retrieving the images can further include receiving one or more unique identifiers (including derived value) associated with the images and / or series of images associated with the patient.

[0143] A step 230 includes retrieving image-related data associated with the images, including a set of findings, for instance from a second datasource. The image-related data is free of PHI, and therefore the second datasource storing the image-related data can be inside or outside the secured environment. The image- related data can be retrieved by querying the second data source for image-related data associated with the images or series of images retrieved in step 220, for example using one or more unique identifier (including derived value) associated therewith.

[0144] It can be appreciated that steps 220 and 230 can include substeps, and that some substeps of steps 220 and 230 can be performed in any order and / or in parallel. As an example only, a first substep of step 220 can include querying the first datasource using a patient name, which results in obtaining a Patient ID. The Patient ID can subsequently be used by substeps of step 220 to retrieve imaging studies, image series and / or image instances. Before, after or during these substeps, the Patient ID can be used by substeps of step 230 to retrieve existing analyses for this patient, to fetch findings, lesions and / or registration parameters associated with the relevant analysis, and / or to retrieve contours associated with the retrieved findings.

[0145] Steps 240, 250 and 260 can be performed at the endpoint device and / or on another system within the secured environment, if information is missing from the image-related data. If the set of findings is an empty set, if no image-related data associated with the images can be retrieved, or if the set of findings is incomplete, the set of findings can be computed from the images in step 240. If the images correspond to a longitudinal series of images and a transform is not found among the image-related data, the transform can be computed from the images in step 250. If the images correspond to a longitudinal series of images and lesion assignments are not found among the image-related data, a lesion identifier can be computed for each finding in step 260.

[0146] A subsequent step 270 can include displaying the images, or one of the images, and conveying the associated image-related data, for instance by overlaying the findings over the image. The images and / or findings can be displayed by a display of endpoint device and / or other device within the secured environment.

[0147] A subsequent step 280 can include allowing the user to edit and / or validate the findings and / or other image-related data. For instance, the user can be allowed to edit, to add, to remove findings, to rename and / or to validate findings. Theediting can be performed using input and output devices operatively connected to endpoint device and / or other device within the secured environment.

[0148] At any time, step 290 can be performed at the endpoint device and / or on another system within the secured environment, to ensure that image-related data that was updated or added in any one of steps 240, 250, 260 and 280 are persisted to the second datasource. Step 290 can include, for example, transmitting updated and / or added image-related data to the second datasource for storage. The image- related data can be transmitted along with a corresponding unique identifier (including derived value) so that the image-related data can be stored by the second datasource in association with the appropriate image and / or series of images.

[0149] The disclosed neural network implementations may be realized through various configurations of computer hardware, software, or a combination of both, depending on the requirements and constraints of the particular application. For instance, the neural networks may leverage specialized hardware, such as GPUs, tensor processing units (TPUs), FPGAs, and ASICs, which are designed to efficiently handle the high computational demands of training and deploying neural networks. These hardware components are particularly advantageous for accelerating matrix operations, which are central to neural network computations, and can significantly reduce the time needed for training large models and performing inference tasks.

[0150] Alternatively, neural networks can be implemented using traditional computer hardware, including CPUs, which are versatile and widely available. While CPUs are not optimized specifically for neural network computations, they can still handle smaller models and less computationally intensive tasks effectively. In cases where flexibility is essential, such as in general-purpose computing environments, implementing neural networks on CPUs allows for integration with other software systems without the need for specialized hardware.

[0151] On the software side, neural networks can be created using various programming languages and frameworks. High-level languages such as Python, Java, and C++ are commonly used for neural network development, particularly in conjunction with deep learning libraries and frameworks like TensorFlow, PyTorch, Keras, and Theano. These frameworks provide prebuilt functions, modules, and tools that simplify the process of designing, training, and deploying neural networks. They allow developers to define network architectures, optimize training parameters, and manage data flows with relative ease. For instance, TensorFlow and PyTorch offer extensive support for GPU and TPU integration, enabling seamless transitions between hardware and software environments.

[0152] Neural networks implemented in software may also vary based on the type of language and runtime environment used. For example, imperative languages such as Python and Java allow developers to create neural networks using clear, step-by-step procedural code, making the design process intuitive and manageable. Alternatively, functional languages like Lisp and Haskell may also be employed to build neural networks, particularly when focusing on functional aspects of data flow and transformation. Moreover, neural networks can be implemented in either compiled or interpreted languages, where compiled languages, such as C++ or Java, can offer improved execution speed, while interpreted languages like Python provide flexibility and ease of development.

[0153] In certain configurations, neural networks may be deployed in distributed computing environments, allowing for parallel processing across multiple processing units or even across different geographic locations. Distributed implementations can be achieved through cloud computing platforms or high- performance computing (HPC) systems, where workloads are split among numerous machines to improve efficiency and scalability. This is particularly useful for training large-scale models that require significant processing power and storage capacity. Distributed computing frameworks such as Apache Spark and Horovod can facilitate the parallelization of neural network computations, enablinglarge datasets and complex models to be processed in a fraction of the time that would be required on a single machine.

[0154] Neural network implementations may also employ hybrid configurations that combine both hardware and software elements. For example, the core neural network computations might be performed on dedicated hardware accelerators like GPUs or TPUs, while the overall system, including data preprocessing and postprocessing steps, can be managed by general-purpose software running on CPUs. This hybrid approach optimizes performance by leveraging the strengths of both hardware and software environments, ensuring efficient resource utilization across different components of the system.

[0155] Furthermore, it is understood that the neural networks described herein are not limited to any specific type of architecture. Various neural network architectures, including but not limited to convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM) networks, transformers, and generative adversarial networks (GANs), may be implemented depending on the task requirements. These architectures can be tailored to perform tasks such as image recognition, natural language processing, and predictive modelling, each benefiting from different configurations of hardware and software resources to optimize performance.

[0156] For secure and reliable deployment, neural networks may also incorporate mechanisms for data integrity, confidentiality, and robustness against adversarial attacks. Security protocols, such as data encryption and access control, may be applied to safeguard sensitive data processed by the neural network. Techniques like differential privacy and secure multiparty computation can be employed to protect data confidentiality during training and inference. Additionally, the implementation may include error-handling mechanisms and redundancy measures to ensure robust operation, even in environments where hardware failures or software bugs may occur.

[0157] Overall, the neural networks in this disclosure may be implemented as flexible, scalable systems that leverage combinations of hardware and software elements tailored to the needs of specific applications. This approach provides versatility, allowing the neural networks to be deployed in a wide range of environments, from dedicated hardware systems to virtualized cloud platforms, thereby supporting a diverse set of use cases and performance requirements.

[0158] Any reference to prior art publications within this disclosure should not be taken as an acknowledgment or admission that these publications form part of the common general knowledge in the relevant field, whether in any particular jurisdiction or globally.

[0159] The examples provided in the above description serve to illustrate specific embodiments and convey certain features and principles. However, those skilled in the art will recognize that individual features, elements, and functionalities within the disclosed embodiments may be adapted, modified, or combined in numerous ways without departing from the core spirit or intended scope of the described subject matter. Therefore, the foregoing description is meant to be illustrative rather than limiting, with the scope being defined by the appended claims, which are intended to encompass all variations and modifications within the broadest interpretation permitted by applicable law.

Claims

CLAIMS1. A system for annotating one or more anatomical images of a subject, the system comprising: at least one input device and at least one output device; at least one communication device configured for: retrieving the anatomical images from a first datasource comprised in a secured environment also comprising the system, and sending and / or retrieving image-related data to and / or from a second datasource, the second datasource being distinct from the first datasource, wherein the image-related data comprises at least a set of findings, and wherein the image-related data is free of protected health information; at least one memory configured to store the anatomical images, the image- related data and instructions implementing a viewer application; and at least one processor configured to run the viewer application, causing the system to: receive from a user via the input device a query defining the subject, display on the output device the anatomical images overlaid with indications of the findings, and allow the user to edit and / or validate the findings, thereby updating the image-related data.

2. The system of claim 1 , wherein the anatomical images are radiographic images of an organ of the subject.

3. The system of claim 2, wherein the anatomical images are radiographic images of the brain of the subject.

4. The system of any one of claims 1 to 3, wherein the communication device is a network interface.

5. The system of any one of claims 1 to 4, wherein the first datasource is a picture archiving and communication system.

6. The system of claim 5, wherein the communication device is configured to connect to the first datasource using a DICOMweb™ service.

7. The system of any one of claims 1 to 6, wherein the second datasource is hosted in a cloud.

8. The system of any one of claims 1 to 7, wherein the second datasource is outside of the secure environment.

9. The system of any one of claims 1 to 8, wherein the second datasource is protected by an authentication mechanism and / or by an authorization mechanism.

10. The system of any one of claims 1 to 9, wherein each finding corresponds to a lesion detected in at least one of the anatomical images.11 . The system of claim 10, wherein each finding correspond to a tumour.

12. The system of claim 11 , wherein each finding corresponds to a metastasis.

13. The system of any one of claims 1 to 12, wherein each finding is defined by a contour.

14. The system of any one of claims 1 to 13, further comprising an inference module configured, in response to the set of findings being empty, to detect findings and to update the set of findings with the detected findings.

15. The system of claim 14, further comprising a neural network trained for detecting and contouring structures of interest in anatomical images.

16. The system of any one of claims 1 to 15, wherein the one or more anatomical images comprises a plurality of subsets of images, wherein each subset of images comprises anatomical images of a generally equivalent area of the subject captured at different times.

17. The system of claim 16, comprising a transform defining a mapping of pixels or voxels of a reference subset of images to corresponding pixels or voxels to other subsets of images.

18. The system of claim 17, wherein the transform is an affine and / or rigid transform.

19. The system of claim 17 or 18, further comprising a registration module configured to compute the transform.

20. The system of any one of claims 17 to 19, wherein the image-related data comprises the transform.

21. The system of any one of claims 16 to 20, wherein the findings are assigned finding identifiers to link corresponding findings detected across two or more of the subsets of images, allowing for tracking the findings between the subsets of images.

22. The system of claim 21 , further comprising a longitudinal module configured to assign the finding identifiers.

23. The system of claim 21 or 22, wherein the image-related data comprises the finding identifiers.

24. The system of any one of claims 1 to 23, wherein the viewer application causes the system to allow the user to rename, modify and / or delete a finding from the set of findings.

25. The system of any one of claims 1 to 24, wherein the viewer application causes the system to allow the user to create a new finding and insert the new finding in the set of findings.

26. The system of any one of claims 1 to 25, wherein the viewer application is configured, in response to being shut down, to remove the anatomical images and / or the image-related data from the memory.

27. The system of any one of claims 1 to 26, wherein the viewer application is configured to run in a web browser.

28. The system of claim 27, wherein the viewer application is a WebAssembly application.

29. A computer-implemented method for annotating one or more anatomical images of a subject at an endpoint device, the method comprising: receiving a query from a user defining the subject; retrieving the anatomical images from a first datasource comprised in a secured environment also comprising the endpoint device; retrieving image-related data comprising a set of findings from a second datasource, the second datasource being distinct from the first datasource, wherein the image-related data is free of protected health information; displaying the anatomical images overlaid with the findings; providing an interface for the user to edit and / or validate the findings, thereby updating the image-related data; and sending the updated image-related data to the second datasource.

30. The method of claim 29, wherein the anatomical images are radiographic images of an organ of the subject.

31. The method of claim 30, wherein the anatomical images are radiographic images of the brain of the subject.

32. The method of any one of claims 29 to 31 , wherein the first datasource is a picture archiving and communication system.

33. The method of claim 32, wherein retrieving the anatomical images from a first datasource comprises connecting to the first datasource using a DICOMweb™ service.

34. The method of any one of claims 29 to 33, wherein the second datasource is hosted in a cloud.

35. The method of any one of claims 29 to 34, wherein the second datasource is outside of the secure environment.

36. The method of any one of claims 29 to 35, wherein the second datasource is protected by an authentication mechanism and / or by an authorization mechanism.

37. The method of any one of claims 29 to 36, wherein each finding corresponds to a lesion detected in at least one of the anatomical images.

38. The method of claim 37, wherein each finding correspond to a tumour.

39. The method of claim 38, wherein each finding corresponds to a metastasis.

40. The method of any one of claims 29 to 39, wherein each finding is defined by a contour.

41. The method of any one of claims 29 to 40, further comprising, in response to the set of findings being empty, detecting findings and updating the set of findings with the detected findings.

42. The method of claim 41 , wherein detecting findings comprises using a neural network trained for detecting and contouring structures of interest in anatomical images.

43. The method of any one of claims 29 to 42, wherein the one or more anatomical images comprises a plurality of subsets of images, wherein each subset of images comprises anatomical images of a generally equivalent area of the subject captured at different times.

44. The method of claim 43, further comprising computing a transform defining a mapping of pixels or voxels of a reference subset of images to corresponding pixels or voxels to other subsets of images.

45. The method of claim 44, wherein the transform is an affine and / or rigid transform.

46. The method of claim 44 or 45, wherein the image-related data comprises the transform.

47. The method of any one of claims 44 to 46, further comprising assigning finding identifiers to the findings to link corresponding findings detected across two or more of the subsets of images, allowing for tracking the findings between the subsets of images.

48. The method of claim 47, wherein the image-related data comprises the finding identifiers.

49. The method of any one of claims 29 to 48, wherein the interface further allows the user to rename, modify and / or delete a finding from the set of findings.

50. The method of any one of claims 29 to 49, wherein the interface further allows the user to create a new finding and insert the new finding in the set of findings.

51. The method of any one of claims 29 to 50, further comprising, in response to the interface being shut down, removing the anatomical images and / or the image-related data from memory of the endpoint device.

52. The method of any one of claims 29 to 51 , wherein the interface is a web interface.

53. The method of claim 52, wherein the web interface is implemented as a WebAssembly application.

54. A computer-readable memory having recorded thereon instructions, the instructions, when executed by at least one processor of an endpoint device, causing the at least one processor to: receive for a query from a user defining the subject; retrieve the anatomical images from a first datasource comprised in a secured environment also comprising the endpoint device; retrieve image-related data comprising a set of findings from a second datasource, the second datasource being distinct from the first datasource, wherein the image-related data is free of protected health information; display the anatomical images overlaid with the findings; provide an interface for the user to edit and / or validate the findings, thereby updating the image-related data; and send the updated image-related data to the second datasource.

55. The computer-readable memory of claim 54, wherein the anatomical images are radiographic images of an organ of the subject.

56. The computer-readable memory of claim 55, wherein the anatomical images are radiographic images of the brain of the subject.

57. The computer-readable memory of any one of claims 54 to 56, wherein the first datasource is a picture archiving and communication system.

58. The computer-readable memory of claim 57, wherein retrieving the anatomical images from a first datasource comprises connecting to the first datasource using a DICOMweb™ service.

59. The computer-readable memory of any one of claims 54 to 58, wherein the second datasource is hosted in a cloud.

60. The computer-readable memory of any one of claims 54 to 59, wherein the second datasource is outside of the secure environment.

61. The computer-readable memory of any one of claims 54 to 60, wherein the second datasource is protected by an authentication mechanism and / or by an authorization mechanism.

62. The computer-readable memory of any one of claims 54 to 61 , wherein each finding corresponds to a lesion detected in at least one of the anatomical images.

63. The computer-readable memory of claim 62, wherein each finding correspond to a tumour.

64. The computer-readable memory of claim 63, wherein each finding corresponds to a metastasis.

65. The computer-readable memory of any one of claims 54 to 64, wherein each finding is defined by a contour.

66. The computer-readable memory of any one of claims 54 to 65, the instructions further causing the at least one processor to, in response to the set of findings being empty, detecting findings and updating the set of findings with the detected findings.

67. The computer-readable memory of claim 66, wherein detecting findings comprises using a neural network trained for detecting and contouring structures of interest in anatomical images.

68. The computer-readable memory of any one of claims 54 to 67, wherein the one or more anatomical images comprises a plurality of subsets of images, wherein each subset of images comprises anatomical images of a generally equivalent area of the subject captured at different times.

69. The computer-readable memory of claim 68, the instructions further causing the at least one processor to compute a transform defining a mapping of pixels or voxels of a reference subset of images to corresponding pixels or voxels to other subsets of images.

70. The computer-readable memory of claim 69, wherein the transform is an affine and / or rigid transform.

71. The computer-readable memory of claim 69 or 70, wherein the image-related data comprises the transform.

72. The computer-readable memory of any one of claims 54 to 71 , the instructions further causing the at least one processor to assign finding identifiers to the findings to link corresponding findings detected across two or more of the subsets of images, allowing for tracking the findings between the subsets of images.

73. The computer-readable memory of claim 72, wherein the image-related data comprises the finding identifiers.

74. The computer-readable memory of any one of claims 54 to 73, wherein the interface further allows the user to rename, modify and / or delete a finding from the set of findings.

75. The computer-readable memory of any one of claims 54 to 74, wherein the interface further allows the user to create a new finding and insert the new finding in the set of findings.

76. The computer-readable memory of any one of claims 54 to 75, the instructions further causing the at least one processor to, in response to the interface being shut down, remove the anatomical images and / or the image-related data from memory of the endpoint device.

77. The computer-readable memory of any one of claims 54 to 76, wherein the interface is a web interface.

78. The computer-readable memory of claim 77, wherein the web interface is implemented as a WebAssembly application.

Images