System and method for registering digital slide images - Patents.com

Abstract

The disclosed computer-implemented method includes the steps of: (1) receiving an image of a section of a cuttable tissue sample block, the image including image data of (A) the tissue sample section and (B) a section of cuttable fiducial markers embedded in the section of the cuttable tissue sample block, (2) identifying attributes of the section of the cuttable fiducial markers from the received image of the section of the cuttable tissue sample block, and (3) performing a tissue sample management operation based on the identified attributes of the section of the cuttable fiducial markers. Various other methods, systems, devices, and computer-readable media are also disclosed.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 256,575, filed October 16, 2021, the disclosure of which is relied upon and incorporated by reference in its entirety into this disclosure. [Technical field]

[0002] SUMMARY OF THE DISCLOSURE This application relates to a system and method for registering digital slide images. [Background technology]

[0003] When preparing a patient's tissue sample for microscopic analysis, a pathologist may cut the tissue sample into multiple slices, also called sections or levels. Each section or level of the tissue sample may be placed on a carrier medium, such as a transparent (e.g., glass) slide.

[0004] A pathologist may find it useful to examine multiple sections or slices from a tissue sample to ensure that nothing is missed when analyzing the tissue sample microscopically. Some procedures may require multiple sections of the same specimen to be examined under the microscope from various cut slices or levels of the specimen. Additionally, when a special stain is ordered to be added to a tissue sample, all of the stained tissue may be examined. Such examination may include multiple sections or levels of the tissue sample.

[0005] Furthermore, when a pathologist identifies a region of interest at one level during analysis of a tissue sample, it may be desirable to review corresponding regions in other levels and / or stained sections. Unfortunately, tissue samples may exhibit significantly different morphologies from level to level, which may make it difficult to visually identify corresponding locations or positions across different levels. Conventional processes for identifying a region of interest at a first level and switching the image to a second level to identify, in particular, a location on the second level that corresponds to the region of interest on the first level may be tedious and / or prone to failure.

[0006] Furthermore, to aid and / or facilitate some analytical processes, when using transparent (e.g., glass) slides, the pathologist may choose to stack at least two slides and physically manipulate the slides until corresponding sections or levels of the tissue samples are registered (e.g., positionally and / or optically) with one another. The pathologist may then mark the slides to identify regions of interest. Traditional methods of manipulating transparent slides carrying tissue samples may be cumbersome, inconvenient, and / or prone to error. Summary of the Invention [Problem to be solved by the invention]

[0007] Accordingly, the present application recognizes and seeks to address the need for improved systems, methods, and apparatus for registering tissue sample slide images to aid in the analysis of such tissue samples. [Means for solving the problem]

[0008] The present disclosure generally relates to methods and systems for registering (e.g., orienting, scaling, stretching, compressing, resizing, keystone correction, etc.) digital slide images of cut tissue samples. In some embodiments, the disclosed systems and methods may utilize at least one cuttable fiducial marker present on each section of the cut tissue sample (and thus on each digital slide image of each section of the tissue sample). The cuttable fiducial marker may be embedded with the specimen (e.g., in a paraffin wax block containing the specimen) to ensure that the marker appears consistently in the same position across all levels, creating and / or enabling a common reference point for registration.

[0009] Embodiments of the systems and methods described herein may identify one or more sections of the cuttable fiducial markers in each image and determine how to register the images with one another, which may include any suitable reorienting, scaling, stretching, compressing, etc. of the images. In some embodiments, portions of cut tissue samples taken from adjacent portions of tissue (e.g., at each level) may be positioned (e.g., overlapping, adjacent to one another) for simultaneous and / or overlaid viewing in the registered images. Effect of the Invention

[0010] Thus, the methods and systems of the present disclosure may facilitate and / or improve the viewing of digital slide images of sectioned tissue samples, such as for identifying and diagnosing abnormal tissue structures.

[0011] Throughout the drawings, like reference numbers and descriptions refer to like, but not necessarily identical, elements. While the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are described in detail herein. Nevertheless, the exemplary embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the disclosure is intended to cover all modifications, equivalents, and alternatives falling within the scope of the appended claims.

[0012] The accompanying drawings illustrate several exemplary embodiments and are a part of this specification and, together with the following description, illustrate and explain various principles of the present disclosure. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of an exemplary system for registering digital slide images. [Diagram 2] FIG. 1 is a block diagram of an example use of the system for registering digital slide images. [Diagram 3]1 is a flow diagram of an exemplary method for registering digital slide images. [Figure 4A] FIG. 1 is a plan view of a cuttable fiducial marker according to some embodiments described herein. [Figure 4B] FIG. 1 is a perspective view of a cuttable fiducial marker according to some embodiments described herein. [Figure 5A] FIG. 2 is a top view of a first tissue sample slide according to certain embodiments described herein. [Figure 5B] FIG. 2 is a top view of a second tissue sample slide according to certain embodiments described herein. [Figure 5C] FIG. 2 is a top view of a third tissue sample slide according to certain embodiments described herein. [Figure 6] 1 includes illustrations of tissue samples and / or tissue sample slides aligned in a common orientation according to several embodiments described herein. [Figure 7] 1 includes an illustration of a graphical user interface showing a section of a tissue sample oriented within the graphical user interface according to certain embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] A system for registering digital slide images will now be described in detail with reference to Figures 1, 2 and 4-7. A corresponding computer-implemented method will also be described in detail with reference to Figure 3.

[0015] FIG. 1 is a block diagram of an exemplary system 100 for registering digital slide images. As shown in this figure, the exemplary system 100 may include one or more modules for performing one or more tasks. As will be described in greater detail below, the module 102 may include a receiving module 104 that may be configured to receive an image of a section of a cuttable tissue sample block. The image may include image data of (1) the tissue sample section and (2) the cuttable fiducial marker section embedded in the cuttable tissue sample block section. As further shown in FIG. 1, the module 102 may also include an identifying module 106 that is configured to identify attributes of the cuttable fiducial marker section and / or attributes of the tissue sample section from the received image of the cuttable tissue sample block section. Additionally, the module 102 may also include an executing module 108 that is configured to execute a tissue sample management operation based on the identified attributes of the cuttable fiducial marker section.

[0016] 1, the exemplary system 100 may also include one or more memory devices, such as memory 120. Memory 120 generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, memory 120 may store, load, and / or maintain one or more modules 102. Examples of memory 120 include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD), optical disk drive, cache, one or more variations or combinations thereof, or any other suitable storage memory.

[0017] As further shown in FIG. 1, the exemplary system 100 may also include one or more physical processors, such as a physical processor 130. The physical processor 130 generally refers to any type or form of hardware-based processor capable of interpreting and / or executing computer-readable instructions. In one example, the physical processor 130 may access and / or modify one or more modules 102 stored in the memory 120. Additionally or alternatively, the physical processor 130 may execute one or more modules 102 in accordance with the systems and methods described herein to facilitate registration of digital slide images. Examples of the physical processor 130 include, but are not limited to, a microprocessor, a microcontroller, a central processing unit (CPU), a field programmable gate array (FPGA) using soft-core processors, an application specific integrated circuit (ASIC), one or more portions thereof, one or more variations or combinations thereof, or any other suitable physical processor.

[0018] As further shown in FIG. 1 , the exemplary system 100 may further include one or more data stores, such as data store 140, that may receive, store, and / or maintain data. Data store 140 may represent a single data store or computing device, or portions of multiple data stores or computing devices. In some embodiments, data store 140 is a logical container of data and may be embodied in a variety of forms (e.g., databases, files, file systems, data structures, etc.). Examples of data store 140 may include, but are not limited to, files, file systems, data stores, databases, and / or database management systems, such as operational data storage (ODS), relational databases, No SQL databases, New SQL databases, and / or any other suitable organized collection of data.

[0019] In at least one example, data store 140 may include (e.g., store, host, access, maintain, etc.) image data 142. As described in greater detail below, in some examples, image data 142 may include and / or represent any image data associated with, including, but not limited to, one or more tissue samples, one or more tissue sample blocks, one or more sections of one or more cuttable tissue sample blocks, one or more tissue sample slides, etc.

[0020] 1, the exemplary system 100 may optionally include a machine learning model 150. A machine learning model may be a data analysis tool that may find patterns therein and / or make predictions about previously unseen data sets. In some examples, the machine learning model 150 may include any suitable model and / or algorithm, which may include, but is not limited to, a linear regression algorithm, a logistic regression algorithm, a support vector machine (SVM), a nearest neighbor classifier, a principal component analysis algorithm, a decision tree, a naive Bayes classifier, a k-means clustering algorithm, etc.

[0021] One machine learning model that may be particularly useful in analyzing data sets such as unstructured image data, and therefore may be included as part of the machine learning models 150, may be an artificial neural network. An artificial neural network is a computational system inspired by biological neural networks. An artificial neural network may "learn" tasks that are often not pre-programmed with task-specific rules by processing examples or training data. Effectively trained artificial neural networks may be powerful tools to aid in modern computational tasks such as pattern recognition, process control, data analysis, and social filtering.

[0022] An example of training an artificial neural network from a given example may include identifying the difference (e.g., error) between the artificial neural network's output (e.g., a prediction) and a target output. The training system may then adjust the artificial neural network according to a learning rule and an internal probability-weighted association between the difference between the output and the target output. Successive adjustments may cause the artificial neural network to generate outputs (e.g., one or more predictions) that are closer to the target output.

[0023] In some embodiments, an artificial neural network may include any software and / or hardware comprised of interconnected processing nodes. These processing nodes, which may be referred to as "artificial neurons," may receive inputs and communicate outputs to other artificial neurons. The output of each artificial neuron may be determined by a nonlinear functional combination of each of the inputs to that artificial neuron, and each connection between artificial neurons may be assigned a "weight" that determines the degree to which a particular connection contributes to the output of a destination neuron.

[0024] Artificial neural networks can be used in a variety of contexts, including, but not limited to, computer vision (eg, image recognition and object detection), natural language processing (eg, translation and speech recognition), medical diagnosis and recommendation systems.

[0025] An artificial neural network may be implemented in a variety of ways. In some embodiments, an artificial neural network may be implemented as a software program executing on one or more physical processing units and / or any other suitable form of computer readable instructions. In further embodiments, an artificial neural network may be implemented in physical hardware, such as a series of interconnected physical processing units, each processing unit acting as an artificial neuron. Thus, although some examples described herein may describe and / or illustrate machine learning model 150 in the context of a software-based artificial neural network, in some examples, machine learning model 150 may be implemented in any suitable physical hardware.

[0026] The example system 100 of Figure 1 may be implemented in a variety of ways. For example, all or a portion of the example system 100 may represent a portion of an example system 200 ("system 200") of Figure 2. As shown in Figure 2, the system 200 may include a computing device 202. In at least one example, the computing device 202 may be programmed with one or more modules 102.

[0027] 1, when executed by the computing device 202, may cause the computing device 202 to perform one or more operations to register the digital slide images. For example, as described in greater detail below, the receiving module 104, when executed by the computing device 202, may cause the computing device 202 to receive an image of a section of a cuttable tissue sample block (e.g., image 204). The received image may include image data of a section of the tissue sample embedded in the tissue sample block (e.g., tissue sample image 206) and a section of a cuttable fiducial marker embedded in the tissue sample block (e.g., cuttable fiducial marker image 208).

[0028] Additionally, the identification module 106, when executed by the computing device 202, may cause the computing device 202 to identify attributes (e.g., attributes 210) of the cuttable fiducial marker section from the received image 204 of the cuttable tissue sample block section. The attributes may include any suitable attribute, including, but not limited to, at least a wavelength (e.g., color) of light reflected by the cuttable fiducial marker section, a shape of the cuttable fiducial marker section, an orientation of the cuttable fiducial marker section, or a position of the cuttable fiducial marker section within the cuttable tissue sample block section.

[0029] The identification module 106, when executed by the computing device 202, may also cause the computing device 202 to identify attributes of the tissue sample section from the received image 204. The attributes may include any suitable attributes including, but not limited to, the morphology, phenotype, shape, size, at least one wavelength (e.g., color) of the tissue sample, etc.

[0030] In some embodiments, the tissue sample image 206 may include an image of a tissue array that includes multiple sections of tissue from one or more patients. The identification module 106, when executed by the computing device 202, may cause the computing device 202 to identify the location of each tissue section in the tissue array relative to the section of the cuttable fiducial markers. This location information may facilitate automated identification of each tissue section in the tissue array and its source (e.g., the patient from which it was obtained). Additionally, the location information may facilitate providing a single image or multiple images of tissue samples from an individual patient or multiple patients for review by a user (e.g., a pathologist).

[0031] 2, one or more of the systems described herein may optionally include and / or perform operations involving images of additional sections of the cuttable tissue block (e.g., additional image 214). For example, in some embodiments, the receiving module 104 may optionally receive images of additional sections of the cuttable tissue block (e.g., additional image 214), which may include images of additional sections of the tissue sample (e.g., additional tissue sample image 216) and images of additional sections of the cuttable fiducial markers (e.g., additional cuttable fiducial marker image 218).

[0032] Additionally, the execution module 108, when executed by the computing device 202, may cause the computing device 202 to perform a tissue sample management operation (e.g., tissue sample management operation 212) based on the determined attributes of the section of the cuttable fiducial marker. For example, the execution module 108 may determine an orientation (e.g., orientation 220) of the cuttable fiducial marker based on attributes of the cuttable fiducial marker (e.g., the shape of the cuttable fiducial marker, the wavelength gradient of light reflected by the section of the cuttable fiducial marker, etc.). In some examples, one or more of the modules 102 (e.g., the execution module 108) may further provide one or more images to a user via a suitable graphical user interface (e.g., graphical user interface 222). In some examples, the graphical user interface 222 is separate from the computing device 202 and may communicate (e.g., wirelessly and / or via wired communication) with the computing device 202 to enable a user to remotely access information (e.g., the image 204 and / or the further image 214) from the computing device 202.

[0033] Computing device 202 generally refers to any type or form of computing device capable of reading and / or executing and / or hosting computer-executable instructions. Examples of computing device 202 include, without limitation, an application server, a storage server, a database server, a web server, and / or any other suitable computing device configured to run a software application and / or provide various application, storage, and / or database services.

[0034] In at least one example, the computing device 202 can be a computing device programmed with one or more modules 102. All or a portion of the functionality of the modules 102 can be performed by the computing device 202 and / or any other suitable computing system. As described in greater detail below, one or more of the modules 102 of FIG. 1, when executed by at least one processing unit of the computing device 202, can enable the computing device 202 to register digital slide images (e.g., image 204 and further image 214) in any manner described herein.

[0035] Numerous other devices or subsystems may be connected to system 100 of FIG. 1 and / or system 200 of FIG. 2. Conversely, not all components and devices shown in FIG. 1, 2 need be present to practice the embodiments described and / or illustrated herein. The devices and subsystems may be interconnected in ways other than those shown in FIG. 2. Systems 100, 200 may use any number of software, firmware, and / or hardware configurations. For example, one or more embodiments disclosed herein may be encoded as a computer program (also referred to as computer software, software applications, computer readable instructions, and / or computer control logic) on a computer readable medium.

[0036] Figure 3 is a flow diagram of an exemplary computer-implemented method 300 for registering digital slide images. The steps illustrated in Figure 3 may be performed by any suitable computer-executable code and / or computing system including the system of Figure 1, the system 200 of Figure 2, and / or one or more variations or combinations thereof. In one example, each step illustrated in Figure 3 may represent an algorithm whose structure includes and / or is represented by multiple sub-steps, some examples of which are detailed below.

[0037] 3, at step 310, one or more systems described herein may receive an image of a section of a cuttable tissue sample block. The image may include image data of the tissue sample section and the cuttable fiducial marker section. For example, the receiving module 104, as part of the computing device 202, may receive an image 204 from the data store 140, which may include a tissue sample image 206 and a cuttable fiducial marker image 208. In some examples, the receiving module 104 may also receive an additional image 214 adjacent, separate, or overlapping with the image 204, as described above.

[0038] The material forming the cuttable fiducial markers described herein can be embedded in a wax block and infused with paraffin without significant change in size or shape. The material can be cut cleanly on a microtome without dulling the blade. By way of example and not limitation, the material forming the fiducials can include polymeric materials, gelatinous materials, and / or tissue-mimicking materials. For example, the tissue-mimicking materials can include biological and / or synthetic materials that mimic biological samples. In some embodiments, the tissue-mimicking materials can include at least one biological cell, protein material, and / or lipid material. Tissue-mimicking materials suitable for forming the cuttable fiducials of the present disclosure are disclosed in U.S. Patent No. 9,851,349, issued December 26, 2017, and entitled "MATRIX FOR RECEIVING A TISSUE SAMPLE AND USE THEREOF," the entire disclosure of which is incorporated herein by reference. Exemplary materials that can be used to form the fiducials can include at least one of a protein (e.g., animal protein), one or more lipids (e.g., animal fats, vegetable oils, etc.), glycerin, water, a gelling agent (e.g., an ionic gelling agent), an inorganic buffer material, an antifoaming agent, and / or a paraffin wax material.

[0039] In some examples, the cuttable fiducial markers can include a material or gaps in a material such that the fiducial markers are optically distinguishable from the material surrounding the section (e.g., paraffin wax). For example, the cuttable fiducial markers can have a distinguishable shape (e.g., rectangle, square, triangle, star, arrow, cross, circle, parallelogram, trapezoid, etc.), orientation, and / or size. In some examples, multiple cuttable fiducial markers can be included on each section to improve and facilitate image registration. In some embodiments, the shape, size, and orientation of surrounding structures of the tissue sample can themselves be used as fiducials for digital image registration.

[0040] By way of example and / or illustration, Figure 4A is a plan view of an example of a cuttable fiducial marker 400 that may be embedded in a tissue sample block according to some embodiments described herein. Figure 4B shows a perspective view of the cuttable fiducial marker 400. The cuttable fiducial marker 400 may be included as part of a cuttable tissue sample block and / or may be formed from a cuttable material.

[0041] As shown, the cuttable fiducial marker 400 can have a distinct and / or identifiable shape or aspect that can indicate a particular direction and / or orientation. It is noted that the shape of the cuttable fiducial marker 400 is shown for illustrative purposes only, and the cuttable fiducial marker can have any suitable shape, form, and / or aspect.

[0042] 4B , cuttable fiducial marker 400 can be cut (i.e., cut transversely) into multiple fiducial marker sections 402 (e.g., fiducial marker section 402(1), fiducial marker section 402(2), fiducial marker section 402(3)). When cuttable fiducial marker 400 is embedded in a tissue sample block as described herein and the tissue sample block is cut as described herein, each section of the cut tissue sample block can include a different fiducial marker section 402.

[0043] In a further embodiment, a tissue block may be formed, one or more holes may be formed in the tissue block, and the one or more holes may be filled with a cuttable substance that may be optically distinguishable from the surrounding material of the tissue block after cutting is completed. For example, the cuttable substance may include a colored paraffin wax substance, a particulate substance (e.g., pigments), etc.

[0044] A cuttable structure may be used to hold or surround the tissue, limiting the area in which the tissue may be found and containing markers that may correspond to the orientation of the tissue. Thus, in some embodiments, a tissue sample may be placed into a preformed structure containing one or more cuttable fiducial markers. The tissue sample and structure may be processed together to form a tissue block. Thus, the cuttable fiducial markers and tissue sample may be embedded into a cuttable tissue sample block. Sections of the tissue block containing the tissue sample and the structure containing the cuttable fiducial markers may be obtained (e.g., cut with a microtome) and placed on a slide.

[0045] For example, FIGS. 5A-5C show a set of tissue sample slides 500 (e.g., tissue sample slide 500(1), tissue sample slide 500(2), tissue sample slide 500(3)), each including a different section of a cut tissue sample block. As shown, each slide can include a structure (e.g., a rectangular structure) that at least partially surrounds each section of the tissue sample. The structure can be a cuttable structure that can include one or more cuttable fiducial markers. The one or more cuttable fiducial markers, alone or in combination with other fiducial markers, can have a distinguishable orientation to facilitate automatic orientation of a corresponding digital slide image.

[0046] In some examples, digital images of the tissue sample slide may be acquired (e.g., via one or more digital imaging devices) thus generating one or more images of one or more sections of the cut tissue sample block. These images may be included as part of the image data 142, and each image may include image data of the tissue sample section and image data of the cuttable fiducial marker section. The receiving module 104 may receive images 204, which may include a tissue sample image 206 and a cuttable fiducial marker image 208. As described in greater detail below, one or more modules 102 (e.g., identification module 106) may identify fiducial markers within the images of the structures (e.g., sections of the cuttable fiducial markers), and one or more modules 102 (e.g., execution module 108) may register at least some of the images with one another based on the position, size, and orientation of each fiducial marker.

[0047] 3, at step 320, one or more systems described herein may determine attributes of the section of the cuttable fiducial marker from the received image of the section of the cuttable tissue sample block. For example, the determination module 106, as part of the computing device 202, may determine attributes 210 of the section of the cuttable fiducial marker from the image 204 and / or the cuttable fiducial marker image 208.

[0048] Attributes 210 may include any suitable attributes of the cuttable fiducial marker that may be determined from an image of the cuttable fiducial marker, including, but not limited to, the wavelength (e.g., color) of light reflected by a section of the fiducial marker, the shape of the section of the cuttable fiducial marker, the orientation of the section of the cuttable fiducial marker, the position of the cuttable fiducial marker within the section of the cuttable tissue sample block, etc.

[0049] The identification module 106 may identify the attributes 210 in a variety of contexts. For example, the identification module 106 may receive and / or analyze any suitable data or metadata associated with the images 204, the tissue sample images 206, and / or the cuttable fiducial marker images 208 to identify the attributes 210 using any suitable image analysis method or technique.

[0050] In some examples, the identification module 106 may identify the attributes 210 by analyzing images of sections of the cuttable tissue sample block with a machine learning model that has been pre-trained to predict attributes of sections of cuttable fiducial markers based on images of the sections of the cuttable fiducial markers. As described above, a machine learning model such as an artificial neural network may be pre-trained (e.g., with a plurality of pre-labeled images of sections of cuttable fiducial markers) to predict attributes of sections of the cuttable fiducial markers when provided with a new image of a section of the cuttable fiducial markers. The identification module 106 may provide the cuttable fiducial marker image 208 to a machine learning model 150 that may predict attributes of sections of the cuttable fiducial markers imaged in the cuttable fiducial marker image 208. Thus, the identification module 106 may assign attributes received as output from the machine learning model 150 as attributes 210.

[0051] In some examples, the identification module 106 may train the machine learning model 150 to predict attributes of the sections of the cleavable fiducial markers by analyzing a plurality of pre-labeled images of the sections of the cleavable fiducial markers with a predetermined machine learning model training method (e.g., a supervised training method). The identification module 106 may then adjust one or more attributes (e.g., one or more parameters, one or more hyperparameters, one or more mathematical models, etc.) of the machine learning model based on the analysis of the plurality of pre-labeled images of the sections of the cleavable fiducial markers.

[0052] Using machine learning, cleavable fiducial markers can be reliably detected even in the presence of small image-to-image variations that may occur due to processing, cutting, placement on the slide, and / or scanning.

[0053] In a further example, the identification module 106 may train the machine learning model 150 to predict attributes (e.g., morphological and / or phenotypic attributes) of the tissue sample section by analyzing a plurality of pre-labeled images of the tissue sample section with a predetermined machine learning model training method (e.g., a supervised training method). The identification module 106 may then adjust one or more attributes (e.g., one or more parameters, one or more hyperparameters, one or more mathematical models, etc.) of the machine learning model based on the analysis of the plurality of pre-labeled images of the tissue sample section.

[0054] 3, at step 330, one or more systems described herein may perform a tissue sample management operation based on the identified attributes of the section of the severable fiducial marker. For example, the execution module 108, as part of the computing device 202, may perform the tissue sample management operation 212 based on the attributes 210.

[0055] In some examples, tissue sample management operations may include any operation on a tissue sample that may enable a pathologist and / or a tissue sample management system to access, analyze, maintain, and / or store the tissue sample. For example, tissue sample management operation 212 may include accessing records associated with the tissue sample and stored by the tissue sample management system based on attributes 210.

[0056] In at least one embodiment, the attributes 210 may include a shape of a section of the cuttable fiducial marker. In such an example, the tissue sample management operation 212 may include an operation for determining an orientation and / or location (e.g., orientation 220) of the section of the cuttable fiducial marker within a section of a tissue sample block based on the shape of the section of the cuttable fiducial marker. The orientation 220 may be determined with respect to a cuttable tissue sample block, a tissue sample slide including a section of the cuttable tissue sample block, a section of a tissue sample embedded in a section of a cuttable tissue sample block, etc.

[0057] The tissue sample management operations 212 may further include calculating a transformation using the orientation and / or position (e.g., orientation 220) of the cuttable fiducial markers to effectively offset, rotate, stretch, skew, or flip the images required to orient them within a unified coordinate system. In instances where multiple images (e.g., at least image 204 and further image 214) are included in the set, the unified coordinate system may be common to all images in the set. The unified coordinate system may then be used (e.g., by execution module 108) to register the images of the various sections of the tissue sample. This method may work equally well for different levels and different dye types, so long as all images are from the same paraffin block and contain structures with fiducial markers.

[0058] In a further example, the tissue sample management operations 212 performed by the execution module 108 may include operations based on one or more attributes (e.g., morphological and / or phenotypic attributes) of the tissue sample identified by the identification module 106. For example, the tissue sample management operations 212 may include operations of sending the tissue sample image 206 to a particular pathologist and / or a particular algorithm for review and possible diagnosis. If the identification module 106 identifies the tissue sample image 206 as indicating a high likelihood of abnormal tissue, the tissue sample image 206 may be classified as a complex case that requires careful review by a pathologist and / or other specialized image analysis algorithms. Thus, the tissue sample management operations 212 may send the complex image review task to an expert in the type of tissue associated with the tissue sample image 206. In another example, multiple complex image review tasks may be sent to various pathologists, each depending on a given workload (e.g., equally or approximately equally), or to multiple pathologists for diagnosis confirmation.

[0059] For example, referring back to FIGS. 5A-5C, FIGS. 5A-5C illustrate a set of tissue sample slides 500 (e.g., tissue sample slide 500(1), tissue sample slide 500(2), and tissue sample slide 500(3)), each including a different section of a tissue sample block. As illustrated, the tissue sample blocks included cuttable fiducial markers that indicate a common orientation for the different sections of the tissue sample block. As an example, sections 502 (e.g., section 502(1), section 502(2), and section 502(3)) may each represent a different section of a cuttable fiducial marker that was included in the tissue sample block prior to cutting the tissue sample block. Using the systems and methods described herein, one or more modules 102 (e.g., executive module 108) may identify a common orientation and / or a uniform coordinate system for the tissue sample slides 500. FIG. 6 includes a diagram 600 illustrating the actual or virtual alignment of a tissue sample slide 500 in a specified common orientation and / or within a specified unified coordinate system.

[0060] In some examples, the execution module 108 may perform the tissue sample management operation 212 via the graphical user interface 222 by further providing a tissue sample image 206 oriented based on the identified orientation of the section of the cuttable fiducial marker included in the cuttable fiducial marker image 208 within the graphical user interface 222.

[0061] Additionally or alternatively, embodiments of the present invention may be used to view and / or compare images of multiple different sections of a cuttable tissue sample block, as described throughout this specification. In such embodiments, tissue sample management operation 212 may include identifying orientations of different sections of cuttable fiducial markers included in different sections from the cuttable tissue sample block, which images may be included in image 204 and / or further image 214, for example. Thus, in some examples, executive module 108 may perform tissue sample management operation 212 via a graphical user interface, such as graphical user interface 222, by providing (1) tissue sample image 206 oriented based on the identified orientations of the sections of cuttable fiducial markers included in cuttable fiducial marker image 208 within graphical user interface 222, and (2) further tissue sample image 216 oriented based on the identified orientations of further sections of cuttable fiducial markers included in further cuttable fiducial marker image 218 within graphical user interface 222.

[0062] Using the unified coordinate system, multiple images from the set may be displayed overlaid in the same window (e.g., as in tissue sample slide 500 of diagram 600 of FIG. 6) within the graphical user interface 222. Additionally or alternatively, multiple images from the same set may be provided simultaneously in different windows within the graphical user interface 222, with the same orientation relative to the tissue sample, regardless of possible misalignment on the original slide (e.g., as shown in FIGS. 5A-5C).

[0063] In some examples, the execution module 108 may provide the tissue sample image 206 and the further tissue sample image 216 aligned in a common orientation based on the determined orientation of the section of the cuttable fiducial marker included in the cuttable fiducial marker image 208 and the determined orientation of the further section of the cuttable fiducial marker included in the further cuttable fiducial marker image 218. In some embodiments, this may include overlaying the tissue sample image 206 and the further tissue sample image 216 in the graphical user interface 222 and / or positioning the tissue sample image 206 adjacent to and in the same orientation as the further tissue sample image 216.

[0064] 7 illustrates a diagram 700 of a graphical user interface, such as graphical user interface 222. In this example, three different tissue sample section images 702 (e.g., tissue sample section image 702(1), tissue sample section image 702(2), and tissue sample section image 702(3)) are displayed in different windows with the same orientation relative to the tissue sample, regardless of possible misalignment on the original slide. As shown, the orientation of tissue sample image 702 may be determined based on images of the sections of cuttable fiducial marker 704 (e.g., section 704(1), section 704(2), and section 704(3)). Each tissue sample image 702 may be associated with a different image of a different section of the same cuttable fiducial marker (e.g., section 704(1) may be associated with tissue sample section 702(1), section 704(2) may be associated with tissue sample section 702(2), and section 704(3) may be associated with tissue sample section 702(3)). As shown by the dashed lines in FIG. 7, images of cuttable fiducial markers 704 may optionally be included in and / or provided with tissue sample section images 702. In other examples, images of sections 704 may be omitted.

[0065] In some examples, a user (e.g., a pathologist) may interact with the graphical user interface 222 by moving one of the images. In response, the execution module 108 may automatically move one or more of the other images to the corresponding second tissue sample location. Thus, embodiments of the disclosed method and system may enable synchronized panning and zooming across two or more images. To the user, the effect may appear very natural and may greatly reduce the time it takes a pathologist to review all levels of a case.

[0066] Additionally, a pathologist may compare a tissue sample stained with a standard stain (e.g., an "H&E" stain) side-by-side (or overlaid) with a tissue sample of the same area stained with a special stain. Embodiments of the present disclosure may also help ensure best practice (e.g., by reducing the likelihood that a pathologist will fail to examine all levels of a tissue sample).

[0067] In further embodiments, the executive module 108 may enable a user to interact with the graphical user interface 222 by automatically transferring and / or displaying slide annotation information entered by a user across multiple images. Annotation information from one image may be displayed transparently ("semi-transparently") in corresponding areas of other images. Thus, embodiments of the systems and methods described herein may facilitate digital or virtual techniques comparable to the analog or real ink dot techniques used by pathologists to overlay identified lesions on transparent (e.g., glass) slides, but with greater accuracy, overlap, and speed.

[0068] Semi-transparent annotation information may be applied across the images to generate a training data set for further machine learning models. For example, two or more images 702 may show tissue stained with different dyes. A tissue of interest (e.g., abnormal tissue) may be identified and annotated in one of the images 702 of tissue stained with a first dye, and the annotation information may be transferred to another image 702 of tissue stained with a second, different dye. The transferred annotation information may be added to a training data set for a machine learning model that may be specifically designed to examine images of tissue stained with the second, different dye.

[0069] As described throughout this disclosure, the disclosed systems and methods may provide one or more advantages over conventional options for registering and / or presenting tissue sample images and / or slides. Using embodiments of the present disclosure, digital slide images may be registered automatically without the need for user intervention (e.g., identifying similar and / or overlapping different levels of tissue structures, scaling images, etc.). Furthermore, in some examples, the methods and systems of the present disclosure may register images more accurately by identifying and using cuttable fiducial markers for registration compared to conventional techniques that may use tissue structures for registration.

[0070] Other possible use cases and / or embodiments may include Z-plane stacking (e.g., examining a third dimensional focal plane simultaneously), or scrolling of overlapping linked images within a single window instead of viewing multiple images side-by-side.

[0071] The following exemplary embodiments are also included in the present disclosure:

[0072] Example 1: A computer-implemented method comprising: (1) receiving an image of a section of a cuttable tissue sample block, the image including image data of (A) the tissue sample section and (B) a section of a cuttable fiducial marker embedded in the section of the cuttable tissue sample block; (2) identifying attributes of the section of the cuttable fiducial marker from the received image of the section of the cuttable tissue sample block; and (3) performing a tissue sample management operation based on the identified attributes of the section of the cuttable fiducial marker.

[0073] Example 2: The computer-implemented method of Example 1 further comprising the step of embedding at least one of a tissue sample and a cuttable fiducial marker in the cuttable tissue sample block prior to cutting the cuttable tissue sample block into a plurality of sections, the sections of the cuttable tissue sample block being included in the plurality of sections.

[0074] Example 3: The computer-implemented method of Example 1 or 2, wherein identifying attributes of the cuttable fiducial marker section from the received image of the cuttable tissue sample block section includes identifying at least one of: (1) at least one wavelength of light reflected by the cuttable fiducial marker section; (2) a shape of the cuttable fiducial marker section; (3) an orientation of the cuttable fiducial marker section; or (4) a position of the cuttable fiducial marker section within the cuttable tissue sample block section.

[0075] Example 4: The computer-implemented method of any one of Examples 1 to 3, wherein the step of performing a tissue sample management operation includes a step of determining an orientation of the section of the cuttable fiducial marker relative to at least one of (1) a cuttable tissue sample block, (2) a tissue sample slide including a section of the cuttable tissue sample block, or (3) a section of a tissue sample embedded in the section of the cuttable tissue sample block.

[0076] Example 5: The computer-implemented method of Example 4, wherein the step of performing a tissue sample management operation further includes providing, via the graphical user interface, a section of the tissue sample oriented within the graphical user interface based on the identified orientation of the section of the cuttable fiducial marker.

[0077] Example 6: The computer-implemented method of any one of Examples 1 to 5, further comprising: (1) receiving an image of a further section of the cuttable tissue sample block, the further section including image data of (A) the further section of the tissue sample and (B) the further section of the cuttable fiducial marker embedded in the further section of the cuttable tissue sample block; and (2) identifying attributes of the further section of the cuttable fiducial marker from the received image of the further section of the cuttable tissue sample block.

[0078] Example 7: The computer-implemented method of Example 6, wherein performing a tissue sample management operation includes determining an orientation of (1) a section of the cuttable fiducial marker relative to at least one of (A) the cuttable tissue sample block, (B) a tissue sample slide including a section of the cuttable tissue sample block, or (C) a section of a tissue sample embedded in the section of the cuttable tissue sample block, and (2) an orientation of an additional section of the cuttable fiducial marker relative to at least one of (A) the cuttable tissue sample block, (B) a further tissue sample slide including a further section of the cuttable tissue sample block, or (C) a further section of a tissue sample embedded in the further section of the cuttable tissue sample block.

[0079] Example 8: The computer-implemented method of Example 7, wherein the tissue sample management operation further includes providing, via the graphical user interface, (1) image data of a section of the tissue sample oriented within the graphical user interface based on a specified orientation of the section of the cuttable fiducial marker, and (2) image data of an additional section of the tissue sample oriented within the graphical user interface based on a specified orientation of the additional section of the cuttable fiducial marker.

[0080] Example 9: The computer-implemented method of Example 8, further comprising providing image data of the section of the tissue sample and image data of the further section of the tissue sample aligned in a common orientation based on the identified orientation of the section of the cuttable fiducial marker and the identified orientation of the further section of the cuttable fiducial marker.

[0081] Example 10: The computer-implemented method of Example 9, wherein the step of providing image data of the tissue sample section and the image data of the further section of the tissue sample aligned to a common orientation based on the identified orientation of the section of the cuttable fiducial marker and the identified orientation of the further section of the cuttable fiducial marker includes a step of overlaying the image data of the tissue sample section with the image data of the further section of the tissue sample.

[0082] Example 11: The computer-implemented method of any one of Examples 7 to 10, wherein the tissue sample management operations further include: (1) receiving annotation information from a user for a portion of image data of a section of the tissue sample; (2) transferring the annotation information to a corresponding portion of image data of an additional section of the tissue sample; and (3) adding the transferred annotation information to a training dataset of a machine learning model for analyzing image data of the tissue sample that shares at least one attribute with the additional section of the tissue sample.

[0083] Example 12: The computer-implemented method of any one of Examples 1 to 11, wherein identifying attributes of a section of the cuttable fiducial marker from a received image of a section of the cuttable tissue sample block includes analyzing the image of the section of the cuttable tissue sample block with a machine learning model pre-trained to predict attributes of a section of the cuttable fiducial marker based on the image of the section of the cuttable fiducial marker.

[0084] Example 13: The computer-implemented method of Example 12, further comprising training a machine learning model to predict attributes of a section of the cuttable fiducial marker, the training step comprising: (1) analyzing a plurality of pre-labeled images of the section of the cuttable fiducial marker using a predetermined machine learning model training method; and (2) adjusting at least one attribute of the machine learning model based on the analysis of the plurality of pre-labeled images of the section of the cuttable fiducial marker.

[0085] Example 14: A system including: (1) a receiving module stored in memory and configured to receive an image of a section of a cuttable tissue sample block, the image including image data of (A) the tissue sample section and (B) a section of a cuttable fiducial marker embedded in the section of the cuttable tissue sample block; (2) an identifying module stored in memory and configured to identify attributes of the section of the cuttable fiducial marker from the received image of the section of the cuttable tissue sample block; (3) an execution module stored in memory and configured to perform a tissue sample management operation based on the identified attributes of the section of the cuttable fiducial marker; and (4) at least one physical processing unit that executes the receiving module, the identifying module, and the execution module.

[0086] Example 15: The system described in Example 14, wherein the identification module identifies attributes of the section of the cuttable fiducial marker from the received image of the section of the cuttable tissue sample block by identifying at least one of: (1) at least one wavelength of light reflected by the section of the cuttable fiducial marker, (2) the shape of the section of the cuttable fiducial marker, (3) the orientation of the section of the cuttable fiducial marker, or (4) the position of the section of the cuttable fiducial marker within the section of the cuttable tissue sample block.

[0087] Example 16: The system of Example 14 or 15, wherein the execution module performs a tissue sample management operation by identifying an orientation of a section of the cuttable fiducial marker relative to at least one of (1) a cuttable tissue sample block, (2) a tissue sample slide including a section of the cuttable tissue sample block, or (3) a tissue sample section embedded in the section of the cuttable tissue sample block.

[0088] Example 17: The system described in Example 16, wherein the execution module executes a tissue sample management operation via a graphical user interface by providing a section of the tissue sample oriented within the graphical user interface based on a specified orientation of the section of the cuttable fiducial marker.

[0089] Example 18: A system described in any one of Examples 14 to 17, wherein the identification module identifies attributes of a section of the cuttable fiducial marker from a received image of a section of the cuttable tissue sample block by analyzing the image of the section of the cuttable tissue sample block with a machine learning model pre-trained to predict attributes of a section of the cuttable fiducial marker based on the image of the section of the cuttable fiducial marker.

[0090] Example 19: The system described in Example 18, wherein the identification module is further configured to train a machine learning model to predict attributes of the cut portion of the cut fiducial marker by (1) analyzing a plurality of pre-labeled images of the cut portion of the cut fiducial marker using a predetermined machine learning model training method, and (2) adjusting at least one attribute of the machine learning model based on the analysis of the plurality of pre-labeled images of the cut portion of the cut fiducial marker.

[0091] Example 20: A non-transitory computer readable medium including computer readable instructions, which when executed by at least one processing unit of a computing system, cause the computing system to: (1) receive an image of a section of a cuttable tissue sample block, the image including image data of (A) the tissue sample section and (B) a section of a cuttable fiducial marker embedded in the section of the cuttable tissue sample block; (2) identify attributes of the section of the cuttable fiducial marker from the received image of the section of the cuttable tissue sample block; and (3) perform a tissue sample management operation based on the identified attributes of the section of the cuttable fiducial marker.

[0092] As noted above, the computing devices and systems described and / or illustrated herein broadly refer to any type or form of computing device or system capable of executing computer-readable instructions, such as those contained in the modules described herein. In their most basic configurations, each such computing device may include at least one memory device and at least one physical processing unit.

[0093] Although shown as separate elements, the modules described and / or illustrated herein may represent portions of a single module or application. Moreover, in some embodiments, one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, cause the computing device to perform one or more tasks. For example, one or more of the modules described and / or illustrated herein may represent modules stored and configured to execute on one or more of the computing devices or systems described and / or illustrated herein. One or more of these modules may also represent all or a portion of one or more special purpose computers configured to perform one or more tasks.

[0094] Additionally, one or more of the modules described herein may convert data, a physical device, and / or a representation of a physical device from one form to another. For example, one or more of the modules described herein may receive image data to be converted, convert the image data, output a conversion result to register a tissue slide image included in the image data, use the conversion result to provide a registered tissue slide image included in the image data, and further store the conversion result to train a machine learning model to recognize an image of a cuttable fiducial marker. Additionally or alternatively, one or more of the modules described herein may convert a processing unit, a volatile memory, a non-volatile memory, and / or any other portion of a physical computing device from one form to another by executing on a computing device, storing data on a computing device, and / or otherwise interacting with a computing device.

[0095] The term "computer-readable medium" as used herein generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmission-type media such as carrier waves, as well as non-transitory media such as magnetic storage media (e.g., hard disk drives, tape drives, and floppy disks), optical storage media (e.g., compact discs (CDs), digital video discs (DVDs), and BLU-RAY discs), electronic storage media (e.g., solid-state drives and flash media), and other distribution systems.

[0096] The process parameters and order of steps described and / or illustrated herein are exemplary only and may be varied as desired. For example, although the steps illustrated and / or described herein may be shown or described in a particular order, these steps do not necessarily have to be performed in the order shown or described. Various exemplary methods described and / or illustrated herein may omit one or more of the steps described or illustrated herein or may include additional steps to those disclosed.

[0097] The above description is provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not exhaustive and is not intended to limit the present invention to any precise form disclosed. Numerous changes and modifications are possible without departing from the spirit and scope of the present disclosure. The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. Reference should be made to the appended claims and equivalents to determine the scope of the present disclosure.

[0098] Unless otherwise indicated, the terms "connected" and "coupled" (and derivatives) used in this specification and claims should be interpreted as allowing both direct and indirect (i.e., via other elements or components) connections. Furthermore, in this specification and claims, the original English indefinite article should be interpreted as "at least one." Finally, for ease of usage, the terms "including" and "having" (and derivatives) used in this specification and claims are interchangeable and have the same meaning as the term "comprising." [Explanation of symbols]

[0099] 104 Receiver Module 106 Specific Modules 108 Execution Module 120 Memory 130 Physical Processing Unit

Claims

1. In a computer-assisted method, receiving an image of a section of a cuttable tissue sample block, the image including image data of a section of a tissue sample and a section of a cuttable fiducial marker embedded in the section of the cuttable tissue sample block; identifying attributes of the section of the cuttable fiducial marker from the received image of the section of the cuttable tissue sample block; performing a tissue sample management operation based on the identified attributes of the section of the severable fiducial marker; A method comprising:

2. determining the attributes of the section of the cuttable fiducial marker from the received image of the section of the cuttable tissue sample block, at least one wavelength of light reflected by the segment of the severable fiducial marker; the shape of the section of the severable fiducial marker; the orientation of the section of the severable fiducial marker; or the location of the section of the severable fiducial marker within the section of the severable tissue sample block; 10. The computer-implemented method of claim 1, further comprising identifying at least one of:

3. The step of performing the tissue sample management operation includes: the cuttable tissue sample block; a tissue sample slide containing said section of said cuttable tissue sample block; or the section of the tissue sample embedded in the section of the cuttable tissue sample block 10. The computer implemented method of claim 1, further comprising determining an orientation for at least one of the images.

4. 4. The computer-implemented method of claim 3, wherein performing the tissue sample management operation further comprises providing, via a graphical user interface, the section of the tissue sample oriented within the graphical user interface based on the identified orientation of the section of the severable fiducial marker.

5. receiving an image of a further section of the cuttable tissue sample block, the further section including image data of a further section of the tissue sample and a further section of the cuttable fiducial marker embedded in the further section of the cuttable tissue sample block; determining attributes of the further section of the cuttable fiducial marker from the received image of the further section of the cuttable tissue sample block; The computer-implemented method of claim 1 further comprising:

6. performing the tissue sample management operation; The section of the severable fiducial marker the cuttable tissue sample block; a tissue sample slide containing said section of said cuttable tissue sample block; or the section of the tissue sample embedded in the section of the cuttable tissue sample block an orientation to at least one of the the further section of the severable fiducial marker the cuttable tissue sample block; a further tissue sample slide comprising said further section of said cuttable tissue sample block; or the further section of the tissue sample embedded in the further section of the cuttable tissue sample block 6. The computer implemented method of claim 5, further comprising determining an orientation for at least one of the images.

7. The tissue sample management operations are performed via a graphical user interface, the image data of the section of the tissue sample oriented within the graphical user interface based on the identified orientation of the section of the cuttable fiducial marker; the image data of the further section of the tissue sample oriented within the graphical user interface based on the identified orientation of the further section of the cuttable fiducial marker; 7. The computer-implemented method of claim 6, further comprising the step of providing:

8. providing the image data of the section of the tissue sample and the image data of the further section of the tissue sample aligned in a common orientation based on the identified orientation of the section of the cuttable fiducial marker and the identified orientation of the further section of the cuttable fiducial marker; The computer-implemented method of claim 7 further comprising:

9. 9. The computer-implemented method of claim 8, wherein providing the image data of the section of the tissue sample and the image data of the further section of the tissue sample aligned to a common orientation based on the identified orientation of the section of the cuttable fiducial marker and the identified orientation of the further section of the cuttable fiducial marker includes overlaying the image data of the section of the tissue sample with the image data of the further section of the tissue sample.

10. The tissue sample management operation includes: receiving annotation information from a user for a portion of the image data of the section of the tissue sample; transferring the annotation information to a corresponding portion of the image data of the further section of the tissue sample; adding the transferred annotation information to a training dataset of a machine learning model for analyzing image data of tissue samples that share at least one attribute with the further sections of the tissue sample; 7. The computer-implemented method of claim 6, further comprising:

11. 2. The computer-implemented method of claim 1, wherein identifying attributes of the section of the cuttable fiducial marker from the received image of the section of the cuttable tissue sample block comprises analyzing the image of the section of the cuttable tissue sample block with a machine learning model pre-trained to predict attributes of the section of the cuttable fiducial marker based on the image of the section of the cuttable fiducial marker.

12. In the system, a receiving module stored in memory and configured to receive an image of a section of a severable tissue sample block, the image including image data of a section of a tissue sample and a section of a severable fiducial marker embedded in the section of the severable tissue sample block; an identification module stored in memory and configured to identify attributes of the section of the severable fiducial marker from the received image of the section of the severable tissue sample block; an execution module stored in memory and configured to perform a tissue sample management operation based on the identified attribute of the section of the severable fiducial marker; at least one physical processing unit that executes the receiving module, the identifying module, and the executing module; A system including:

13. The identification module identifies the attributes of the section of the cuttable fiducial marker from the received image of the section of the cuttable tissue sample block by: at least one wavelength of light reflected by the segment of the severable fiducial marker; the shape of the section of the severable fiducial marker; the orientation of the section of the severable fiducial marker; or the location of the section of the severable fiducial marker within the section of the severable tissue sample block; The system of claim 12 , wherein the identifying step is performed by identifying at least one of the following:

14. The execution module executes the tissue sample management operation on the section of the severable fiducial marker. the cuttable tissue sample block; a tissue sample slide containing said section of said cuttable tissue sample block; or the section of the tissue sample embedded in the section of the cuttable tissue sample block The system of claim 12 , wherein the determining step is performed by determining an orientation relative to at least one of the following:

15. 15. The system of claim 14, wherein the execution module performs the tissue sample management operation via a graphical user interface by providing the section of the tissue sample oriented within the graphical user interface based on the identified orientation of the section of the severable fiducial marker.

Images