Image atlas systems and methods

Abstract

To provide analysis of images for clinical use.SOLUTION: In some embodiments, a process and a system are provided for generating a user interface for classification of a sample image of a cell. The classification includes receiving a sample image of a sample particle from a biological sample, and selecting reference images that each portray a reference particle of a biological sample. The reference images can be ordered based on similarity, and the reference images can be selected based on the order. A first selected reference image can be aligned with the sample image and expanded such that the adjacent edges of the reference image and sample image are the same. The expanded image can be dynamically filled. The sample image and the expanded reference image can be displayed in a user interface.SELECTED DRAWING: None

Description

[Technical field]

[0001] The present disclosure relates generally to the analysis of images for clinical applications, and more particularly, but not exclusively, to the analysis of images of cells and particles collected for clinical diagnostics, such as hematology and urinalysis, and normalization of the display of the images to allow a user to perform clinical analysis without confusion based on inappropriate discrepancies between images. Summary of the Invention [Means for solving the problem]

[0002] Analysis of cells or particles from living tissues (e.g., human, animal, and plant) can be used as a medical diagnostic tool used to identify disease and cellular disorders as well as healthy cells. Capturing cells or particles for analysis can be done, for example, by collecting the particles through a fluid from the living tissue (i.e., a biological sample). For example, a blood sample from a human contains cells that are analyzed to determine if the cells are healthy or have some problem that can be analyzed. For example, a blood sample from a patient under the care of a physician can be evaluated using a hematology system equipped to obtain multiple light angle detection parameters, such as the system described in patent application Ser. No. 14 / 775,448, entitled "HEMATOLOGY SYSTEMS AND METHODS," filed Sep. 11, 2015, and published Feb. 11, 2016 as US2016-0041083A1. As another example, a urine sample can be evaluated. To analyze a cell or particle, it may be beneficial to image the cell and perform a comparison of the cell to images of other cells with known characteristics. Some analysis and comparison can be done through automated systems, but not all cell images are clear enough, have known issues, or are similar enough to images of cells with known issues for automation to work properly or effectively. In that case, manual comparison by medical diagnostic personnel can be done. If necessary, a user interface can be provided that provides images of sample cells to be diagnosed or classified, and reference images of cells with known characteristics. The sample image for classification may have differences from the reference image that may be confounded by comparison, even though the cells in the reference image and the sample image may have the same features or characteristics, or may be the same type of cell. Differences such as image size, orientation, and gaps between displayed images are all examples of differences that may be confounded by comparison of cell images. The embodiments described herein can compensate for such confounds and provide an optimized user interface for comparing a sample to a reference image using an image atlas.

[0003] Unless expressly stated otherwise, references made in this disclosure to a "particle" or "particles" are understood to encompass any discrete or formed object dispersed in a fluid. As used herein, a "particle" can include any measurable and detectable (e.g., by image and / or other measurable parameters) constituent in a biological fluid. A particle can be of any substance, any shape and any size. A particle can include a cell. Examples of particles include, but are not limited to, blood cells, fetal cells, epithelial cells, stem cells, cells including tumor cells, or bacteria, parasites, or debris of any of the foregoing or other debris in a biological fluid. A blood cell can be any blood cell, including any normal or abnormal, mature or immature cell potentially present in a biological fluid, such as red blood cells (RBCs), white blood cells (WBCs), platelets (PLTs), and other cells. Constituents further include immature or abnormal cells. Immature WBCs can include metamyelocytes, myelocytes, promyelocytes, and blasts. In addition to mature RBCs, the constituents of RBCs may include nucleated RBCs (NTRCs) and reticulocytes. PLTs may include "giant" PLTs and PLT aggregates. Platelets, reticulocytes, nucleated RBCs, and WBCs including neutrophils, lymphocytes, monocytes, eosinophils, basophils, and immature WBCs including blasts, promyelocytes, myelocytes, or metamyelocytes are counted and analyzed as particles. Throughout this specification, an image is described as being an image of a cell or particle. Although often referred to as a cell, an image can be any particle. "Image" means any visual impression produced by an optical device such as a lens, mirror, camera, or microscope.

[0004] Exemplary urine particles may include urinary sediment particles. Exemplary urine sediment particles may include red blood cells (RBCs), dysmorphic red blood cells, white blood cells (WBCs), neutrophils, lymphocytes, phagocytes, eosinophils, basophils, squamous epithelial cells, transitional epithelial cells, decoy cells, renal tubular epithelial cells, casts, crystals, bacteria, yeast, parasites, oval fat bodies, fat droplets, sperm, mucus, trichomonas, cell clumps, and cell debris. Exemplary cells may include red blood cells, white blood cells, and epithelium. Exemplary casts may include acellular pigmented casts, unclassifiable casts (e.g., granular casts). Exemplary acellular casts may include, for example, waxy casts, broad casts, fatty casts, and crystalline casts. Exemplary cellular casts may include, for example, RBC casts, WBC casts, and cellular casts. Exemplary crystals may include, for example, calcium oxalate, triple phosphate, calcium phosphate, uric acid, calcium carbonate, leucine, cystine, tyrosine, and amorphous crystals. Exemplary non-squamous epithelial cells may include, for example, renal epithelium and transitional epithelium. Exemplary yeast may include, for example, Saccharomyces cerevisiae and yeast with pseudohyphae. Exemplary urinary sediment particles may also include RBC aggregates, fat, oval fat bodies, and trichomonas. The described systems and methods may be useful for characterizing, classifying and subclassifying, counting, and analyzing particles in biological fluids, such as, for example, detection and quantification of red blood cells (e.g., RBCs), dysmorphic red blood cells, white blood cells (e.g., WBCs), neutrophils, lymphocytes, phagocytes, eosinophils, basophils, squamous epithelial cells, transitional epithelial cells, decoy cells, renal tubular epithelial cells, casts, crystals, bacteria, yeast, parasites, oval fat bodies, fat droplets, sperm, mucus, trichomonas, cell aggregates, and cell debris.

[0005] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present invention. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting.

[0006] The systems described in the various figures may be provided in a variety of configurations. If desired, the systems may be configured as distributed systems in which one or more system components are distributed across one or more networks of a cloud computing system. All of the features of the systems described may be applied mutatis mutandis to the methods described, and vice versa. The present invention provides, for example, the following: (Item 1) A medical device for assisting in the analysis of biological sample particles, comprising: A processor; A memory device storing instructions executable by the processor, the instructions causing the processor to: receiving a sample image of a sample particle from a biological sample; selecting a plurality of reference images from a database of reference images, each reference image of the plurality of reference images being a reference image of a reference particle from a plurality of reference particles from one or more biological samples; determining an order for the plurality of reference images based on a similarity of each reference image to the sample image; selecting a first reference image from the plurality of reference images based on the determined order; aligning the first reference image with the sample image; dynamically filling a background of the aligned first reference image; and displaying the sample image and the dynamically filled and aligned first reference image on a user interface. (Item 2) The instructions for determining the order of the plurality of reference images may include instructions for the processor: comparing the sample image to each of the plurality of reference images; 2. The medical device of claim 1, further comprising instructions executable by the processor to cause the processor to: determine a similarity of each of the reference images to the sample image. (Item 3) The instructions for comparing the sample image to each reference image of the plurality of reference images include instructions for the processor: generating feature vectors for each of the reference images of the plurality of reference images and for the sample image; 3. The medical device of claim 1 or 2, further comprising instructions executable by the processor to cause the processor to calculate a distance metric for each of the plurality of reference images from the sample image. (Item 4) 4. The medical device of claim 3, wherein features used to generate the feature vector for each of the reference images of the plurality of reference images and the sample image include at least one of patch size, particle size, particle shape, average pixel intensity, and image mean grayscale value. (Item 5) The instructions for aligning the first reference image with the sample image may include instructions for the processor to: determining a first display position for displaying the sample image in the user interface, the first display position being within a first cell of a grid of cells; determining a second display position adjacent to the first display position for displaying the first reference image in the user interface, the second display position being within a second cell of the grid of cells; The medical device described in items 1 to 4, further comprising instructions executable by the processor to cause the processor to align a centroid of the first reference image in the second display position with a centroid of the sample image in the first display position. (Item 6) The instructions for dynamically filling the background of the aligned first reference image may include instructions for the processor to: Stretching the background of the first reference image in the second cell to fill the second cell; and determining a background color of the first reference image; 6. The medical device according to items 1 to 5, further comprising instructions executable by the processor to cause the processor to fill the enlarged background of the first reference image with the background color. (Item 7) The instructions for determining the background color of the first reference image may include instructions for the processor to: determining four corner pixels of the first reference image; 7. The medical device of claim 6, further comprising instructions executable by the processor to cause the processor to: calculate an average color value of the four corner pixels. (Item 8) 8. The medical device according to items 1 to 7, wherein the plurality of reference images are selected based on a tag associated with each reference image of the plurality of reference images. (Item 9) 1. A method comprising: receiving, at the computer system, a sample image of a sample particle from a biological sample; selecting, by the computer system, a plurality of reference images from a database of reference images, each reference image of the plurality of reference images being a reference image of a reference particle from a plurality of reference particles from one or more biological samples; determining, by the computer system, an order for the plurality of reference images based on a similarity of each reference image to the sample image; selecting, by the computer system, a first reference image from the plurality of reference images based on the determined order; aligning, by the computer system, the first reference image with the sample image; dynamically filling, by the computer system, a background of the aligned first reference image; displaying, by the computer system, the sample image and the dynamically filled and aligned first reference image in a user interface. (Item 10) determining the order of the plurality of reference images comparing, by the computer system, the sample image to each of the plurality of reference images; and determining, by the computer system, a similarity of each of the reference images to the sample image. (Item 11) said comparing said sample image to each reference image of said plurality of reference images; generating, by the computer system, feature vectors for each of the reference images of the plurality of reference images and for the sample image; and calculating, by the computer system, a distance metric for each of the plurality of reference images from the sample image based on the generated feature vector for each image. (Item 12) 12. The method of claim 11, wherein the features used to generate the feature vector for each of the reference images of the plurality of reference images and the sample image include at least one of patch size, particle size, particle shape, average pixel intensity, and image mean grayscale value. (Item 13) aligning the first reference image with the sample image; determining, by the computer system, a first display location for displaying the sample image in the user interface, the first display location being within a first cell of a grid of cells; determining, by the computer system, a second display location adjacent to the first display location for displaying the first reference image in the user interface, the second display location being within a second cell of the grid of cells; 13. The method according to any one of items 9 to 12, further comprising: aligning, by the computer system, a centroid of the first reference image in the second display position with a centroid of the sample image in the first display position. (Item 14) dynamically filling the background of the aligned first reference image; expanding, by the computer system, the background of the first reference image in the second cell to fill the second cell; determining, by the computer system, a background color of the first reference image; 14. The method according to any one of items 9 to 13, further comprising filling the enlarged background of the first reference image with the background color by the computer system. (Item 15) Determining the background color of the first reference cell determining, by the computer system, four corner pixels of the first reference image; and calculating, by the computer system, an average color value of the four corner pixels. (Item 16) 17. The computer-implemented method of claim 9, wherein the plurality of reference images are selected based on a tag associated with each reference image of the plurality of reference images. A computer readable memory device comprising program code that, when executed by a processor, causes the processor to: receiving a sample image of a sample particle from a biological sample; selecting a plurality of reference images from a database of reference images, each reference image of the plurality of reference images being a reference image of a reference particle from a plurality of reference particles from one or more biological samples; determining an order for the plurality of reference images based on a similarity of each reference image to the sample image; selecting a first reference image from the plurality of reference images based on the determined order; aligning the first reference image with the sample image; dynamically filling a background of the aligned first reference image; and displaying the sample image and the dynamically filled and aligned first reference image in a user interface. The program code for determining the order of the plurality of reference images, when executed by the processor, causes the processor to: comparing the sample image to each of the plurality of reference images; and determining a similarity of each of the reference images to the sample image. (Item 19) The program code for comparing the sample image with each of the plurality of reference images, when executed by the processor, causes the processor to: generating feature vectors for each of the reference images of the plurality of reference images and for the sample image; Item 19. The computer readable memory device of item 18, comprising program code for causing the computer to: calculate a distance metric for each reference image of the plurality of reference images with the sample image. (Item 20) 20. The computer-readable memory device of claim 19, wherein features used to generate the feature vector for each of the reference images of the plurality of reference images and the sample image include at least one of patch size, particle size, particle shape, average pixel intensity, and image mean grayscale value. (Item 21) The program code for aligning the first reference image with the sample image, when executed by the processor, causes the processor to: determine a first display position for displaying the sample image in the user interface, the first display position being within a first cell of a grid of cells; determining a second display position adjacent to the first display position for displaying the first reference image in the user interface, the second display position being within a second cell of the grid of cells; 21. The computer-readable memory device of any one of items 17 to 20, further comprising program code for causing the device to align a centroid of the first reference image in the second display position with a centroid of the sample image in the first display position. (Item 22) The program code for dynamically filling the background of the aligned first reference image, when executed by the processor, causes the processor to: Stretching the background of the first reference image in the second cell to fill the second cell; and determining a background color of the first reference image; 22. The computer-readable memory device of any one of items 17 to 21, comprising program code for causing the computer to perform the steps of: filling the enlarged background of the first reference image with the background color. (Item 23) The program code for determining the background color of the first reference image, when executed by the processor, causes the processor to: determining four corner pixels of the first reference image; 23. The computer readable memory device of claim 22, comprising program code for causing the computer to: (Item 24) 24. The computer-readable memory device of any one of items 17 to 23, wherein the plurality of reference images are selected based on a tag associated with each reference image of the plurality of reference images. [Brief description of the drawings]

[0007] A further understanding of the nature and advantages of various examples may be realized by reference to the following drawings.

[0008] [Figure 1] FIG. 1 is a block diagram of an exemplary system that can be used to perform image analysis, normalization, and display, according to one embodiment.

[0009] [Diagram 2] FIG. 1 is a block diagram of an exemplary computer system that can be used to perform image analysis, normalization, and display, according to one embodiment.

[0010] [Diagram 3] 1 is a flowchart of an example process for image analysis, normalization, and display, according to one embodiment.

[0011] [Figure 4] 4 is a flow chart of an example of a method for ordering images based on similarity according to one embodiment.

[0012] [Diagram 5] 4 is a flow chart of an example of a method for determining similarity between images according to one embodiment.

[0013] [Figure 6] 1 is a flowchart of an example of a method for aligning images according to one embodiment.

[0014] [Figure 7] 1 is a flow chart of an example of a method for dynamically filling the background of an image according to one embodiment.

[0015] [Figure 8] 4 is a flow chart of an example of a method for determining a background color of an image according to one embodiment.

[0016] [Figure 9] 1 illustrates an example of an image displayed according to an embodiment. [Figure 10] 1 illustrates an example of an image displayed according to an embodiment. [Figure 11] 1 illustrates an example of an image displayed according to an embodiment. [Figure 12] 1 illustrates an example of an image displayed according to an embodiment. [Figure 13] 1 illustrates an example of an image displayed according to an embodiment. [Figure 14] 1 illustrates an example of an image displayed according to an embodiment.

[0017] [Figure 15] 1 illustrates an example of a user interface according to an embodiment. [Figure 16] 1 illustrates an example of a user interface according to an embodiment. [Figure 17] 1 illustrates an example of a user interface according to an embodiment. [Figure 18] 1 illustrates an example of a user interface according to an embodiment. [Figure 19] 1 illustrates an example of a user interface according to an embodiment. [Figure 20] 1 illustrates an example of a user interface according to an embodiment. [Figure 21] 1 illustrates an example of a user interface according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] 1 illustrates a block diagram of an exemplary system 100 that can be used to perform image analysis, normalization, and display. System 100 can include a network 105 over which various components can communicate, including database 120, database 125, server 110, computing device 115, and analysis equipment 130.

[0019] The analyzer 130 can collect images of living tissue particles and cells through a body fluid system that captures images of body fluid cells, for example, as described in detail in patent application Ser. No. 14 / 775,448, filed Sep. 11, 2015, entitled "HEMATOLOGY SYSTEMS AND METHODS." The analyzer 130 can collect images and store the images as sample cell images in the database 120. Reference images can be collected through the analyzer 130 and / or through other capture methods and stored in the database 125 for comparison.

[0020] Database 120 can store sample images for analysis. Database 125 can store an image atlas (i.e., a database of reference images) for comparison during analysis of the sample images stored in database 120. If desired, database 120 and database 125 can be a single database (e.g., with the sample images stored in different tables other than the reference images). Databases 120 and 125 can be any suitable database including, for example, a Microsoft® SQL Server® database, an Oracle® database, or a Microsoft® Excel® spreadsheet.

[0021] Network 105 may be any suitable number or type of networks or links, including, but not limited to, a dial-up network, a local area network ("LAN"), a wide area network ("WAN"), a public switched telephone network ("PSTN"), a cellular network, a WiFi network, the Internet, an intranet, or any combination of wired and / or wireless communication links. Network 105 may include one or more sub-networks. In some examples, two or more components of the system (e.g., computing device 115, server 110, databases 120 and 125, analysis equipment 130, or any combination thereof) may be connected to network 105 and communicate via network 105.

[0022] Computing device 115 may be computer system 200 as described with respect to Figure 2. Computing device 115 may be a system capable of performing image analysis, normalization, and display as discussed in more detail herein. Computing device 115 may be a system utilized by a user, and although depicted as a desktop computer, may be any user device including a laptop or wireless device (e.g., a smartphone, tablet, personal digital assistant (PDA), e-reader, or smartwatch).

[0023] 2. Server 110 may perform, for example, image analysis and normalization, and may send the resulting images to computing device 115 for display in a user interface, since the server may not, at times, include its own display device. Optionally, server 110 may be any suitable type of server. The server may be, for example, in a server farm or rack servers. Computing device 115 and server 110 may be, for example, in a cloud computing environment.

[0024] 2 illustrates a block diagram of an exemplary computer system 200 that can be used to perform image analysis, normalization, and display. Computing device 200 may include, for example, a laptop computer, a desktop computer, a tablet, an e-reader, a smartphone or mobile device, a smart watch, a personal digital assistant (PDA), or other electronic device. Computing device 200 may be, for example, computing device 115 or server 110.

[0025] Computing device 200 may include a processor 240 that interfaces with other hardware via a bus 205. Memory 210, which may include any suitable tangible (and non-transitory) computer-readable medium, such as RAM, ROM, EEPROM, etc., may embody program components (e.g., instructions 215) that configure the operation of computing device 200. In some examples, computing device 200 may include input / output ("I / O") interface components 225 (e.g., for interface with a display 245, keyboard, or mouse), and additional storage mechanism 230.

[0026] The computing device 200 may include a network component 220. The network component 220 may represent one or more of any components that facilitate a network connection. In some examples, the network component 220 may facilitate a wireless connection and may include a radio interface such as IEEE 802.11, Bluetooth, or a radio interface for accessing a cellular network (e.g., a transceiver / antenna for accessing CDMA, GSM, UMTS, or other mobile communication network). In another example, the network component 220 may be wired and may include an interface such as Ethernet, USB, or IEEE 1394.

[0027] 2 shows a single computing device 200 with a single processor 240, a system may include any number of computing devices 200 and any number of processors 240. For example, multiple computing devices 200 or multiple processors 240 may be distributed over a wired or wireless network (e.g., a wide area network, a local area network, or the Internet), such as network 105. Multiple computing devices 200 or multiple processors 240 may perform any of the steps of the disclosure individually or in coordination with one another. For example, processor 240 and / or computing device 200 may be, for example, computing device 115 and / or server 110 of FIG. 1.

[0028] 3 illustrates a flow chart of one embodiment of a process 300 for image analysis, normalization, and display, which may be implemented, for example, as program code on the system 100 of FIG. 1, or more specifically, on the computer system 200 of FIG.

[0029] The process 300 begins at 305, for example on the computer system 200 of FIG. 2. At 310, the computer system 200 can receive a sample image of a sample particle (e.g., a cell) from a living tissue. As briefly discussed above, cells and particles of living tissue can be imaged or photographed using an image processing system (e.g., the analyzer 130) using a biological sample (e.g., a urine or blood sample) collected from a microorganism. The image can be zoomed in before or after capture to represent an image of a single cell or particle, such as the sample image 905 of FIG. 9. The sample image can be received at a computing device 115 (shown in more detail as the computer system 200 of FIG. 2) through the network 105 of FIG. 1, for example from the database 120 of FIG. 1.

[0030] At 315, the computer system 200 may select a number of reference images from a database of reference images. The database of reference images may be, for example, database 125 of FIG. 1. The database may include any number of reference images. The reference images may be populated through similar capture methods as described above with respect to the sample images. The reference images may be, for example, reference images 910 of FIG. 9. The reference images may be populated through a user-selected library of reference images stored in the database 125. Optionally, the user-selected images may be from a library of previously captured reference images. Optionally, the user-selected images may include images that were sample images that the user adds to the reference image database. Optionally, the user-selected images may use file names based on randomly generated globally unique identifiers (GUIDs) to ensure uniqueness and portability across systems. The reference images may be stored in any suitable format (e.g., * .png, * .gif, * The reference images can be stored as they are captured, meaning that the images can vary, for example, in file size, pixel height, pixel width, and / or magnification of the imaged cells.

[0031] The reference image may be a cell or particle containing a known characteristic, or a known cell or particle of a particular type. For example, a whole blood sample typically contains three major classes of blood cells, including red blood cells (erythrocytes), white blood cells (leukocytes), and platelets (thrombocytes). Each class is divided into subclasses of constituents. It can be further divided. For example, subclassifications of the five major types of white blood cells (WBCs) have different shapes and functions. White blood cells can include neutrophils, lymphocytes, monocytes, eosinophils, and basophils. There are also subclassifications of red blood cell types. The appearance of particles in a sample can vary depending on disease states, cell maturity, and other factors. Subclassifications of red blood cells can include reticulocytes and nucleated red blood cells. As another example, the imaged cells can be sickle cells. The reference images can include annotations that can confirm the category of cells in the reference images, or tags that identify features or characteristics of cells in the reference images. As an example, annotations on an image of a cell known to be a sickle cell can identify the cell as a sickle cell. As another example, annotations on an image of a cell known to be a red blood cell can identify the cell as a red blood cell. Some reference images can include multiple tags that identify multiple features or characteristics of a cell. For example, a cell can be annotated as a red blood cell and can have an additional tag that identifies the cell as a reticulocyte. The cells or particles can be tagged as, for example, neutrophils, lymphocytes, monocytes, eosinophils, basophils, platelets, reticulocytes, nucleated red blood cells (RBCs), blasts, promyelocytes, myelocytes, metamyelocytes, red blood cells (RBCs), cells, bacteria, particulate matter, cell aggregates, or cell debris or components. Optionally, the computer system 200 can select multiple reference images from a database of reference images, for example, by selecting all reference images with unique tags and / or annotations. As an example, the computer system 200 can select all reference images from the database that are annotated as red blood cells. As another example, the computer system 200 can select all reference images from the database that have annotations of white blood cells. Optionally, the computer system 200 can allow a user to select tags or annotations to be used to select the reference images. Optionally, the computer system 200 can select all reference images in the reference image database.Optionally, the computer system 200 can allow a user to select a desired reference image from a reference image database. Optionally, the computer system 200 can allow a user to select additional tags as a filter, for example, to select all images that are classified as white blood cells and further tagged as neutrophils.

[0032] Tagging or annotating images can provide an effective method for fast image retrieval. For a large number of reference images with potentially many different subtypes, it can be beneficial to provide subsets that target one or more subtypes based on variations in either physical characteristics or disease stages for the same classification type. This can be done through hierarchical tagging, such as tagging within annotations that identify the classification of the imaged cells. For example, a reference image can include an annotation that the cells are red blood cells ("RBC") (i.e., an RBC annotation that the imaged cells are within the RBC classification), and tags that mark the reference image as having specific characteristics and annotations of RBCs, e.g., a nucleated red blood cell can have a tag that identifies it as nucleated.

[0033] At 320, the computer system 200 can determine an order for the multiple reference images based on the similarity of each reference image to the sample image. This can save a lot of time in ordering the reference images so that the most similar ones are displayed first. As discussed above, the sample images can be displayed together with the reference images in the user interface to compare and identify the properties, characteristics, or types of cells imaged in the sample images. Displaying the most similar images first can limit the amount of time it takes to find a reference image that matches the sample image to classify the sample image to the same type of cells in the reference image or to identify that the sample image has the same properties or characteristics as the reference image. The computer system 200 can determine the order by, for example, comparing the sample image to each of the selected reference images, as described in more detail with respect to FIG. 4 and FIG. 5.

[0034] At 325, computer system 200 may select a first reference image from the plurality of reference images based on the determined order. After computer system 200 determines the order of the reference images, such that the first reference image is the one determined to be most similar, the second reference image is the one determined to be second most similar, etc., computer system 200 may select the first reference image for analysis, normalization, and display. After performing the remaining steps of process 300 on the first reference image, computer system 200 may select a second reference image for analysis, normalization, and display in the remaining steps of process 300, and continue to repeat the steps for each reference image of the plurality of reference images until all selected reference images have been analyzed, normalized, and displayed according to process 300.

[0035] At 330, the computer system 200 may align the first reference image with the sample image. The alignment of the first reference image with the sample image may be for displaying the images to a user through a user interface, for example. The alignment may include aligning adjacent edges of the images and centering the images relative to one another for display in the user interface. One example of one type of alignment that may be performed is described in more detail with respect to FIG. 6. Optionally, the adjacent edges of the sample image and the reference image are not the same size. In such a case, the smaller image may be enlarged to match the size of the larger image. For example, the smaller image may be zoomed to enlarge to the same size as the larger image, if desired. Optionally, the image may be overlaid on an image patch having adjacent edges the same size as the larger image. Optionally, each image may be positioned for display within a cell of a grid, which may have a matching background color that appears on the display to resize the size of the smaller image to the size of the larger image.

[0036] Optionally, the number of reference images for display in the user interface can range from 1 to 8. Alignment and expansion can be performed for any number of reference images, and the reference images can be aligned and expanded in sequence based on images that share adjacent edges. For further details regarding optional image display layouts, see the discussion of FIG. 11. In mathematical terms, I i,j is W i,j Width and H i,j Let the reference image be the i-th row and j-th column in an m × n grid with i ∈ [0, m-1], j ∈ [0, n-1], with height RH = 1. The alignment and magnification are calculated by the following equation: i and the jth column width CW j This can be done by calculating:

number

[0037] Depending on the relative size of the sample image and the list of reference images to display, the final row height or column width of a particular row / column in the display grid may require enlargement of either the sample image or the reference image. If desired, an image background fill can be applied to any images that have a size different than the calculated grid cell width or height. Also, the image itself can be centered in the associated grid cell without scaling or resizing to maintain 100% fidelity to the original image.

[0038] At 335, the computer system 200 can dynamically fill the background of the aligned first reference image. During the alignment process, the background of the first reference image can be expanded to ensure that the displayed image has the same size adjacent edges to the sample image. The expanded background can be dynamically filled with substantially the same color as the existing background of the reference image to ensure uniformity and to remove confusion such as gaps between images with different background colors. The colors of any image can include, for example, red, green, blue, and alpha channels. As described in more detail with respect to FIG. 7, the background can be filled by determining a color that matches the background of the reference image and filling the expanded background with the determined color. If desired, the background is filled by overlaying the reference image over a filled patch of the determined background color. During display, the patch can be, for example, one cell of a grid of cells. At this point, it is noted that two different types of cells are discussed herein. A cell image (i.e., a reference image or a sample image) is an image of a somatic cell of living tissue. A grid cell is a different type of cell, such as a cell in a table. Throughout this specification, when referring to a cell, an imaged somatic cell is intended, unless a grid cell is specifically mentioned.

[0039] If desired, the alignment of the sample image and the reference image may be resized (e.g., because the sample image is smaller than the reference image) to result in an enlarged sample image. In such a case, the background of the sample image may be dynamically filled in the same manner as described above with respect to dynamically filling the background of the first reference image.

[0040] At 340, the computer system 200 can display the sample image and the dynamically filled and aligned first reference image in a user interface. Examples of user interface displays of the sample image and one or more reference images are shown in FIGS. 9-18. Optionally, the user interface can be displayed on a display device associated with the computer system 200. For example, the computing device 115 can be any user device having a monitor capable of displaying a user interface, including a desktop, laptop, smartphone, tablet, or any other suitable computing device 115 including a display monitor, can be the computer system 200. Optionally, the computer system 200 can be a server 110. In such an embodiment, the server 110 may not include a specific display monitor, and thus the server 110 can send information for display (e.g., the reference image and the sample image, and associated calculated layout and color information) to the computing device 115 for display to the user in the user interface.

[0041] Process 300 may end at 345. As discussed above, if desired, steps 325, 330, and 335 may be performed with respect to a first reference image and repeated for each selected reference image until each of the selected reference images have been aligned and dynamically filled. If desired, once each reference image has been aligned and dynamically filled, a display step may be performed for each individual reference image, either for all selected reference images simultaneously (only once after all selected reference images have been aligned and dynamically filled), or for a set of a certain number of reference images simultaneously (e.g., displaying eight reference images simultaneously with the sample image, as in FIG. 12).

[0042] Figure 4 illustrates a flow chart of one embodiment of a process 400 for determining an order of images based on similarity. Process 400 may be performed as part of step 320 of Figure 3. Process 400 may be implemented, for example, as program code on system 100 of Figure 1, or more specifically, on computer system 200 of Figure 2.

[0043] At 405, the computer system 200 may compare the sample image to each of a plurality of reference images by, for example, generating feature vectors for each of the reference and sample images and calculating a distance metric between each of the reference and sample images, as described in more detail with respect to FIG.

[0044] At 410, the computer system 200 may determine the similarity of each reference image to the sample image. This determination may be made, for example, based on the calculated distance metric discussed with respect to FIG.

[0045] Figure 5 illustrates a flow chart of one embodiment of a process 500 for determining similarity between images. Process 500 may be performed as part of step 405 of Figure 4. Process 500 may be implemented, for example, as program code on system 100 of Figure 1, or more specifically, on computer system 200 of Figure 2.

[0046] At 505, the computer system 200 may generate feature vectors for each of the reference images of the plurality of reference images and the sample image. The feature vectors may include information about a plurality of features of each image. For example, the feature vectors may identify patch size, particle size, particle shape, average pixel intensity, and / or image average grayscale value. The patch size may be the overall image size. The image size may be zoomed, as discussed above. The background may also be magnified, as discussed above. The patch size is the size of the image prior to any of these modifications. The particle size may be the size of the sample or reference cells in the image. If desired, the cell size may appear larger or smaller in the image, for example, as shown in FIG. 9. The particle size of the sample cells in the sample image 905 is smaller than the particle size of the reference cells in the reference image 910. The particle size may appear larger or smaller in the image, for example, based on the magnification used during image capture. The particle shape may be the shape of the sample or reference cells in the image. The average pixel intensity can be calculated by selecting a number of pixels from the image, determining the intensity of each of the selected pixels, and calculating the average intensity of the selected pixels. The average grayscale value of the image can be calculated by selecting a number of pixels from the image, determining a grayscale value for each pixel, and calculating the average of the calculated grayscale values. Other features can be identified and utilized for each image's feature vector.

[0047] At 510, the computer system 200 can calculate a distance metric for each reference image of the plurality of reference images from the sample image. For example, the computer system 200 can use the feature vector to calculate a distance metric between each reference image and the sample image. The lower the distance metric value, the more similar the reference image is to the sample image. The distance metric value can then be used to rank the reference images in order of most similar to the sample image.

[0048] If desired, the distance metric can be defined as follows:

[0049]

number

[0050] In this definition of the distance metric, the i-th reference image feature vector

number

[0051] Figure 6 illustrates a flow chart of one embodiment of a process 600 for aligning images. Process 600 may be performed as part of step 330 of Figure 3. Process 600 may be implemented, for example, as program code on system 100 of Figure 1, or more specifically, on computer system 200 of Figure 2.

[0052] At 605, the computer system 200 can determine a first display location for displaying the sample image in the user interface within a first cell of the grid. For example, the image can be displayed in the user interface within the grid such that the sample image is in the center and the reference images are adjacent to the sample image. For example, FIG. 9 shows a sample image 905 adjacent to a reference image 910. Similarly, as another example, FIG. 11 shows a sample image 1105 adjacent to both reference images 1110 and 1115. As yet another example, FIG. 12 shows a sample image 1205 adjacent to reference images 1210, 1215, 1220, and 1225. FIG. 12 also shows a sample image that shares a corner with reference images 1230, 1235, 1240, and 1245, each of which is adjacent to another reference image. If desired, as shown in FIG. 12, the sample image is placed in the center of the grid so that it can be adjacent to as many reference images as possible.

[0053] At 610, the computer system 200 can determine a second display location adjacent to the first display location within a second cell of the grid for displaying the first reference image in the user interface. Examples of sample images adjacent to reference images can be found in Figures 9-14. For example, sample image 905 can be adjacent to reference image 910 as shown in Figure 9.

[0054] At 615, the computer system 200 can align the centroid of the first reference image in the second display position with the centroid of the sample image in the first display position. For example, the center of the sample image can be determined and positioned so that it is located at the center of the first display position, which can be the center of a first cell of the grid (e.g., the location of the sample image 905 in FIG. 9). The computer system 200 can also determine and position the center of the first reference image so that it is located at the center of the second display position, which can be the center of a second cell of the grid (e.g., the location of the sample image 910 in FIG. 9). Due to the definition of the grid, the center of the first display position and the center of the second display position that are adjacent in the grid have a common center of adjacent edges (as shown in FIG. 9). If necessary, other methods can be used to align the centers of the sample image and the first reference image by selecting a display position for each result in which the centroids of each image are aligned.

[0055] Figure 7 illustrates a flow chart of one embodiment of a process 700 for dynamically filling the background of an image. Process 700 may be performed as part of step 335 of Figure 3. Process 700 may be implemented, for example, as program code on system 100 of Figure 1, or more specifically, on computer system 200 of Figure 2.

[0056] At 705, the computer system 200 can expand the background of the first reference image in the second cell to fill the second cell. If desired, the first reference image can have a lower height than the sample image, for example. For example, if the adjacent sides of the sample image and the first reference image are vertical sides, the adjacent sides of the first reference image and the sample image are different. Similarly, for example, if the adjacent sides of the sample image and the first reference image are horizontal sides and the width of the first reference image is narrower than the width of the sample image, the adjacent sides of the images are different. The background of the first reference image can be expanded to ensure that the adjacent sides of the first reference image and the sample image are the same. If desired, the size (height and / or width) of the sample image can be smaller than the size of the first reference image. In such an embodiment, the background of the sample image can be expanded.

[0057] At 710, the computer system 200 can determine a background color of the first reference image. The background color of the first reference image can be determined in a number of ways. For example, the background color can be determined using a process for determining an average background color as described in detail in FIG. 8. If desired, a standard color can be selected to be used as the background color of the reference image. For example, if all reference images have the same background color, the background color will always be the same and can be known.

[0058] At 715, the computer system 200 can fill the enlarged background of the first reference image with the background color. For example, the second cells of the grid can be filled with the determined background color, so that when displayed in the user interface, the first reference image appears enlarged without modifying the first reference image itself. If desired, the first reference image can be processed to increase the image size and fill the enlarged pixels with the determined background color.

[0059] Figure 8 illustrates a flow chart of one embodiment of a process 800 for determining a background color of an image. Process 800 may be performed as part of step 710 of Figure 7. Process 800 may be implemented, for example, as program code on system 100 of Figure 1, or more specifically, on computer system 200 of Figure 2.

[0060] At 805, the computer system 200 can determine the four corner pixels of the first reference image. A typical image has four corners, and a corner pixel can be used at each corner. For example, if the image patch is of size W×H pixels, the four corner pixels are [0,0], [W-1,0], [0,H-1], [W-1,H-1]. More or fewer pixels can be used as needed. Any number of pixels can be selected from the edge pixels and used as needed. Optionally, the edge pixels can be any pixel along the edge of the image. Optionally, the edge can be a pixel within a threshold distance to the absolute edge of the image, for example, a distance of 3 pixels from the absolute edge of the image.

[0061] At 810, the computer system 200 may calculate an average color value of the four corner pixels. The average color value may be determined by first determining the color value of each of the four corner pixels (or each pixel identified for determining the background color, e.g., if other edge pixels are used). Once the color value of each of the four corner pixels has been determined, the computer system 200 may use the color values ​​from each of the four corner pixels to calculate an average color value of the four corner pixels. The average may be calculated for each color component, e.g., red, green, blue, and alpha. If desired, any selected pixels may be used to calculate the average color value of the background. The average color value may then be used as the background color of the image. Determining the average color value may help reduce clutter by filling the background of the reference image with a color that substantially matches the entire background of the image. An example of a dynamically filled grid of aligned sample images and eight reference images is shown in Figures 12-14.

[0062] 9-14 illustrate examples of images displayed according to embodiments. Figure 9 displays a portion of a user interface 900 that can be used to compare a sample image 905 of a sample particle 930 to a reference image 910 of a reference particle 935. The user interface 900 can include a classification indicator 915, a forward navigation button 925, and a back navigation button 920.

[0063] The classification indicator 915 may be an indication of an annotation associated with the reference image 910. As shown in FIG. 9, the reference image 910 may have the annotation "HYAL" associated with it. HYAL may mean, for example, that the reference image 910 was classified as a hyaline cast. The classification annotation or feature tag may be selected by the user using, for example, a drop-down selection list, as shown in FIG. 9. The options included for selection may include, for example, the list provided in Table 1. The list provided in Table 1 includes different cell types or cell features that may be identified. Once a selection is made by the user of the classification indicator 915, a selection of the reference image may be made, for example, as described in detail with respect to step 315 of FIG. 3. [Table 1]

[0064] As shown in Figure 9, there are no gaps between images 905 and 910 shown in user interface 900 to remove confusion associated with having different colors between the images. The sample image 905 can be acquired as described with respect to receiving a sample image in step 310 of Figure 3. The reference image 910 can be selected as described with respect to selecting a reference image in step 315 of Figure 3.

[0065] The forward navigation button 925, when selected by a user, can cause the reference image 910 to be replaced with another reference image in the user interface display. Optionally, the forward navigation button 925 can select the next reference image in an ordered list, determined by similarity, as described with respect to step 325 of Figure 3. Optionally, the forward navigation button 925 can select the next reference image, selected by the computer system in random order, for example, from a group of reference images that include annotations or tags as indicated by the classification indicator 915.

[0066] The back navigation button 920, when selected by a user, can cause the reference image 910 to be replaced with another reference image in the user interface display. Optionally, the back navigation button 920 can select a previous reference image in an ordered list, determined by similarity, as described with respect to step 325 of Figure 3. Optionally, the back navigation button 920 can select a previously viewed reference image, whether from an ordered list or not.

[0067] FIG. 10 shows a user interface 1000 that is very similar to a portion of the user interface 900 of FIG. 9. As shown in FIG. 10, the sample image 1005 is the same image as the sample image 905 of FIG. 9. The sample image 1005 is an image of a sample particle 1030, and the reference image 910 is an image of a reference particle 1035. The user interface 1000 also includes a classification indicator 1015, a forward navigation button 1025, and a back navigation button 1020. The user interface 1000 may be the result from the classification indicator 915 of FIG. 9 with a new selection identified by the user. For example, if a user utilizes the user interface 900 and selects using the classification indicator 915 from HYAL, shown in the classification indicator 915 of FIG. 9, to WBC, shown by the classification indicator 1015 of FIG. 10. WBC may mean white blood cells. The resulting user interface 1000 may then show the reference image 1010 with the annotation of WBC.

[0068] The classification indicator 1015 may be implemented in the same manner as described above with respect to the classification indicator 915 of Figure 9. The forward navigation button 1025 may be implemented in the same manner as described above with respect to the forward navigation button 925 of Figure 9. The back navigation button 1020 may be implemented in the same manner as described above with respect to the back navigation button 920 of Figure 9.

[0069] As shown in Figures 9 and 10, a portion of the user interface 900 is larger than a portion of the user interface 1000. By looking at Figures 9 and 10, it can be seen that the sample image 905 has the same width as the sample image 1005, but the height of the sample image 905 is larger than the height of the sample image 1005. Both the sample image 905 and the sample image 1005 are the same sample cell and are also the same sample image. However, the sample image 905 and the sample image 1005 are different sizes (heights). This may be because the reference image 910 is larger than the reference image 1010. In such a case, the height of the sample image 905 can be expanded and dynamically filled as described with respect to steps 330 and 335 of Figure 3 to ensure that the adjacent edges of the sample image 905 and the reference image 910 are the same size.

[0070] 11 illustrates a portion of a user interface 1100 that is similar to user interface 900 and user interface 1000. User interface 1100 includes a sample image 1105 of a sample particle 1135, reference images 1110 and 1130 of reference particles 1140 and 1145, respectively, a forward navigation button 1125, a back navigation button 1120, and a classification indicator 1115. The classification indicator 1115 may be implemented in the same manner as described above with respect to classification indicator 915 of FIG. 9. The forward navigation button 1125 may be implemented in the same manner as described above with respect to forward navigation button 925 of FIG. 9. The back navigation button 1120 may be implemented in the same manner as described above with respect to back navigation button 920 of FIG. 9.

[0071] The sample image 1105 can be acquired as described with respect to receiving a sample image in step 310 of FIG. 3. The reference images 1110 and 1130 can be selected as described with respect to selecting a reference image in step 315 of FIG. 3. For illustrative purposes only, the reference images 1110 and 1130 of the reference particles 1140 and 1145 are the same image and of the test particle. If desired, for example, if the reference database has few reference images, the same reference image can be displayed to the user to enhance the user's ability to identify the similarity between the sample image and the reference image. If desired, the same reference image can be displayed multiple times in the same or different positions of the grid display after selection of the forward navigation button 1120 or the back navigation button 1125. Multiple displays can be due to, for example, the number of reference particles in the reference database, a desire to fill all cells of the display grid.

[0072] 11, it can be seen that two reference images 1110 and 1130 are displayed, respectively, on different adjacent edges of the sample image 1105. The selection of the positions of the reference image 1110 and the reference image 1130 can be based, for example, on the similarity of the images: the more similar image can be displayed first, where the reference image 1110 is located, the second most similar image can be displayed second, where the reference image 1130 is located.

[0073] For image layout modes that involve presenting two or more reference images and that are mostly one dimensional, such as a "3x1" layout (as depicted in FIG. 19), the reference images can be ordered according to the image similarity metric in the direction of the X-axis. The following figure illustrates the points.

number

[0074] Similarly, for a "3x3" image layout mode that contains two dimensions, the reference images can be ordered based on the spiral coordinates defined by the image neighborhood. The following diagram shows the ordering of the reference images ([0], [1], ··· ,[7]) coincides with a natural image neighborhood ordering.

number

[0075] When the number of reference images is not divisible by the sliding window size (the sliding window size for an m×n image layout is m×n−1, e.g., an image layout mode of “3×3” has a sliding window size of 3×3−1=8), the selection is adjusted so that there are no empty grid cells when displaying the reference images. For example, regardless of the navigation direction (e.g., moving forward or backward in the list of reference images using the forward or back navigation buttons as described with respect to FIG. 9), there may not be any selection difference. For example, if there are 9 reference images in the “3×3” image layout mode, i.e., the sliding window size is 8, there are two sliding windows that can be displayed, the first window uses the first 8 reference images and the last window uses the last 8 reference images. Specifically, for a list of N reference images, for an image layout of m×n, the count of the sliding windows is given by:

[0076]

number

[0077] where m ≠ 1 or n ≠ 1, mod denotes the mathematical modulo operation, and the i-th sliding window start index (into the list of N reference images) is given by:

[0078]

number

[0079] where m ≠ 1 or n ≠ 1, i ∈ [0, K-1]

[0080] Figure 12 illustrates a portion of a user interface 1200 that is similar to the user interfaces of Figures 9-11. User interface 1200 displays eight reference images 1210, 1215, 1220, 1225, 1230, 1235, 1240, and 1245. Figure 12 also includes forward navigation buttons, back navigation buttons, and a sorting indicator.

[0081] Figure 13 illustrates a portion of a user interface 1300 similar to the user interfaces of Figures 9-12. User interface 1300 displays eight reference images (not numbered). Figure 13 also includes forward and back navigation buttons and a sorting indicator. User interface 1300 is one example of a user interface that a user can use to compare a sample image in a box in the center of the image to the eight reference images, where the reference images surround the sample image.

[0082] Figure 14 illustrates a portion of a user interface 1400 similar to the user interfaces of Figures 9-13. User interface 1400 displays eight reference images (not numbered). Figure 14 also includes forward and back navigation buttons and a sorting indicator. User interface 1400 is an example of a user interface that a user can use to compare a sample image in the central box of Figure 14 to the eight reference images, where the reference images surround the sample image.

[0083] 15-18 illustrate additional examples of user interfaces. FIG. 15 illustrates a user interface 1500 including a floating window 1505, which may be part of a user interface 1900 as shown in FIG. 19, for example. The floating window 1505 may include a context menu 1510, which may be displayed when a user right-clicks within the floating window 1505. The sample image displayed within the floating window 1505 (behind the context menu 1510) may be selected, for example, by the user selecting one of the images in an image selection section 1515. Selection of options available within the context menu may include, for example, changing the display layout, changing the reference image source, selecting one or more tags, changing the orientation of one or more images, and zooming in or out on one or more images.

[0084] Fig. 16 illustrates a user interface 1600 including a floating window 1605, which may be part of a user interface 2000 as shown in Fig. 20, for example. The floating window 1605 may include a context menu 1610, which may be displayed when a user right-clicks within the floating window 1605. The sample image 1620 displayed in the floating window 1605 may be selected, for example, by the user selecting an image from an image selection section 1615. In this case, the sample image 1620 is selected by the user selecting an image 1625 from the image selection section 1615.

[0085] Fig. 17 illustrates a user interface 1700 including a floating window 1705, which may be part of a user interface 2100 as shown in Fig. 21, for example. The floating window 1705 may include a classification identifier 1710, which may allow a user to select a tag or annotation for selecting a reference image, as described with respect to step 325 of Fig. 3. The sample image 1720 displayed in the floating window 1705 may be selected, for example, by the user selecting an image from the image selection section 1715. In this case, the sample image 1720 is selected by the user selecting an image 1725 from the image selection section 1715.

[0086] Figure 18 illustrates a user interface 1800 that includes a floating window 1805, which may be part of a user interface 2100, such as that shown in Figure 21. The floating window 1805 may include a context menu 1810 that may be displayed when a user right-clicks within the floating window 1805.

[0087] FIG. 19 illustrates a portion of a user interface 1900, which may be a portion of a user interface represented as a floating window 1505 as shown in FIG. 15. A context menu 1905 may be displayed when a user right-clicks on a portion of the user interface 1900. The context menu 1905 may change the display layout of the sample image and the reference image. The "3x1" selection is grayed out because the display is already set to a 3x1 grid. The 3x1 option is shown, for example, in FIG. 11. Alternative option selections may include "2x1", for example, as shown in FIGS. 9 and 10. Alternative option selections may include "3x3", for example, as shown in FIGS. 12-14.

[0088] FIG. 20 illustrates a portion of a user interface 2000, which may be a portion of a user interface represented as a floating window 1605 as shown in FIG. 16. A context menu 2005 may be displayed when a user right-clicks on a portion of the user interface 2000. The context menu 2005 allows the user to "select a tag", which may be a selection of tags to identify the reference image for selection in the reference image database, and to further refine the displayed images not only to the selected classification, but also by the selected tag. In this way, the user can customize multiple selections of classifications and tags for filtering. In FIG. 20, it can be seen that the user can select a classification of the reference image, if desired, from those indicated by the classification selection box 2010. In this case, the selected classification is "HYST". In addition, tags can be selected to further refine the selection based on other characteristics of the particles tagged to the reference image in the reference database. The context menu 2005 allows the user to select one or more tags to filter the displayed reference images. The selected tags are indicated at 2015. The reference image to be displayed may be selected from a reference image database based on whether the reference image falls within a selected classification and whether a selected tag is associated with the reference image.

[0089] FIG. 21 illustrates a portion of a user interface 2100, which may be a portion of a user interface represented as a floating window 1805 as shown in FIG. 18. A context menu 2105 may be displayed when a user right-clicks on the portion of the user interface 2100. The context menu 2105 allows a user to change the image source. The selection of the image source may select a database from which a reference image is selected. Options for the selection of the image source may be, for example, to use a built-in source or, for example, to use a user-selected reference image database. The user-selected reference image database may be generated, for example, as described above.

[0090] Each of the calculations or operations described herein may be performed using a computer or other processor having hardware, software, and / or firmware. Various method steps may be performed by modules, which may comprise any of a wide variety of digital and / or analog data processing hardware and / or software arranged to perform the method steps described herein. Modules, optionally comprising data processing hardware, are suitable for performing one or more of these steps by having appropriate machine programming code associated therewith, where modules for two or more steps (or portions of two or more steps) may be integrated on a single processor board or separated on different processor boards in any of a wide variety of integrated and / or distributed processing architectures. These methods and systems will often make use of tangible media that incorporate computer readable code with instructions for performing the method steps described above. Suitable tangible media may include memory (including volatile and / or non-volatile memory), storage media (magnetic storage such as floppy disks, hard disks, tapes, etc., optical memory such as CDs, CD-R / W, CD-ROMs, DVDs, etc., or any other digital or analog storage medium), etc.

[0091] Different arrangements of the components depicted in the figures or described above are possible, as well as components and steps not shown or described. Similarly, some features and subcombinations are advantageous and may be used without reference to other features and subcombinations. The embodiments of the invention are described for purposes of illustration and not limitation, and alternative embodiments will become apparent to the reader of the present invention. In certain cases, method steps or operations may be performed or carried out in a different order, or operations may be added, deleted, or modified. It may be understood that in certain aspects of the invention, a single component may be replaced with multiple components, and multiple components may be replaced with a single component, to provide an element or structure or to perform a function. Except where such a substitution is ineffective in the practice of a particular embodiment of the invention, such substitution is deemed to be within the scope of the invention.

[0092] It is understood that the drawings and descriptions of the embodiments of the present invention have been simplified to illustrate elements relevant to a clear understanding of the present invention. However, one of ordinary skill in the art will recognize that these and other elements may be desirable. However, because such elements are well known in the art and do not facilitate a better understanding of the present invention, descriptions of such elements are not provided herein. It is understood that the drawings are shown for illustrative purposes and not as construction drawings. The omitted details and modifications or alternative embodiments are within the expertise of one of ordinary skill in the art.

[0093] It will be understood that in certain aspects of the invention, a single component may be replaced with multiple components, and multiple components may be replaced with a single component, to provide a certain element or structure or to perform a certain function. Except to the extent that such substitution is not effective in the practice of a particular embodiment of the invention, such substitution is deemed to be within the scope of the invention.

[0094] The examples shown herein are intended to illustrate potential and specific embodiments of the present invention. Those skilled in the art may appreciate that the examples are primarily intended to illustrate the present invention. There may be changes to these diagrams or operations described herein without departing from the spirit of the present invention. For example, in certain cases, method steps or operations may be performed or executed in a different order, or operations may be added, deleted, or modified.

[0095] Moreover, while specific embodiments of the invention are described herein for purposes of illustrating the invention and not for purposes of limiting the invention, it will be understood by those skilled in the art that many changes may be made in the details, materials, and arrangements of elements, steps, structures, and / or components within the principles and scope of the invention without departing from the invention as set forth in the claims.

[0096] All patents, patent publications, patent applications, journal articles, books, technical literature, and the like, discussed in this disclosure are hereby incorporated by reference in their entirety for all purposes.

Claims

[Claim 1] A system 100 comprising a network 105, a database 120 of reference images, a server 110, a computing device 115 for performing image analysis, and an analysis apparatus 130 for collecting sample images and reference images, wherein the computing device 115 selects a number of reference images from the database 120 and displays them within a number of cells of a grid of a user interface for comparison with the sample images.

Images