Systems and methods for assessing tissue remodeling - Patents.com

Abstract

A method for assessing tissue remodeling risk is provided, comprising: (a) acquiring a plurality of images of a target area of ​​a subject's skin, the plurality of images including (1) a set of visible light images and (2) a set of fluorescent images, (b) processing the plurality of images to determine an optimal image pair for the target area, the optimal image pair including (i) a first visible light image selected from (1) and (ii) a first fluorescent image selected from (2), (c) generating an assessment of the target area based at least on the optimal image pair, and (d) displaying on a graphical user interface a schematic diagram illustrating the assessment, the optimal image pair, and a location of the target area on the subject's body, the assessment being indicative of a probability of tissue remodeling in the target area.

Description

[Technical field]

[0001] cross reference

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 132,979, filed December 31, 2020, which is incorporated by reference in its entirety herein. [Background technology]

[0002] Cancer is a leading cause of death worldwide. Skin cancer is the most common cancer, and melanoma is the most aggressive form of skin cancer. Current techniques for diagnosing melanoma include observing changes in the visual appearance (size, shape, color) of existing moles or the appearance of new moles, determining medical history risk factors (age, family history, etc.), biopsy, and histological analysis. The decision to biopsy a mole depends largely on the surface appearance of the mole. However, changes in the visual appearance of a mole do not always conclude the presence of melanoma within the mole, as subsurface features are often missed using standard techniques. For example, one such feature is the presence of neovascularization and tissue remodeling. As a tumor grows, it sends out signals to prepare the local environment for expanded tumor growth (e.g., tissue remodeling) and increased nutritional supply (e.g., angiogenesis). Such signals may result from growth factors, cytokines, and other proteins released from tumor cells and / or from the tumor microenvironment during expansion. These processes may be investigated by examining the tissues surrounding the tumor (e.g., the macroenvironment) and may be used in combination with standard diagnostic techniques to inform diagnostic and / or therapeutic procedures. However, most methods for such investigations require biopsy and histological analysis, which is invasive and time-consuming, making them less than ideal. Summary of the Invention [Problem to be solved by the invention]

[0003]

[0003] It is therefore desirable to provide a system and method for improved visual pre-biopsy assessment of tissue remodeling risk in a fast and easy to use format to allow a user to visually compare and assess the probability of tissue remodeling in clinically suspicious lesions. Not necessarily all such aspects and advantages are achieved by any particular embodiment. Thus, various embodiments may be implemented in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other aspects or advantages as may also be taught or suggested herein. [Means for solving the problem]

[0004]

[0004] The present disclosure relates generally to medical devices and methods, and more particularly to methods and apparatus for assessing tissue remodeling in the skin.

[0005] In a first aspect, a method for assessing tissue remodeling risk is provided. The method includes: (a) acquiring a plurality of images of a target area of ​​a subject's skin, the plurality of images including (1) a set of visible light images and (2) a set of fluorescent images; (b) processing the plurality of images to determine an optimal image pair for the target area, the optimal image pair including (i) a first visible light image selected from (1) and (ii) a first fluorescent image selected from (2); (c) generating an assessment of the target area based at least on the optimal image pair; and (d) displaying on a graphic user interface a schematic diagram showing the assessment, the optimal image pair, and the location of the target area on the subject's body, the assessment being indicative of the probability of tissue remodeling in the target area.

[0005]

[0006] In some embodiments, the target area may include a mole on the skin of the subject. In some embodiments, the first visible light image and the first fluorescent image may be selected to complement or match each other. Alternatively, or in combination, the first visible light image may include a first set of visual features and the first fluorescent image may include a second set of visual features. At least a portion of the second set of visual features may not be visible in the first set of visual features and at least a portion of the first set of visual features may not be visible in the second set of visual features. In some embodiments, the first set of visual features may be generated by using visible light or white light to image the target area. In some embodiments, the second set of visual features may be generated by using infrared light to image the target area. In some embodiments, the first set of visual features may be associated with the mole and the exposed skin of the subject. In some embodiments, the second set of visual features may be associated with the underlying tissue beneath the skin and around the mole in the macro environment (i.e., tissue surrounding but not including the mole).

[0006]

[0007] In some embodiments, an optimal image pair may be determined from a plurality of images based at least on one or more of the following: focus quality, contrast, sharpness, brightness, color, or resolution.

[0007]

[0008] In some embodiments, an optimal image pair may be determined from multiple images based at least on (1) the location of the target area in each of the visible light images and each of the fluorescent images, (2) the degree of similarity or correlation between each of the visible light images and each of the fluorescent images, and / or (3) the focus quality of each of the visible light images and each of the fluorescent images.

[0008]

[0009] In some embodiments, the step of processing the multiple images may include at least one of the following: size filtering, normalization, standardization, noise reduction, removal of imaging artifacts, background subtraction, cropping, enlarging, resizing, repositioning, brightness adjustment, contrast adjustment, or object segmentation.

[0009]

[0010] In some embodiments, the optimal image pair may be the first optimal image pair. In some embodiments, the method may include, after (b) and before (c), displaying options to the user on a graphic user interface, the options allowing the user to accept or reject the first optimal image pair, and receiving an input from the user in response to the options displayed on the graphic user interface. In some embodiments, the method may further include processing the plurality of images to determine a second optimal image pair for the target region when the input indicates that the user rejects the first optimal image pair, the second optimal image pair being different from the first optimal image pair. In some embodiments, the second optimal image pair and the first optimal image pair may not share any common image. Alternatively, the second optimal image pair and the first optimal image pair may share a common image. In some embodiments, the common image may include either the first visible light image or the first fluorescent image. In some embodiments, the method may include redisplaying options to the user on a graphic user interface, the options allowing the user to accept or reject the second optimal image pair, and receiving another input from the user in response to the options displayed on the graphic user interface. In some embodiments, (c) may include generating an assessment for the target area based at least on the second optimal image pair instead of the first optimal image pair when the another input indicates that the user accepts the second optimal image pair. Alternatively, or in combination, the method may include continuing with (c) and (d) when the input indicates that the user accepts the first optimal image pair. In some embodiments, the method may include processing the plurality of images to determine a plurality of other optimal image pairs for the target area when the input indicates that the user rejects the first optimal image pair, and displaying the plurality of other optimal image pairs on the graphic user interface.Multiple other optimal image pairs may be sequentially displayed to the user on the graphic user interface.

[0010]

[0011] In some embodiments, (b) may further include processing the plurality of images to determine a plurality of image pairs, the plurality of image pairs may include an optimal image pair. In some embodiments, the optimal image pair may be annotated to visually distinguish it from other image pairs. In some embodiments, the optimal image pair may be annotated by placing a predefined boundary around the optimal image pair.

[0011]

[0012] In some embodiments, (b) may further include automatically detecting moles within the target area and generating a graphic border or outline around the mole. In some embodiments, the graphic border or outline may be automatically generated to follow the shape or contour of the mole. Alternatively, or in combination, the graphic border or outline may be adjustable or created by a user via a graphical user interface.

[0012]

[0013] In some embodiments, the evaluation may include an aggregate score for the optimal image pair. The aggregate score may be a composite of two or more separate scores. In some embodiments, the two or more separate scores may include (1) a score for the first visible light image, and (2) a score for the first fluorescent image. In some embodiments, the composite of the two or more separate scores may be equally weighted, such that the score for the first visible light image and the score for the first fluorescent image are both weighted the same. In some embodiments, the composite of the two or more separate scores may be weighted differently, such that the score for the first visible light image and the score for the first fluorescent image are weighted differently. For example, the score for the first visible light image may be weighted higher than the score for the first fluorescent image. Alternatively, the score for the first visible light image may be weighted lower than the score for the first fluorescent image. In some embodiments, the aggregate score may be within a range of values. In some embodiments, the two or more separate scores may be within a range of values. In some embodiments, the score for the first visible light image and the score for the first fluorescent image may be based on a standardized range of values.

[0013]

[0014] In some embodiments, the probability of tissue remodeling may be associated with the probability of melanoma development in the mole. In some embodiments, the assessment may be usable to determine a clinical diagnosis or course of action based on the probability of melanoma development in the mole. In some embodiments, the method may further include generating one or more notifications on a graphical user interface when the probability of tissue remodeling or the probability of melanoma development in the target area is greater than one or more thresholds.

[0014]

[0015] In some embodiments, the assessment may be generated based at least on (1) a first set of indices associated with the first set of visual features in the first visible light image, and (2) a second set of indices associated with the second set of visual features in the first fluorescent image. In some embodiments, the first set of indices may include one or more of the following: size, shape, volume, color, or surface texture of the mole and the area around it. In some embodiments, the second set of indices may include one or more of the following: size, shape, area or degree of tissue remodeling, pixel intensity, fluorescent intensity, pattern, or texture of the target area beneath the subject's skin and around the mole. In some embodiments, the method may further include segmenting the first visible light image or the first fluorescent image to generate the first set of indices or the second set of indices, respectively. Segmenting the first fluorescent image may include segmenting the first fluorescent image into a plurality of zones around the target area. In some embodiments, the method may further include dividing the plurality of zones into a plurality of sub-areas.

[0015]

[0016] In some embodiments, the first visible light image and the first fluorescent image may be adjusted and aligned to a set of coordinates in the optimal image pair. In some embodiments, the first visible light image and the first fluorescent image may be adjusted and aligned by rotating, translating, cropping, enlarging, and / or shrinking at least one of the first visible light image and the first fluorescent image. In some embodiments, the first visible light image and the first fluorescent image may be adjusted and aligned using one or more fiducial markers present in both the first visible light image and the first fluorescent image. The one or more fiducial markers may be annotated on the first visible light image and the first fluorescent image. Alternatively, or in combination, the one or more fiducial markers may be visible in both the first visible light image and the first fluorescent image. Alternatively, or in combination, the one or more fiducial markers may include one or more alphanumeric characters. Alternatively, or in combination, the one or more fiducial markers may include a line, an edge, a point, or a two-dimensional shape. In some embodiments, one or more fiducial markers may be provided at one or more predefined locations within the target area. Alternatively, or in combination, one or more fiducial markers may be provided at a predefined distance and / or orientation relative to the location of the mole.

[0016]

[0017] In some embodiments, the method may further include comparing the optimal image pair to one or more reference images. In some embodiments, the one or more reference images may include a reference image pair. In some embodiments, the reference image pair may include (1) a visible light reference image not from the set of visible light images, and (2) a fluorescent reference image not from the set of fluorescent images. In some embodiments, the optimal image pair may include fiducials, and the reference image pair may not include fiducials. In some embodiments, the optimal image pair may include a first set of fiducials located at a predefined location and / or orientation relative to the mole, and the reference image pair may include a set of fiducials located near a border of each reference image in the reference image pair.

[0017]

[0018] In some embodiments, the graphical user interface may be configured to allow a user to toggle or switch between the first visible light image or the first fluorescent image in the optimal image pair.

[0018]

[0019] In some embodiments, the first visible light image may be focused on a mole on the subject's skin, and the first fluorescent image may be focused on an area around the mole beneath the subject's skin, optionally, the area may be at least 1.5 times larger than the size of the mole.

[0019]

[0020] In some embodiments, obtaining multiple images of the target area may include autofocusing the integrated imaging system using structured light, and capturing multiple images with the integrated imaging system after autofocusing. Incorporation by Reference

[0021] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference.

[0020]

[0022] The novel features of the present disclosure are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the disclosure are utilized, and the accompanying drawings. [Brief description of the drawings]

[0021] [Figure 1] 1 is a schematic diagram of an exemplary system for acquiring at least a pair of images, according to an embodiment. [Diagram 2] 1 is a flowchart of a method for acquiring at least a pair of images, according to an embodiment. [Diagram 3]1 is a flowchart of a method for operating a user interface to assess risk of tissue remodeling for a pair of images, according to an embodiment. [Figure 4] FIG. 1 illustrates an exemplary user interface depicting a login screen, according to an embodiment. [Diagram 5] FIG. 2 illustrates an exemplary user interface depicting an initial menu, according to an embodiment. [Figure 6] FIG. 13 illustrates an exemplary user interface depicting a new patient entry screen, according to an embodiment. [Figure 7] 13 illustrates an exemplary user interface depicting a new patient card before image import, according to an embodiment. [Figure 8] FIG. 11 illustrates an exemplary user interface depicting an image import screen, according to an embodiment. [Figure 9] FIG. 13 illustrates an exemplary user interface depicting a patient card with side-by-side optimal image pairs and tissue scores, according to an embodiment. [Figure 10] 1 illustrates an exemplary user interface depicting a patient card with a visible light image displayed, according to an embodiment. [Figure 11] 1A-1C illustrate an exemplary user interface depicting a patient card with an anatomical image displayed, according to an embodiment. [Figure 12] FIG. 13 illustrates an exemplary user interface depicting a patient history screen, according to an embodiment. [Figure 13] FIG. 11 illustrates an exemplary user interface depicting a report generation screen, according to an embodiment. [Figure 14] FIG. 13 illustrates an exemplary user interface depicting a pathology report entry screen, according to an embodiment. [Figure 15] FIG. 11 illustrates an exemplary user interface depicting a backup screen, according to an embodiment. [Figure 16]FIG. 13 illustrates an exemplary user interface depicting a restore from backup screen, according to an embodiment. [Figure 17] FIG. 11 illustrates an exemplary user interface depicting a backup in progress screen, according to an embodiment. [Figure 18] 1 is a flowchart of a method for determining an optimal image pair, according to an embodiment. [Figure 19] 1 is a flowchart of a method for segmenting and scoring optimal image pairs, according to an embodiment. [Figure 20] FIG. 1 illustrates an image of a mole and surrounding skin divided into multiple segments, according to an embodiment. [Figure 21] FIG. 1 illustrates an image of a mole and surrounding skin divided into multiple segments and subregions, according to an embodiment. [Figure 22] FIG. 1 illustrates a computer control system that is programmed or otherwise configured to carry out the methods provided herein, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022]

[0045] In the following detailed description, reference is made to the accompanying drawings, which form a part of this specification. In the drawings, like numerals typically identify like components, unless otherwise indicated by context. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein. The aspects of the present disclosure as generally described herein and illustrated in the figures may be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0023]

[0046] Although specific embodiments and examples are disclosed below, the subject matter of the present invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as modifications and equivalents thereof. Thus, the claims appended hereto are not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Although various operations may be described in sequence as multiple separate operations in a manner that may be helpful in understanding a particular embodiment, the order of description should not be construed as implying that these operations are order dependent. Additionally, the structures, systems, and / or devices described herein may be embodied as integrated components or as separate components.

[0024]

[0047] For purposes of comparing various embodiments, certain aspects and advantages of those embodiments are described. Not necessarily all such aspects and advantages are achieved by any particular embodiment. Thus, for example, various embodiments may be practiced in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other aspects or advantages as may also be taught or suggested herein.

[0025]

[0048] The present disclosure is described with respect to the deployment of a system, device, or method for assessing tissue remodeling in the skin of a patient. However, those skilled in the art will understand that this is not intended to be limiting, and that the devices, systems, and methods disclosed herein can be used in other anatomical regions and / or to assess other aspects of tissue. For example, tissue cavities or surfaces such as the mouth, colon, digestive system, cervix, bladder, lungs, lymph nodes, etc. The devices, systems, and methods disclosed herein can be used to image surface and / or non-surface tissues as desired by those skilled in the art.

[0049] 1 and 2 show exemplary systems, devices, and methods for acquiring a pair of images from a target region. However, it should be apparent to one of ordinary skill in the art that this is not intended to be limiting, and that the image analysis systems and methods disclosed herein may be used in conjunction with any suitable pair of images acquired from a target region, regardless of what device, system, or method was used to acquire the images. The selection of the image acquisition device may be determined based on the accessibility of the tissue location, the facilities and / or technology available at the treatment site, operational costs, etc.

[0026]

[0050] FIG. 1 is a schematic diagram of an exemplary system 100 for acquiring at least a pair of images from a target area 1. The system 100 may include an excitation unit and a luminescence collection unit operatively coupled to each other by a processor. The system 100 may include a processor configured to operate the components of the luminescence collection unit and the excitation unit. The excitation collection unit may include a fluorescent light source and a visible light source. The visible light source may be a white light source, e.g., a white LED. The fluorescent light source may include, e.g., an infrared LED. One or more excitation filters may be coupled to the light source to refine the excitation light that is directed toward the target area 1. Light may be emitted by the light source in response to input from a user (e.g., selecting a light mode and pressing a capture button) and directed toward the target area 1. Light may be reflected, refracted, and / or emitted back from the target area 1 toward the luminescence collection unit. The luminescence collection unit may include a camera or other imaging device (e.g., PMT, CCD, etc.). One or more emission filters may be disposed in the optical path between the target area 1 and the camera to filter the incoming emission light. For example, a slide may house a fluorescent emission filter and a visible light emission filter. The fluorescent emission filter may be specific to the wavelength or range of wavelengths emitted by the target area 1 when excited by the fluorescent light source. In some embodiments, the visible light emission filter may be a polarizing filter, and a second polarizing filter (e.g., a visible light excitation filter) rotated 90 degrees relative to the visible light emission filter may be disposed in the optical path between the visible light source and the target area 1. The use of two polarizing filters may remove blemishes (i.e., reflections from oils on the skin) from the target area 1. The system 100 may be used to collect at least a first pair of images of the target area 1, including a first visible light image and a first fluorescent image.

[0027]

[0051] Note that the elements shown in FIG. 1 are not to scale, and that the arrangement of elements as shown is merely exemplary. The number of light sources may vary. Light directing elements such as mirrors, prisms, light pipes, optical fibers, and / or splitters may be used to direct the light. Not all elements are required in all embodiments.

[0028]

[0052] System 100 may include one or more elements of the systems described in U.S. Patent Application Publication No. 2017 / 0049380 or U.S. Patent Application Publication No. 2019 / 0307391, the disclosures of which are incorporated herein in their entireties for all purposes.

[0029]

[0053] The system 100 may be an integrated imaging system including an integrated or non-integrated camera having the following components: a case, a power supply, a lens, an image sensor, an image storage memory, user controls, a user display, internal control electronics including stored instructions for an embedded processor, and internal image processing logic including stored instructions for an embedded processor. A consumer or professional digital single-lens reflex (DSLR) camera is one non-limiting example of an integrated imaging system. The integrated imaging system may have interchangeable lenses, although this is not a requirement. The integrated imaging system may have autofocus capabilities, such as a mirrorless contrast detection autofocus method, or a phase difference detection method using a mirror and a separate sensor. The lens may have macro focus capabilities. The integrated imaging system may have a removable image storage module (e.g., SD card) and / or a cable for communicating stored images and / or a wireless communication port for communicating stored images. The integrated imaging system does not require connection to an external computer for operation, although such connection may be optional. An integrated imaging system differs from an industrial, medical, or combination imaging system in which the required components and / or functionality are divided between two or more physical enclosures, one of which is or contains a computer.

[0030]

[0054] One or more filters may be provided in the system 100. The system 100 may, for example, have two emission filters in a slide configured to move each filter into the optical path of the camera. Any number of filters (e.g., 1, 2, 3, 4, 5, or more) may be provided. The filters may allow different wavelengths of electromagnetic radiation to pass relative to each other. The filters may be movable relative to the optical path of the camera and / or each other. The filters may move perpendicular to the optical path of the camera. The desired filter may be slid, swiveled, or rotated into place.

[0031]

[0055] In some embodiments, a single filter may be used rather than two. In this single filter configuration, the filter may have a band-stop notch at an excitation frequency such as 660 nm while passing both visible and emission band light. In this manner, such a single filter may be used for both visible and fluorescent emission exposures without changing filters.

[0032]

[0056] One or more (e.g., two) visible excitation light sources may be provided to achieve uniform illumination of the mole or other target area. One or more (e.g., two) fluorescent excitation light sources may be provided to achieve uniform illumination of the mole or other target area. Uniform illumination may be advantageous in achieving a calibrated or measurable response based on biotags and / or standards for this purpose. The fluorescent light source may be an LED, laser, fluorescent emitter, or other light source. In some embodiments, the fluorescent light source may have a sufficiently narrow bandwidth so that a fluorescent excitation filter is not required.

[0033]

[0057] A structured light illumination component, such as a diffractive element or mask, may be provided, which may be integrated with a visible light source to achieve uniform white light illumination of the object. The structured light illumination component may be an optical element that may pattern, diffuse, or spread light. For example, the structured light illumination component may comprise a diffractive element integrated with a laser light source, or a mask configured to generate a line pattern on the target area when illuminated by LED light. Structured illumination may be used, for example, to identify hairs. Structured illumination may also be used to determine the height and shape of moles above the surface of the skin, as well as the nature of the moles.

[0034]

[0058] The system 100 may have an autofocus configured to focus the camera on the target area under both visible and fluorescent illumination to ensure that the visible and fluorescent images are taken at the same focus. In some embodiments, the autofocus may be configured to focus the camera on the target area using structured light. In at least some cases, the use of structured light for autofocus may enhance the camera autofocus under fluorescent illumination as compared to unstructured light, which may have higher fluorescent light scattering in tissue and may be inappropriate for autofocus in at least some cases.

[0035]

[0059] System 100 may include a cavity for a memory card (not shown), which may include a wireless interface (not shown), a user display (not shown), and user controls (not shown). The user display may include a screen or other display that may show images that may be captured by the integrated imaging device. A lens may be provided or attached to system 100. The lens may either be integral to the integrated imaging device or the device may be adapted to accept interchangeable lenses (such as a macro lens). Operational buttons may also be integrated within system 100. Other user interface mechanisms, such as a touch screen, levers, sliders, knobs, or features, may be used for a user to interface or interact with the integrated imaging device.

[0036]

[0060] FIG. 2 shows a flow chart of a method 200 for acquiring at least a pair of images.

[0061] In step 201, a biotag may be positioned in a target area of ​​a patient. The biotag may be detectably (e.g., fluorescently) labeled. The biotag may selectively bind to a targeted binding partner present in the target area of ​​interest. Alternatively, the biotag may be absorbed, metabolized, internalized, or otherwise retained in the reactive tissue of the target area. Application may be topical (e.g., with a gel, liquid, etc.), such as application to the surface of the skin using a skin penetrating agent or enhancer, or may be applied by subcutaneous or intradermal injection (e.g., with an array of microneedles or by electrical conductivity). The biotag formulation may include a solvent and, optionally, an inhibitor, a skin penetrating agent and / or enhancer, an ion pairing agent, a co-solvent, and / or a moisturizing agent and / or a thickening agent, alone or in various combinations.

[0037]

[0062] Biotags are specific binding partners for targeted molecules of interest. Examples of biotags include, without limitation, peptides, peptidomimetics, peptoids, circular peptides, etc.; nucleic acids such as RNA, DNA, aptamers, etc.; or other organic compounds. One or a mixture of two, three, four, or more different parts of biotags can be used in the methods described herein for multiplex imaging. Biotags can be of sufficiently small molecular weight to effectively penetrate the epidermal surface, for example, typically less than 10,000 daltons, less than 5,000 daltons, less than 2,500 daltons, less than 1,000 daltons, and the penetration can be facilitated by a penetrating agent. Biotags generally include a detectable label.

[0038]

[0063] Molecules suitable as binding partners for biotags can include, for example, cancer-associated markers present on cancer or precancerous cells or in the macroenvironment of cancer or precancerous cells, e.g., in the vasculature at the site of a lesion. Particular markers of interest for this purpose include molecules associated with tumor vasculature, such as integrins, including, without limitation, integrin av, integrin a5, integrin 03, integrin 31, and the like. Biotags suitable for detecting such integrins can include peptides containing an RGD motif or mimetics thereof, as known and used in the art. See, for example, Gaertner et al. (2012) Eur J Nucl Med Mol Imaging.39 Suppl 1:S1 26-38; Danhier et al. (2012) Mo.Pharm.9(11):2961-73, specifically incorporated herein by reference. Other biotags of interest can include, without limitation, hormones, antigen-binding fragments of antibodies, EGF, IGF, and the like. Although tumor-associated biotags are described in detail herein, it will be understood by those skilled in the art that the systems and methods described herein can be used to evaluate non-tumor markers as well, particularly when the tissue to be evaluated is not cancerous.

[0039]

[0064] Tumor associated antigens include, without limitation, immunogenic sequences from MART-1, gp100 (pmel-17), tyrosinase, tyrosinase-related protein 1, tyrosinase-related protein 2, melanocyte stimulating hormone receptor, MAGE1, MAGE2, MAGE3, MAGE12, BAGE, GAGE, NY-ESO-1, β-catenin, MUM-1, CDK4, caspase 8, KIA 0205, HLA-A2R1701, α-fetoprotein, telomerase catalytic protein, G-250, MUC-1, oncofetal protein, p53, Her2 / neu, TERT, PRAME, LINC00518, triosephosphate isomerase, CDC-27, LDLR-FUT, telomerase reverse transcriptase, MUC18, ICAM-1, TNF, as known in the art. alpha / beta, plasminogen activator (uPA), cathepsins (B, D, H, L), PSMA, HMB-45, S-100, Melan-A (A103), (T311), Mitf (D5), glypican-3, GPC3, GPNMB, MIA (melanoma inhibitory activity), MCR-1, EGF, IGF, ARPC2, FN1, RGS1, SPP1, WNT2, PECAM-1, osteopontin, glucose, MMP-s (members of the matrix metalloproteinase family such as MMP-1, MMP-2, MMP-9, MMP-13, MT I-MMP, and others), FDG (or other metabolites), VEGF, and the like.

[0040]

[0065] Optically visible moieties for use as detectable markers may include fluorescent dyes, or visible spectrum dyes, visible particles, and other visible labeling moieties. Fluorescent dyes such as fluorescein, coumarin, rhodamine, Bodie Py Texas Red, and cyanine dyes may be useful when sufficient excitation energy can be provided to the site to be visually inspected. Endoscopic visualization procedures may be highly compatible with the use of such labels. Acceptable dyes may include non-toxic, FDA approved food dyes and pigments, although pharma- ceutically acceptable dyes approved for internal use are preferred. Alternatively, visible particles such as colloidal gold particles or latex particles may be attached to the biotag via a suitable chemical crosslinker.

[0041]

[0066] Fluorescent dyes of interest as detectable labels include, without limitation, fluorescein (e.g., fluorescein isothiocyanate, FITC), rhodamine, indocyanine green (ICG), Texas Red, phycoerythrin, allophycocyanin, 6-carboxyfluorescein (6-FAM), 2',7'-dimethoxy-4',5'-dichloro-6-carboxyfluorescein (JOE), 6-carboxy-X-rhodamine (ROX), 6-carboxy- 2',4',7',4,7-hexachlorofluorescein (HEX), 5-carboxyfluorescein (5-FAM) or N,N,N',N'-tetramethyl-6-carboxyrhodamine (TAMRA), cyanine dyes such as Cy3, Cy5, Cy5.5, Alexa542, Alexa647, Alexa680, Alexa700, Bodipy630 / 650, fluorescent particles, fluorescent semiconductor nanocrystals, derivatives thereof, and the like.

[0042]

[0067] In some embodiments, the wavelength for emission from the label may be in the near ultraviolet or near infrared range. Characteristics considered for label selection may include its light absorption and minimization of autofluorescence from the body surface to be measured. The probe may respond to fluorescent illumination of a particular wavelength and then emit light at a different wavelength.

[0043]

[0068] In some embodiments, the wavelength for emission from the label may be in the near infrared range. Such labels may include, without limitation, Alexa dyes, such as Alexa647, Alexa680, Alexa700, and cyanine dyes, such as Cy5, Cy5.5, and Cy7.

[0044]

[0069] Other dyes may include, without limitation, any of the dyes approved by the FDA for use in food, such as, for example, FD&C Blue No. 1 E133, FD&C Blue No. E132, FD&C Green No. 3, Orange B(3), FD&C Red No. 3 E127, FD&C Red No. 40(3) E129, FD&C Yellow No. 5 E102, FD&C Yellow No. 6, D&C Black No. 2&3, D&C Red Nos. 6, 7, 17, 21, 22, 27, 28, 30, 31, 33, 34, 36, 40, D&C Violet No. 2, and the like.

[0045]

[0070] In alternative embodiments, the biotags may be imaged by one or more imaging modalities, which may include, without limitation, optical coherence tomography, Raman spectroscopy, photoacoustic imaging, ultrasound imaging, endoscopy, and the like.

[0046]

[0071] In step 202, the biotags may be incubated on the target area. The biotags may interact with the tissue and bind to an appropriate binding partner, a process that typically takes several minutes.

[0047]

[0072] In step 203, excess unbound biotags may be removed. In some embodiments, removal may occur by rinsing or wiping with water or saline, with or without detergent. Depending on the application and embodiment, excess (unbound or unretained) biotags may be removed after 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 minutes, or within 20-25, 25-30, 30-35, 35-40, 40-45, 50-55, 55-60 minutes, or within 1 The biotag may be removed within 2-2, 2-3, 3-4, 4-5, 5-6, 6-7, 7-8, 8-9, 9-10, 10-11, 11-12, 12-13, 13-14, 14-15, 15-16, 16-17, 17-18, 18-19, 19-20, 20-21, 21-22, 22-23, 23-24 hours, or within 1-2 days. The preferred time of biotag application is between 2-15 minutes and less than 2 hours. In some embodiments, excess biotag may not be removed (e.g., when the biotag is injected). Retention of the biotag in the luminal / tissue compartment may occur when an appropriate binding partner is found in the lesion macroenvironment.

[0048]

[0073] In step 204, one or more fiducial markers may be placed adjacent to the target area. Prior to imaging the target area, a calibration marker in the form of a fiducial marker may be applied adjacent to a lesion in the target area. The fiducial marker may be removably provided on the patient, depicted on the patient, affixed (removably or permanently) to the imaging device, or provided separately from the imaging device. The image may be acquired using a camera or any of the devices, systems, and methods described herein. The fiducial marker may enable a processor and / or user to compare an image captured using a visible light source (also referred to herein as a visible light image) to an image captured using a fluorescent light source (also referred to herein as a fluorescent image). The one or more fiducial markers may be visible both in the visible light spectrum and in the fluorescent emission band of the biotag, for example, to facilitate image alignment. The markers do not need to appear identical in both images, but they must be clearly aligned. Because the camera may be handheld or because the patient may move between exposures, images taken with visible and fluorescent emission spectrum light may not naturally align, and therefore alignment reference features may be particularly beneficial.

[0049]

[0074] In some embodiments, the one or more fiducial markers may include a unique barcode or other identifier for identification of the target area to be imaged. (Barcode generally refers to information that is unique to a particular tag: e.g., one-dimensional barcode, 2D metric barcode.) The one or more fiducial markers may include a visual identifier. The fiducial marker may include a fluorescent marker or tag that includes either the same fluorescent compound as the fluorescent marker present on the biotag, or a compound that emits light within a compatible spectrum as the biotag (e.g., FD&C Green No. 3), so that it can be detected by the camera optics and used as a target for autofocus. In some embodiments, the fiducial may not be used for autofocus, but instead (or additionally) may be used to verify focus after the image is uploaded into the image processing system. The compatible spectrum may include, for example, a spectrum that includes an excitation light within the spectrum of the excitation of the biotag, and the light emission of the fiducial marker includes a spectral emission within the spectrum of the biotag emission. In some cases, common food coloring may be used as the fluorescent compound in the fiducial marker. The fiducial markers may be applied directly to the tissue / lumen surface or onto a medium that is later applied to the surface, e.g., a sticker, or may be transferred from a medium to the skin, such as a temporary or permanent tattoo. In some embodiments, multiple fiducial markers may be applied onto an exemplary sticker or tattoo.

[0050]

[0075] In step 205, an anatomical reference image may be captured to mark the location of the target area.

[0076] In step 206, the system may be set to a visible mode and a visible light image of the target area may be captured. The visible light image may include a first set of visual features. The first set of visual features may be generated by using visible or white light to image the target area. In some embodiments, structured or patterned light may be used to image the target area, for example, for 3D and / or roughness analysis.

[0051]

[0077] Depending on the application, images may be taken before as well as after application of the biotag.

[0078] In step 207, the system may be transitioned to a fluorescence mode and a fluorescence image of the target area may be captured. An image (e.g., a photograph) of the tissue surface may be taken with a camera and the correct (excitation) wavelength of light to activate the biotag detectable label (e.g., fluorescent label). The fluorescence image may capture the light emitted by the biotag. The fluorescence image may include a second set of visual features. At least a portion of the first set of visual features from the visible light image may not be visible in the second set of visual features from the fluorescence image (e.g., the color of the mole, etc.). The second set of visual features may be generated using fluorescent light (e.g., infrared light) to image the target area. In some embodiments, the biotag may emit a fluorescent signal when excited by fluorescence, which may indicate binding of the biotag to the target marker. In some cases, the presence of the target marker (and biotag fluorescence) may be associated with tissue remodeling and / or a particular disease state. In some embodiments, the absence of the target marker (and biotag fluorescence) may be associated with tissue remodeling and / or a particular disease state. In some embodiments, the biotags may bind to target markers present within the macro-environment (also referred to herein as the macro-region) adjacent to a lesion of interest (e.g., a mole, tumor, etc.) even when diseased cells are not specifically present within the region being imaged.

[0052]

[0079] The visible light image may be focused on the target area (e.g., a mole on the patient's skin), while the fluorescent image may be focused on a region around the target area (e.g., a macro region around the mole) just below the surface of the target area (e.g., just below the subject's skin). A first and second set of visual features may be associated with the mole and the subject's exposed skin. A second set of visual features may be associated with the underlying tissue just below the skin and around the mole. In some embodiments, the macro region may be at least 1.5 times larger than the size of the mole.

[0053]

[0080] The same camera can be used to capture the visible light and fluorescent images, or, less preferably, different cameras can be used, one to capture the visible light images and one to capture the fluorescent images.

[0054]

[0081] In step 208, steps 206 and 207 may be repeated to acquire multiple visible light images and multiple fluorescent images. The system may be configured to alternate between visible and fluorescent modes to capture alternating visible and fluorescent images. In some embodiments, at least two image pairs may be captured. In some embodiments, 3, 4, 5, 6, 7, 8, 9, 10, or more image pairs may be captured. Once a desired number of image pairs have been captured, the images may be transferred from the camera to a processor for further processing and analysis (e.g., as illustrated in FIG. 19).

[0055]

[0082] Although the above steps illustrate a method 200 for acquiring a pair of images according to an embodiment, one of ordinary skill in the art would recognize many variations based on the teachings described herein. Steps may be completed in different orders. Steps may be added or removed. Some of the steps may include sub-steps. Many of the steps may be repeated as many times as necessary to acquire a pair of images.

[0056]

[0083] For example, in some embodiments, step 204 may occur before step 201 so that fiducials are placed adjacent to the target area before the biotags are applied. Alternatively, or in combination, steps 206 and / or 207 may occur in multiple steps, some of which may be automated. In some embodiments, an optional focusing step may be added when autofocus is not used.

[0057]

[0084] FIG. 3 shows a flow chart of a method 300 for operating a user interface to assess risk of tissue remodeling for a pair of images.

[0085] In step 301, the user interface may display a HIPPA-compliant secure login screen (eg, as shown in FIG. 4).

[0058]

[0086] In step 302, new patient data can be entered into the system using a user interface (eg, as shown in FIG. 6).

[0087] In step 303, a unique patient identifier may be created.

[0088] In step 304, when a patient has already been created (e.g., in a previous session using steps 302-303), the user may enter the existing user's data into the system (e.g., using their unique patient identifier) ​​using the user interface to avoid steps 302-303 and load the previously captured data.

[0059]

[0089] In step 305, the user may optionally choose to return to a previously saved session and avoid steps 302-304 when the patient information is already on the system.

[0060]

[0090] In step 306, the multiple images may be imported into the system and displayed on a user interface (e.g., as shown in FIG. 8). The multiple images may include multiple anatomical images, multiple visible images of the target area, and multiple fluorescent images of the target area. The multiple images may be displayed to the user prior to further processing to allow the user to select images to be analyzed as described herein.

[0061]

[0091] In step 307, an optimal pair of visible and fluorescent images may be identified for each mole and a unique reference number may be assigned to each mole. The optimal pair of visible and fluorescent images may be identified automatically, as described herein, or may be selected by a user. In some embodiments, the optimal pair of visible and fluorescent images may be programmatically identified prior to display and reference number assignment.

[0062]

[0092] In step 308, the mole(s) may be mapped to a location on the patient's body using the anatomical image (e.g., as shown in FIGS. 9-11). In some embodiments, the user may map the mole to a location on the body by dragging a fiducial marker onto the displayed body outline. In other embodiments, the mole may be automatically mapped. In some embodiments, mole mapping may be avoided by the user and performed at a later time.

[0063]

[0093] In step 309, the image may be processed to identify the mole boundary line. In some embodiments, the boundary line may be determined automatically. In some embodiments, the user may adjust the boundary line using a user interface.

[0064]

[0094] At step 310, the images may be processed to calculate a mole analysis score, as further described herein. In some embodiments, a visibility score is calculated for the visible light image. In some embodiments, a fluorescence score is calculated for the fluorescence image. In some embodiments, an aggregate score is calculated from the visible light score and the fluorescence score. In some embodiments, the aggregate score may be the maximum of the visible light score and the fluorescence score.

[0065]

[0095] In step 311, the mole analysis score may optionally be refined based on a machine learning algorithm. The machine learning algorithm may be, for example, an unsupervised learning algorithm, a supervised learning algorithm, or a combination thereof.

[0066]

[0096] Unsupervised learning algorithms may include, for example, clustering, hierarchical clustering, k-means, mixture models, DBSCAN, the OPTICS algorithm, anomaly detection, local outlier factor methods, neural networks, autoencoders, deep belief nets, Hebbian learning, generative adversarial networks, self-organizing maps, expectation-maximization algorithms (EM), moment methods, blind signal separation techniques, principal component analysis, independent component analysis, nonnegative matrix factorization, singular value decomposition, or combinations thereof.

[0067]

[0097] The supervised learning algorithm may include, for example, a support vector machine, a linear regression, a logistic regression, a linear discriminant analysis, a decision tree, a k-nearest neighbor algorithm, a neural network, similarity learning, or a combination thereof. In some embodiments, the machine learning algorithm may include a deep neural network (DNN).

[0068]

[0098] The deep neural network may include a convolutional neural network (CNN). The CNN may be, for example, U-Net, ImageNet, LeNet-5, AlexNet, ZFNet, GoogleNet, VGGNet, ResNet18, or ResNet. Other neural networks may include, for example, a deep forward neural network, a recurrent neural network, a long short-term memory (LSTM), a gated recurrent unit (GRU), an autoencoder, a variational autoencoder, an adversarial autoencoder, a denoising autoencoder, a sparse autoencoder, a Boltzmann machine, a restricted neural network (RBM), a deep belief network, a generative adversarial network (GAN), a deep residual network, a capsule network, or an attention / transformer network.

[0069]

[0099] In some embodiments, the machine learning algorithm may be, for example, a random forest, a boosted decision tree, a classification tree, a regression tree, a bagged tree, a neural network, or a rotation forest. The machine learning algorithm may be applied to a number of features extracted from the image.

[0070]

[0100] In some embodiments, the neural network may include neural network layers. The neural network may have at least about 2 to 1000 or more neural network layers.

[0071]

[0101] In some embodiments, the neural network may be trained using associative learning techniques.

[0102] Machine learning can be employed to measure the similarity of patient image data to training examples of melanoma data and benign data. The degree of similarity of training examples to these two sets can be reported in many different formats, such as probability, feature set distance, weighted feature set distance, statistical measures of dominance, and / or the like, to name a few. Regardless of the similarity measure reported by the machine learning algorithm, the systems described herein can be configured to translate it into a common scoring format.

[0072]

[0103] At step 312, a mole analysis score report may be generated and displayed to the user (e.g., as shown in FIGS. 9-11). The mole analysis score report may include an assessment of the target area based at least on the optimal image pair. The assessment may include a mole analysis score(s). The assessment may be indicative of a probability of tissue remodeling in the target area. The probability of tissue remodeling may be associated with a probability of development of a tumor (e.g., melanoma if the target area includes a mole on the subject's skin). The assessment may be used to determine a clinical diagnosis or course of action based on the probability of development of a tumor (e.g., melanoma in a mole) in the target area.

[0073]

[0104] In step 313, the user may import the pathology report and update the mole analysis report to incorporate the histological information.

[0105] In step 314, the data may optionally be automatically updated to a cloud-based storage device (eg, as shown in FIG. 17).

[0074]

[0106] In step 315, some or all of the data may, optionally, be automatically restored from cloud-based storage (eg, as shown in FIG. 16).

[0075]

[0107] In step 316, the data may be synchronized with a cloud-based storage device (CLMS) using a unique patient identifier.

[0108] The above steps illustrate a method 300 of operating a user interface to assess risk of tissue remodeling for a pair of images, according to an embodiment, although one of ordinary skill in the art would recognize many variations based on the teachings described herein. Steps may be completed in different orders. Steps may be added or deleted. Some of the steps may include sub-steps. Many of the steps may be repeated as necessary to operate the user interface.

[0076]

[0109] For example, in some embodiments, step 307 may occur in multiple steps such that multiple image pairs are identified and the user has the option to select or reject image pairs to identify optimal image pairs. The multiple image pairs may be displayed to the user (e.g., sequentially displayed to the user) on a graphical user interface for selection or rejection in top-to-bottom order of best. Alternatively, or in combination, step 307 optionally occurs automatically (e.g., without user input).

[0077]

[0110] FIG. 4 shows an example user interface depicting a login screen. The security of the login screen may be determined by a system administrator. For example, when the user is a clinician, the login screen may be HIPPA compliant and require a strong password to ensure the security of patient data. Alternatively, when the user is a developer or other non-clinical user (using the portal for non-clinical data), the password requirements may be less secure to promote ease of use.

[0078]

[0111] FIG. 5 shows an example user interface depicting an initial menu. The initial menu may include multiple tabs for navigation. The tabs may include a Patient tab, a Cloud tab, a Settings tab, and an About tab. The initial menu may be configured to display the Patient tab as a default on start-up. The Patient tab may initially present the user with a screen with multiple buttons with options for how to begin a session. The user may choose to start a new patient session, load an existing patient into the new session, or return to an ongoing session. Selecting the New Patient Session button may load a New Session screen (e.g., as shown in FIG. 6). Selecting the Existing Patient / New Session button may load a Patient History screen (e.g., as shown in FIG. 12). Selecting the Back to Session button may open a screen of the current session as of the last time it was saved / logged off.

[0079]

[0112] FIG. 6 shows an exemplary user interface depicting a new patient entry screen. A user may select an option to start a new patient session on the initial menu screen shown in FIG. 5. A new patient entry screen may then be displayed within the patient tab. The user may enter information about the patient and the patient's medical history (e.g., first name, last name, patient ID, date of birth, gender, physician, personal or family medical history, etc.). The patient ID may be useful for synchronizing information with external software (e.g., clinic management software) and may be automatically generated or entered by the user to match the external software patient ID.

[0080]

[0113] 7 shows an exemplary user interface depicting a new patient card before image import. Selecting "Create" on the new patient entry screen of FIG. 6 creates a new patient card. The new patient card may include patient information, as well as areas for various images to be displayed and notes to be entered by the user. Those areas may not be populated until the user imports images from a storage medium (e.g., an SD card).

[0081]

[0114] FIG. 8 shows an exemplary user interface depicting an image import screen. Selecting "import images" on the new patient card screen of FIG. 7 allows a user to import patient images into the system. An optimal pair of visible and fluorescent images may be automatically identified for each mole, and a unique reference number may be assigned to each mole. The optimal pair of images may be displayed to the user on the image import screen. Other imported images and / or anatomical images may also be displayed to the user on the image import screen. The user may be given the option to select or reject the first optimal image pair. If the user selects the first optimal image pair, the system may receive an input indicating that the user accepts the first optimal image pair, and the first optimal image pair may be used to generate an assessment of the target area. If the user overrides the first optimal image pair, the system may receive an input indicating that the user rejects the first optimal image pair, and the imported images may be processed to determine a second optimal image pair that is different from the first optimal image pair. In some embodiments, the first and second optimal image pairs may not share any common images. Alternatively, the first and second optimal image pairs may share a common image (e.g., a common visible light image and / or a common fluorescent image). The second optimal image pair may be displayed and the user may be allowed to select or reject the image pair as previously described with respect to the first optimal image pair. If the user selects the second optimal image pair, the system may receive an input indicating that the user accepts the second optimal image pair, and the second optimal image pair may be used to generate an assessment of the target region instead of the first optimal image pair. If the user rejects the second optimal image pair, this process may be repeated until an optimal image pair is selected. The optimal image pairs may be displayed sequentially until the user selects the desired pair.

[0082]

[0115] In some embodiments, the image import screen may also display the assigned anatomical images, for example, by selecting the anatomical images tab to replace the clinical images tab shown in FIG. 8. The screen may be substantially similar to the screen of FIG. 8, except that the assigned anatomical images may replace the unselected imported images after an optimal image pair is selected by the user. For each set of clinical (fluorescence and visible) image pairs, there may be multiple anatomical images imported, which may assist the user in mapping the mole to one or more locations on the body. The anatomical images may be automatically assigned to the optimal image pair. The user may override the automatic selection as described herein. One or more anatomical images (preferably two) may be assigned to each optimal image pair. Once an anatomical image has been assigned to the first mole, the user may select the "Next Mole" button to import an image for the next mole, and so on. Once all of the moles have been assigned optimal image pairs and anatomical images, the user may select the "Import and Calculate" button to process and display to the user the images and / or tissue assessments as described herein.

[0083]

[0116] FIG. 9 shows an exemplary user interface depicting a patient card with optimal image pairs and tissue score(s) displayed side-by-side. Once the optimal image pairs and anatomical images for each mole are selected and confirmed by the user, the images may be processed and displayed to the user. In some embodiments, the moles may be automatically detected within the target area and a graphic border or outline may be generated and displayed around the mole. The graphic border or outline may be automatically generated to follow the shape or contour of the mole. In some embodiments, the graphic border or outline may be adjustable or created by the user via a graphic user interface. The location of the mole(s) on the subject's body may be shown on a schematic body diagram. The user may optionally toggle between different moles and corresponding image sets, each with a unique reference identifier. The patient card may display visible light and fluorescent images of the optimal image pairs side-by-side for comparative viewing. Visible, fluorescent, and aggregate scores may be automatically calculated after an image pair is selected for each mole and displayed to the user on the patient card. The indices used to calculate the score for the visible light images may be displayed to the user on the patient card and may include, for example, the traditional "ABCD" mole criteria, including asymmetry ("A"), border irregularity ("B"), color variegation ("C"), and diameter >6mm ("D"). The indices used to calculate the score for the fluorescent images may be displayed to the user on the patient card and may include, for example, I / T measurements, including fluorescence intensity ("I") and texture ("T"). From our initial training data, at least in some cases, specimens from melanoma may exhibit a different clustering of the ABCD indices than benign specimens. For example, melanoma specimens may exhibit a different clustering of the I / T indices than benign specimens. The visible light score may be calculated as the Euclidean distance from a surface that separates the melanoma and benign clusters of ABCD indices.The fluorescence score may be calculated as the Euclidean distance from the surface separating the melanoma and benign clusters of the I / T index. The aggregate score may be calculated as the maximum of the visibility score and the fluorescence score.

[0084]

[0117] The visible light image and the fluorescence image may be adjusted and aligned to a set of coordinates in the optimal image pair. For example, the visible light image and the fluorescence image may be adjusted and aligned by rotating, translating, cropping, enlarging, and / or shrinking at least one of the visible light image and the fluorescence image. Alternatively, or in combination, the images may be aligned and adjusted using one or more fiducial markers present in both the visible light image and the fluorescence image. In some embodiments, the one or more fiducial markers may be annotated on the visible light image and the fluorescence image. Alternatively, or in combination, the one or more fiducial markers may be visible in both the visible light image and the fluorescence image. The fiducial markers may include one or more alphanumeric characters, lines, edges, points, two-dimensional shapes, or the like, or any combination thereof. In some embodiments, the one or more fiducial markers may be provided at one or more predefined locations within the target area. In some embodiments, the one or more fiducial markers may be provided at a predefined distance and / or orientation relative to the location of the mole. The adjusted and aligned visible light image and the fluorescence image may be displayed to the user on a patient card screen for easy viewing.

[0085]

[0118] The probability of tissue remodeling may be correlated with the probability of tumor development (e.g., melanoma if the target area includes a mole on the subject's skin). For example, an aggregate score (herein) of 7 or greater may indicate a probability of melanoma development. In some embodiments, one or more notifications (e.g., pop-up windows, highlighting, etc.) may be generated on the screen when the probability of tissue remodeling or tumor development (e.g., melanoma development) in the target area is greater than one or more predefined thresholds. It will be understood by those skilled in the art based on the teachings herein that the range of scores (e.g., 1-10) is non-limiting and may be defined as desired (e.g., arbitrarily, or according to medical rating designations, etc.) to best capture and report the subject's probability for the target tissue of interest.

[0086]

[0119] FIG. 10 shows an exemplary user interface depicting a patient card with a visible light image displayed. The screen may be substantially similar to the screen in FIG. 9, except that the user interface is configured to allow the user to toggle or switch between the visible light image or the fluorescent image in the optimal image pair to provide a bi-layer effect. The user may, for example, click the word "fluorescent" to switch to the fluorescent image, or "visible" to switch to the visible light image. Alternatively, or in combination, the user may use a slider to switch between images. In some embodiments, the slider may be operable to overlay the images to various degrees (e.g., 50% visible and 50% fluorescent) at intermediate slider positions between "fluorescent" and "visible". Overlaying the images may allow the user to see where biotag retention occurs on the tissue surface relative to visually evident features (e.g., moles, lesions, etc.).

[0087]

[0120] FIG. 11 shows an exemplary user interface depicting a patient card with an anatomical image displayed. The screen may be substantially similar to the screen in FIG. 9, except that an anatomical image(s) may be displayed in place of the best image pair. The user may assign the anatomical image to a location on the schematic body diagram. For example, the user may place a fiducial marker on the schematic body diagram and assign the anatomical image of the target area containing the fiducial marker to a desired location. The anatomical image may act as a reference image for the user, allowing the user to differentiate moles during a session as well as follow up on the same mole in another session.

[0088]

[0121] 12 shows an exemplary user interface depicting a patient history screen. The patient history screen may be utilized to search within a patient list to find a particular patient, for example, when loading data from an existing patient instead of creating a new patient card. The patient history screen may display information about the patient (e.g., first name, last name, patient ID, etc.), as well as other relevant information about the patient's medical history (e.g., file creation date, last scan date, physician name, record locator, etc.). The patient history screen may also allow the user to search for patients, physicians, sessions, etc., optionally within a specified date range.

[0089]

[0122] 13 shows an example user interface depicting a report generation screen. The report generation screen may be configured to allow a user to generate a report (e.g., in PDF format) for printing and / or sharing with other users, e.g., for consultation. The report generation screen may be configured to allow a user to select information to be included in the report. Such information may optionally include the mole(s) of interest, visible light images, fluorescence images, anatomical images, schematic body diagrams, patient medical history, etc.

[0090]

[0123] FIG. 14 shows an exemplary user interface depicting a pathology report entry screen. A user may import a pathology report into the user interface as a PDF file. The user may then fill in the relevant information from the report. The relevant information about the mole may include, for example, diagnostic classification, Clark level, Breslow thickness, presence of ulceration, presence of mitoses, and / or presence of involution, or the like, or any combination thereof. In some embodiments, the relevant information may be automatically pulled into the program from the pathology report. In some cases, the diagnostic report from the pathologist may not be clear, and the mole may be sent to multiple pathologists for examination. The user may then fill in the blanks using drop-downs and / or radio groups based on the collation of various reports. The pathology report results may be searchable. Correlation of the pathological diagnosis with the tissue remodeling score may allow for further refinement of the scoring method (e.g., by machine-based learning).

[0091]

[0124] 15 shows an exemplary user interface depicting a backup screen. The backup screen may be accessed via a cloud tab. A user may securely backup and / or restore data from the cloud. Patient data may optionally be extracted and backed up on the cloud without any patient identifying information. Such data may then be used for machine-based learning, as an example.

[0092]

[0125] FIG. 16 shows an exemplary user interface depicting a restore from backup screen. A user may select an option to restore patient information from backup on the backup screen shown in FIG. 15. A searchable patient list may then be displayed to the user, and the user may select the patient and / or session to restore. The restore from backup screen may display information about the patient (e.g., first name, last name, patient ID, etc.), as well as other relevant information about the patient's medical history (e.g., file creation date, last scan date, doctor name, record locator, etc.). The restore from backup screen may also allow the user to search for patients, doctors, sessions, etc., optionally within a specified date range. In some cases, the user may choose to restore data in the event of a hardware or database problem on the local processor. The user may selectively restore patient data (e.g., restore only the desired patients, restore only patients still in the clinic, restore only patients arriving at the clinic that day, etc.), or restore the entire database.

[0093]

[0126] 17 shows an exemplary user interface depicting a backup in progress screen. A user may select the option to backup their data to the cloud on the backup screen shown in FIG. 15. Alternatively, or in combination, cloud backups may be scheduled to occur automatically. All patient data stored locally may be backed up to the cloud.

[0094]

[0127] It should be understood by those skilled in the art that the user interface described herein may have many variations to provide the user with a way to input data and read output. For example, one or more of the tabs may be replaced with drop-down menus, toggles, or the like for navigation where it is desired to move between screens. Alternatively, or in combination, one or more of the drop-downs may be replaced with buttons, scroll bars, steppers, radio groups, switches, sliders, text boxes, or other input mechanisms. The user interface may comprise any number or combination of organizational mechanisms and / or input mechanisms as desired to provide a simple and seamless experience to the user and provide input variables to the system. Output may include one or more graphics, one or more ratings, one or more scores, or the like.

[0095]

[0128] FIG. 18 shows a flow chart of a method 1800 for determining an optimal image pair.

[0129] In step 1801, for each image in the set of images, one or more fiducial markers may be identified. The images may be visible or fluorescent images. If the fiducial markers are present, the image may be classified as a clinical image. If the fiducial markers are not present, the image may be classified as a reference image. In some embodiments, the target area may include a mole on the subject's skin, and the multiple images (visible and fluorescent) may be images of the mole. In some embodiments, the one or more fiducial markers identified on the fluorescent image may be the same as or different from the one or more fiducial markers identified on the visible light image.

[0096]

[0130] In step 1802, for each clinical image in the set of clinical images, the location, orientation, and scale of a fiducial marker may be determined.

[0131] In step 1803, for each clinical image, the numerical values ​​printed on the fiducial markers may be read.

[0097]

[0132] In step 1804, for each clinical image, a focus score may be assigned to the fiducial markers.

[0133] In step 1805, for each clinical image, the image may be evaluated to determine whether it is a fluorescent image or a visible image.

[0098]

[0134] In step 1806 , a list may be generated using the unique numeric values ​​obtained from the reference markers in step 1803 .

[0135] In step 1807, all pairs of visible and fluorescent images that have the same numerical value may be found within the set of clinical images.

[0099]

[0136] For each set of images with the same numerical value, a similarity between the locations of the reference markers may be calculated in step 1808. The similarity may include the aggregate distance of the locations of the reference markers between the four corners of each image.

[0100]

[0137] In step 1809, for each image in the set of images having the same numerical value, the focus of each fiducial marker may be tallied between the visible and fluorescent images.

[0138] In step 1810, for each set of images having the same numerical value, an optimal visible and fluorescent image pair may be selected. The optimal image pair may include a first visible light image selected from the set of visible light images. The optimal image pair may further include a first fluorescent image selected from the set of fluorescent images. The optimal image pair may have the highest similarity at the criterion location as determined in step 1808. If two image pairs have the same similarity, the pair with the best focus as determined in step 1809 may be selected as the optimal image pair.

[0101]

[0139] The above steps illustrate a method 1800 for identifying optimal image pairs according to an embodiment, however, one of ordinary skill in the art would recognize many variations based on the teachings described herein. Steps may be completed in different orders. Steps may be added or removed. Some of the steps may include sub-steps. Many of the steps may be repeated as necessary to determine optimal image pairs.

[0102]

[0140] For example, in some embodiments, step 1810 may occur in multiple steps such that multiple image pairs are identified and the user has the option to select or reject image pairs to identify optimal image pairs. Alternatively, or in combination, step 1810 optionally occurs automatically (e.g., without user input).

[0103]

[0141] In some embodiments, the multiple images may be processed before, during, or after determining the optimal image pair. Processing the multiple images may include at least one of the following: size filtering, normalization, standardization, noise reduction, removal of imaging artifacts, background subtraction, cropping, enlarging, resizing, repositioning, brightness adjustment, contrast adjustment, or object segmentation.

[0104]

[0142] FIG. 19 shows a flow chart of a method 1900 for segmenting and scoring optimal image pairs.

[0143] In step 1901, the multiple images may be processed to determine an optimal image pair for the target region as described herein. The optimal image pair may include a first visible light image selected from the set of visible light images. The optimal image pair may further include a first fluorescent image selected from the set of fluorescent images. The visible light image and the fluorescent image may be acquired as described herein (e.g., as shown in FIG. 2). The optimal image pair may be selected to complement or match each other. The optimal image pair may be determined based on one or more of the following: focus image quality, contrast, sharpness, brightness, color, or resolution. Alternatively, or in combination, the optimal image pair may be determined from the multiple images (visible and fluorescent) based at least on the location of the target region in each of the visible light images and each of the fluorescent images, and / or based on the degree of similarity or correlation between each of the visible light images and each of the fluorescent images, as described herein.

[0105]

[0144] In some embodiments, the step of processing the multiple images may include at least one of the following: size filtering, normalization, standardization, noise reduction, removal of imaging artifacts, background subtraction, cropping, enlarging, resizing, repositioning, brightness adjustment, contrast adjustment, or object segmentation.

[0106]

[0145] In some embodiments, multiple image pairs may be determined. The multiple image pairs may include an optimal image pair. The optimal image pair may optionally be annotated to visually distinguish it from other image pairs. For example, the optimal image pair may be annotated by placing a predefined border around the optimal image pair that is displayed to the user (e.g., for selection or rejection by the user as described herein).

[0107]

[0146] In some embodiments, the target area may include a mole on the subject's skin, and the multiple images (visible and fluorescent) may be images of the mole.

[0147] In step 1902, the visible light image may be segmented to identify a region of interest. For example, when the target region includes a mole, the visible light image may be segmented to find the boundary of the mole. In some embodiments, the mole may be automatically detected within the target region and a graphic border or outline may be generated and displayed around the mole. The graphic border or outline may be automatically generated to follow the shape or contour of the mole. In some embodiments, the graphic border or outline may be adjustable or created by a user via a graphic user interface.

[0108]

[0148] In some embodiments, the mole boundary may be identified automatically. Segmentation of the mole boundary from the visible image may begin by converting the visible image (e.g., an RGB image) to an intensity image. For each possible value of a threshold T, a mask of pixel values ​​in the intensity image that are less than the threshold T may be created. The maximum connected component in the mask may be found and any holes in the mask may be filled. Pixels on the mask boundary that are inside the mole may be identified. Pixels on the mask boundary that are outside the mole may be identified. The average contrast between the inner and outer boundary pixels may be calculated. The threshold T that results in the maximum average contrast may be identified. After the threshold T that results in the maximum average contrast is identified, that threshold T may be used to segment the intensity image. Any holes in the segmented image may be filled. The mole outline may be refined by adjusting the mole boundary radially inwards or outwards to maximize local contrast. After adjusting each mole boundary pixel, the refined mole outline may be retained.

[0109]

[0149] In some embodiments, the mole boundary may be initially identified by a user via a graphic user interface and then automatically refined. When a user (e.g., a clinician) creates a manual outline of the mole, the mole outline may be automatically refined by converting a visual image (e.g., an RGB image) into an intensity image. The outline created by the clinician may be used to segment the intensity image. Any holes in the segmented image may be filled. The mole outline may be refined by adjusting the mole boundary radially inwards or outwards to maximize local contrast. After adjusting each mole boundary pixel, the refined mole outline may be retained.

[0110]

[0150] In step 1903, the locations of the fiducial markers (e.g., identified during determination of the optimal image pair) may be used to transform the mole boundary identified in step 1902 from the visible light image to the fluorescence image of the optimal image pair. In some embodiments, one or more fiducial markers may be identified on the fluorescence image of the optimal image pair and one or more fiducial markers may be identified on the visible light image of the optimal image pair. The one or more fiducial markers may be the same as or different from the one or more fiducial markers identified on the visible light image.

[0111]

[0151] At step 1904, a first set of indices may be extracted from visible features of the visible light image. The first set of indices may include one or more of the following: size, shape, volume, color, or surface texture of the mole and the area around it. The visible light image may be segmented as described herein, and the first set of indices may include indices from one or more segments as described herein. For example, a macro area around the mole may be segmented as described herein.

[0112]

[0152] In step 1905, a second set of indices may be extracted from visible (e.g., fluorescent) features of the fluorescence image. The second set of indices may include one or more of the following: size, shape, area or degree of tissue remodeling, pixel intensity, fluorescence intensity, or pattern or character in the target area beneath the subject's skin and around the mole. The fluorescence image may be segmented as described herein, and the first set of indices may include indices from one or more segments as described herein. For example, a macro area around the mole may be segmented as described herein.

[0113]

[0153] In step 1906, the visible image indicia may be scored to generate a visibility score.

[0154] In step 1907, the fluorescence image indicia may be scored to generate a fluorescence score.

[0114]

[0155] At step 1908, an aggregate score may be generated for the optimal image pair. The aggregate score may be indicative of tissue remodeling activity. The aggregate score may be a composite of two or more separate scores. For example, the aggregate score may be a composite of a visible score for the visible light image and a fluorescent score for the fluorescent image. In some embodiments, the two or more separate scores may be equally weighted, such that the score for the visible light image and the score for the fluorescent image are both weighted the same. Alternatively, the two or more separate scores may be weighted differently, such that the score for the visible light image and the score for the fluorescent image are weighted differently. In some embodiments, the score for the visible light image may be weighted higher than the score for the fluorescent image. Alternatively, the score for the visible light image may be weighted lower than the score for the fluorescent image.

[0115]

[0156] The aggregate score may be a numerical score along a scale that may provide a likelihood of tissue remodeling. In some embodiments, the aggregate score may be within a range of values. In some embodiments, two or more separate scores may be within a range of values. In some embodiments, the score for the visible light image and the score for the fluorescent image are based on a standardized range of values.

[0116]

[0157] The probability of tissue remodeling may be correlated with the probability of developing a tumor (eg, melanoma if the target area includes a mole on the subject's skin).

[0158] The aggregate score may be, for example, in the range of 0 to 10. A low score may indicate a low probability of tissue remodeling. A high score may indicate a high probability of tissue remodeling. For example, a score of 5 and above may indicate probable tissue remodeling. In some cases, a score of 7 or above may further indicate a probability of melanoma. In some embodiments, the aggregate score may be a combination of a visible score calculated from the ABCD index calculated from the visible image and a fluorescent score calculated from the I / T index calculated from the fluorescent image.

[0117]

[0159] In step 1909, the optimal image pair may be compared to one or more reference images, e.g., a reference image pair. The reference image pair may include a visible light reference image that is not from the set of visible light images (e.g., a visible light reference image from a previous session with the same patient or from an entirely different patient). The reference image pair may further include a fluorescent reference image that is not from the set of fluorescent images. In some embodiments, the reference image pair may or may not include a fiducial marker. For example, the optimal image pair may include one or more fiducial markers, and the reference image pair may not include a fiducial marker. Alternatively, the optimal image pair may include a first set of fiducial markers located at a predefined location and / or orientation relative to the target area (e.g., relative to the mole), and the reference image pair may include a set of fiducial markers located near the borders of each reference image in the reference image pair. In some embodiments, the reference image pair may include a photograph of the target area anatomy (e.g., a shoulder with a mole) and / or a schematic body diagram. The reference image pair may be used by a user to find the location of the mole on the body.

[0118]

[0160] In step 1910, an analysis report may be generated and displayed as described herein. The analysis report may include an assessment of tissue remodeling and / or the probability of tumor development as described herein. In some embodiments, the analysis report may be usable (alone or in combination with standard techniques) to determine a clinical diagnosis and / or course of action based on the probability of melanoma development in the mole.

[0119]

[0161] The above steps illustrate a method 1900 for identifying, segmenting, and scoring optimal image pairs, according to an embodiment, although one of ordinary skill in the art would recognize many variations based on the teachings described herein. Steps may be completed in different orders. Steps may be added or removed. Some of the steps may include sub-steps. Many of the steps may be repeated as necessary to score optimal image pairs.

[0120]

[0162] For example, in some embodiments, step 1901 may occur in multiple steps such that multiple image pairs are identified and the user has the option to select or reject image pairs to identify optimal image pairs. Alternatively, or in combination, step 1901 optionally occurs automatically (e.g., without user input).

[0121]

[0163] FIG. 20 shows an image of a mole and surrounding skin segmented into multiple segments. FIG. 21 shows an image of a mole and surrounding skin segmented into multiple segments and sub-regions. Fiducial markers may be placed on the skin adjacent to the mole as described herein. Segmentation of the macro-region around the mole into smaller localized regions may improve scoring accuracy and / or reduce false positives compared to analyzing the macro-region en bloc. The skin around the mole M, also referred to herein as the macro-region, may be segmented into multiple segments for analysis. For example, a first segment or region Z1 of the image may include the region of skin immediately around the mole M. A second segment or region Z2 may be radially outward from the first region Z1 and may include tissue of the macro-region second closest to the mole. A third segment or region Z3 may be radially outward from the second region Z2, a fourth segment may be radially outward from the third segment, and so on. In some embodiments, the macro-region may be segmented in both the visible light image and the fluorescent image of the optimal image pair. In some embodiments, the macro-region may be segmented for analysis only in the fluorescent image. The segmented image may be analyzed to identify the intensity of biotag retention in the surrounding macro-region and / or the pattern of that retention within the target region (the mole and / or the macro-region), or any other characteristic desired by one of skill in the art.

[0122]

[0164] For example, the fluorescent image may be segmented and analyzed to identify the intensity and nature of biotag retention in the target and macro regions. The red channel (or any other channel corresponding to the fluorescent signature of the biotag) may be extracted from the fluorescent image. The location of the fiducial marker 2 may be used to convert the mole boundary from the corresponding visible image to the red channel of the fluorescent image as described herein. The mole boundary may be used to create a mask to indicate the location of the mole in the fluorescent image. Concentric regions around the mole may be generated to segment the mole into segments or zones (e.g., zones Z1, Z2, Z3, etc.), as shown in FIG. 20. The segments or zones Z1, Z2, Z3, etc. may contain the signal generated by the biotag. In some embodiments, the concentric regions may be concentric rings spaced apart by the same or different distances. Alternatively, in some embodiments, the concentric regions may mimic the mole boundary and may be generated using a distance map to define the distance from the mole boundary. Each segment or area may be divided into sub-areas as shown in FIG. 21. The sub-areas may vary in number, size, location, and amount of overlap with adjacent sub-areas. In some embodiments, the segment or area may be divided into sub-areas based on an angle from the center of the mole. Each segment of the area may include one or more sub-areas, e.g., multiple sub-areas. The intensity of each sub-area may be measured. The sub-area with the greatest intensity may yield an intensity index value. In many embodiments, the segment or area Z2 that is the second closest to the mole M may ensure uniform biotag retention, and the smallest intensity sub-area within area Z2 may be identified. In some embodiments, area Z1 may be too close to the mole border, and the dye inside the mole may obscure the fluorescent signal. Area Z3, on the other hand, may be too far from the mole and may not have received any biotags. Area Z2 may be far enough away from the mole that it is not affected by the loss of mole dye or biotags. The quality index may be calculated from the maximum intensity sub-region of the image and the minimum intensity sub-region of area Z2. The quality index may be the difference in intensity between the sub-regions that received the biotag. Computer Control System

[0165] The present disclosure provides a computer control system that is programmed to carry out the method of the present disclosure. Figure 22 shows a computer system 2201 that is programmed or otherwise configured to assess tissue remodeling risk. The computer system 2201 can control various aspects of any of the methods for determining the risk of tissue remodeling of the present disclosure, such as, for example, processing a plurality of images to determine an optimal image pair, analyzing images for tissue remodeling, scoring images, and converting images and scores into tangible visual output, such as the risk and / or degree of tissue remodeling. The computer system 2201 can be a user's electronic device or a computer system that is remotely located with respect to the electronic device. The electronic device can be a mobile electronic device.

[0123]

[0166] The computer system 2201 may include a central processing unit (CPU, herein "processor" and "computer processor") 2205, which may be a single-core or multi-core processor, or multiple processors for parallel processing. The computer system 2201 also includes memory or memory locations 2210 (e.g., random access memory, read-only memory, flash memory), electronic storage 2215 (e.g., hard disk), communication interface 2220 (e.g., network adapter) for communicating with one or more other systems, and peripheral devices 2225, such as cache, other memory, data storage, and / or electronic display adapters. The memory 2210, storage 2215, interface 2220, and peripheral devices 2225 are in communication with the CPU 2205 through a communication bus (solid lines), such as a motherboard. The storage 2215 may be a data storage device (or data repository) for storing data. The computer system 2201 may be operatively coupled to a computer network ("network") 2230 with the aid of the communication interface 2220. The network 2230 may be the Internet, an Internet and / or an extranet, or an intranet and / or an extranet in communication with the Internet. The network 2230 is possibly a telecommunications and / or data network. The network 2230 may include one or more computer servers, which may enable distributed computing, such as cloud computing. The network 2230 may implement a peer-to-peer network, possibly with the aid of the computer system 2201, which may enable devices coupled to the computer system 2201 to behave as clients or servers.

[0124]

[0167] The CPU 2205 may execute sequences of machine-readable instructions, which may be embodied in a program or software. The instructions may be stored in a memory location, such as the memory 2210. The instructions may be directed to the CPU 2205, which may then program or otherwise configure the CPU 2205 to perform the methods of the present disclosure. Examples of operations performed by the CPU 2205 may include fetch, decode, execute, and writeback.

[0125]

[0168] The CPU 2205 may be part of a circuit, such as an integrated circuit. One or more other components of the system 2201 may be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

[0126]

[0169] The storage device 2215 may store files such as drivers, libraries, and saved programs. The storage device 2215 may store user data, such as user preferences and user programs. The computer system 2201 may include one or more additional data storage devices, possibly external to the computer system 2201, such as located on a remote server in communication with the computer system 2201 through an intranet or the Internet.

[0127]

[0170] The computer system 2201 may communicate with one or more remote computer systems through the network 2230. For example, the computer system 2201 may communicate with a remote computer system of a user (e.g., an operator, a patient, etc.). Examples of remote computer systems include a personal computer (e.g., a portable PC), a slate or tablet PC (e.g., an Apple® iPad®, a Samsung® Galaxy Tab), a phone, a smartphone (e.g., an Apple® iPhone®, an Android-enabled device, a Blackberry®), or a personal digital assistant. A user may access the computer system 2201 through the network 2230.

[0128]

[0171] Methods as described herein may be implemented using machine (e.g., computer processor) executable code stored in electronic storage of computer system 2201, such as, for example, on memory 2210 or electronic storage 2215. The machine executable or machine readable code may be provided in the form of software. In use, the code may be executed by processor 2205. In some cases, the code may be retrieved from storage 2215 and stored in memory 2210 for immediate access by processor 2205. In some circumstances, electronic storage 2215 may be excluded and machine executable instructions are stored on memory 2210.

[0129]

[0172] The code may be precompiled and configured for use with a machine having a processor adapted to execute the code, or may be compiled during run-time. The code may be provided in a programming language that may be selected to allow the code to be executed in a precompiled or compiled-in-place form.

[0130]

[0173] Aspects of the systems and methods provided herein, such as computer system 2201, may be embodied in programming. Various aspects of the technology may be considered to be "products" or "articles of manufacture" that are typically in the form of machine (or processor) executable code and / or associated data that are performed or embodied in some type of machine-readable medium. The machine executable code may be stored in electronic storage devices such as memory (e.g., read-only memory, random access memory, flash memory), or hard disks. A "storage" type medium may include any or all of the tangible memory of a computer, processor, or the like, or its associated modules, such as various semiconductor memories, tape drives, disk drives, and the like, that may provide non-transitory storage at any time for software programming. All or parts of the software may be communicated at times over the Internet or various other telecommunications networks. Such communication may enable loading of the software, for example, from one computer or processor into another computer or processor, for example, from a management server or host computer into the computer platform of an application server. Thus, another type of medium that may bear software elements includes light waves, radio waves, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optically wired networks, and via various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, or the like, may also be considered as media bearing software. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0131]

[0174] Thus, a machine-readable medium such as a computer executable code may take any form, including but not limited to a tangible storage medium, a carrier wave medium, or a physical transmission medium. Non-volatile storage media include optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, such as those that may be used to implement databases, etc., as shown in the figures. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wire and optical fibers, such as the wires that comprise a bus in a computer system. Carrier wave transmission media may take the form of electric or electromagnetic signals, or acoustic, optical or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punch cards paper tape, any other physical storage media with a pattern of holes, RAM, ROM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves that transport data or instructions, cables or links that transport such carrier waves, or any other medium from which a computer may read programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0132]

[0175] The computer system 2201 may include or be in communication with an electronic display 2235 with a user interface (UI) 2240 for providing patient image input data and / or tissue remodeling risk assessment output data, for example, as described herein. Examples of UIs include, without limitation, graphic user interfaces (GUIs) and web-based user interfaces. FIGS. 4-17 show exemplary user interfaces that may be provided to a user on the display. Data entered by a user into the user interface (e.g., mole border selection, etc.) may be transmitted to a processor. The processor may be configured with instructions for executing image processing and scoring algorithms as described herein to generate one or more outputs. The output(s) of the processing and / or scoring algorithms may be transmitted by the processor to a display that displays the output to a user using the user interface.

[0133]

[0176] The methods and systems of the present disclosure may be implemented using one or more algorithms. The algorithms may be implemented using software upon execution by the central processing unit 2205. The algorithms may, for example, select optimal image pairs, segment one or more images for analysis, compare visual features on the images, score image pairs, etc., as described herein.

[0134]

[0177] While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the present invention. It will be understood that various alternatives to the embodiments of the present invention described herein may be used in practicing the present invention. It is intended that the appended claims define the scope of the invention, and that methods and structures within the scope of these claims and their equivalents are thereby covered.

Claims

1. 1. A method for assessing tissue remodeling risk, comprising: (a) acquiring a plurality of images of a target area of ​​a subject's skin, the plurality of images including: (1) a set of visible light images; and (2) a set of fluorescent images; (b) processing the plurality of images to determine an optimal image pair for the target region, the optimal image pair including: (i) a first visible light image selected from (1); and (ii) a first fluorescent image selected from (2); (c) generating an assessment of the target region based at least on the optimal image pair; and (d) displaying on a graphical user interface a schematic diagram indicating the assessment, the optimal image pair, and a location of the target area on the subject's body, wherein the assessment is indicative of a probability of tissue remodeling in the target area.

2. The method of claim 1 , wherein the target area comprises a mole on the subject's skin.

3. The method of claim 2 , wherein the first visible light image and the first fluorescent image are selected to complement or match each other.

4. 3. The method of claim 2, wherein the first visible light image comprises a first set of visual features and the first fluorescent image comprises a second set of visual features, at least a portion of the second set of visual features being not visible in the first set of visual features and at least a portion of the first set of visual features being not visible in the second set of visual features.

5. 5. The method of claim 4, wherein the first set of visual features is associated with the mole and exposed skin of the subject, and the second set of visual features is associated with underlying tissue beneath the skin and around the mole.

6. 2. The method of claim 1, wherein the optimal image pair is determined from the plurality of images based at least on: (1) a location of the target area in each of the visible light images and each of the fluorescent images; (2) a degree of similarity or correlation between each of the visible light images and each of the fluorescent images; and / or (3) a focus quality of each of the visible light images and each of the fluorescent images.

7. The optimal image pair is a first optimal image pair, and after (b) and before (c), displaying options to a user on the graphical user interface, the options allowing the user to accept or reject the first optimal image pair; 2. The method of claim 1, further comprising the step of: receiving input from the user in response to the selections displayed on the graphical user interface.

8. 8. The method of claim 7, further comprising processing the plurality of images to determine a second optimal image pair for the target region when the input indicates that the user rejects the first optimal image pair, the second optimal image pair being different from the first optimal image pair.

9. 8. The method of claim 7, further comprising the step of: when the input indicates that the user accepts the first best image pair, continuing with (c) and (d).

10. when the input indicates that the user rejects the first optimal image pair, processing the plurality of images to determine a plurality of other optimal image pairs for the target region; The method of claim 7 , further comprising the step of: displaying the plurality of other optimal image pairs on the graphic user interface.

11. 3. The method of claim 2, wherein (b) further comprises automatically detecting the mole within the target area and generating a graphic border or outline around the mole.

12. 3. The method of claim 2, wherein the probability of tissue remodeling is associated with a probability of developing melanoma in the mole, and the assessment is usable to determine a clinical diagnosis or course of action based on the probability of developing melanoma in the mole.

13. 5. The method of claim 4, wherein the assessment is generated based at least on (1) a first set of indices associated with the first set of visual features in the first visible light image, and (2) a second set of indices associated with the second set of visual features in the first fluorescent image, wherein the first set of indices includes one or more of the following: size, shape, volume, color, or surface texture of the mole and the area around it, and the second set of indices includes one or more of the following: size, shape, area or degree of tissue remodeling, pixel intensity, fluorescent intensity, pattern, or texture in the target area beneath the subject's skin and around the mole.

14. 14. The method of claim 13, further comprising segmenting the first visible light image or the first fluorescent image to generate the first set of indices or the second set of indices, respectively, wherein segmenting the first fluorescent image comprises segmenting the first fluorescent image into a plurality of zones around the target area.

15. The method of claim 2 , wherein the first visible light image and the first fluorescent image are adjusted and aligned to a set of coordinates within the optimal image pair.

16. The method of claim 2 , further comprising the step of comparing the optimal image pair to one or more reference images.

17. The method of claim 16 , wherein the one or more reference images comprise a reference image pair.

18. 20. The method of claim 17, wherein a reference image pair includes: (1) a visible light reference image not from the set of visible light images; and (2) a fluorescent reference image not from the set of fluorescent images.

19. The method of claim 17 , wherein the optimum image pair includes a reference and the reference image pair does not include a reference.

20. 18. The method of claim 17, wherein the optimal image pair includes a first set of fiducials located at a predetermined location and / or orientation relative to the mole, and the reference image pair includes a set of fiducials located near a border of each reference image in the reference image pair.

Images