Systems and Methods for Automated Digital Phenotyping and Analysis of Bone Biopsy Images Using Deep Learning

Abstract

In some examples, the disclosure relates to a digital phenotyping method. For example, the method may include operating upon digitized pathology image data of a bone sample from a patient with one or more models to generate one or more feature maps for one or more components. The one or models may include one or more deep learning models, one or more computer vision models, among others, or any combination thereof. In some examples,+ the one or more components may include osteoclast, osteoblast, bone marrow bone marrow adipose tissue (BMAT), mineralized bone, and / or osteoid.

Description

SYSTEMS AND METHODS FOR AUTOMATED DIGITAL PHENOTYPING ANDANALYSIS OF BONE BIOPSY IMAGES USING DEEP LEARNINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 699,650 filed September 26, 2024. The entirety of this application is hereby incorporated by reference for all purposes.BACKGROUND

[0002] Histomorphometric analysis of undecalcified bone biopsy images provides quantitative assessment of bone turnover, volume, and mineralization using static and dynamic parameters respectively. Traditionally, quantification has relied on manual annotation and tracing of relevant tissue structures, a process that can be time-intensive and subject to inter-operator variability.SUMMARY

[0003] Thus, there is a need for automated systems and methods for digital phenotyping.

[0004] Additional advantages of the disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the disclosure. The advantages of the disclosure will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure, as claimed.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosure can be better understood with the reference to the following drawings and description. The components in the figures are not necessarily to scale, emphasis being placed upon illustrating the principles of the disclosure.

[0006] Figure 1 illustrates an example of a block diagram of some embodiments of a system configured to automatically generate component-specific feature maps from digital pathology imaging data to generate one or more parameters and / or medical predictions.

[0007] Figure 2 illustrates an example of a block diagram of some additional embodiments of a system configured to automatically generate component-specific feature maps that can be used for analysis.

[0008] Figure 3 shows an illustrative example of some exemplary images related to the identification of osteoblasts using the system according to some embodiments.

[0009] Figure 4A illustrates some embodiments of a flow diagram showing some embodiments of a method for automatically generating component feature maps from digital pathology imaging data to generate one or more parameters and / or predictions. Figure 4B illustrates an example of a flow diagram showing some embodiments of a method for generating one or more parameters using texture features extracted from component feature maps using the system according to embodiments.

[0010] Figure 5 illustrates an example of a block diagram of some additional embodiments of the system configured to utilize component feature maps generated from digitized pathology imaging data to generate one or more parameters and / or one or more medical predictions.

[0011] Figures 6A-E illustrate some exemplary images of components segmented from digitized pathology imaging data of a patient using the system according to some embodiments. Figures 6A-E show exemplary images of segmented bone, osteoid, osteoclasts, osteoblasts, and BMAT, respectively.

[0012] Figures 7A-E illustrates plots showing exemplary performance results related to the component segmentation using the system according to some embodiments. Figures 7A-E show plots showing performance results related to the segmentation of bone, osteoid, osteoclasts, osteoblasts, and BMAT, respectively.

[0013] Figure 8A illustrates some exemplary images of BMAT regions with high bone turnover and associated heatmaps generated using the system according to some embodiments. Figure 8B illustrates some exemplary images of BMAT regions with low bone turnover and associated heatmaps generated using the system according to embodiments. Figure 8C illustrates exemplary violin plots for MTA features showing values for “low” (x-axis label “0”) and “high” bone turnover images (x-axis label “1”).

[0014] Figure 9 illustrates an example of a block diagram of embodiments of a system configured to generate one or more parameters and / or one or more medical predictions fromradiology image data using a machine learning model(s) trained using the component-specific feature maps generated from digital pathology imaging data according to some embodiments.

[0015] Figure 10 illustrates an example of a block diagram of some additional embodiments of a system configured to generate one or more parameters and / or one or more medical predictions from radiology image data using a machine learning model(s) trained using the componentspecific feature maps generated from digital pathology imaging data according to some embodiments.

[0016] Figure 11 illustrates some embodiments of a block diagram of a system configured to generate one or more component feature maps and / or one or more parameters and / or more medical predications according to some embodiments.DESCRIPTION OF THE EMBODIMENTS

[0017] In the following description and Appendices, numerous specific details are set forth such as examples of specific components, devices, methods, etc., in order to provide a thorough understanding of embodiments of the disclosure. It will be apparent, however, to one skilled in the art that these specific details need not be employed to practice embodiments of the disclosure. In other instances, well-known materials or methods have not been described in detail in order to avoid unnecessarily obscuring embodiments of the disclosure. While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure.

[0018] Diagnostic bone biopsies play a pivotal role in evaluating metabolic bone disorders such as osteoporosis, osteomalacia, hyperparathyroid bone disease, and renal osteodystrophy. Undecalcified bone histology, used in conjunction with fluorochrome labeling, has been the gold standard for evaluating bone turnover (coupling of bone formation and resorption) and mineralization (organic bone matrix filled with calcium phosphate nanocrystals, essential for the hardness and strength of bone); these processes can provide standard therapeutic targets for antiremodeling, anabolic or other medications.

[0019] Histomorphometric reporting has generally relied on visual identification of histologic primitives (structures) by experienced pathologists, which can be very time consuming. Detailed feature maps are manually created by point labeling cells like osteoblasts and osteoclasts, and contour tracing outlines of mineralized bone and osteoid for areal dimensions, erosion cavities for width and depth, and fluorochrome double labels for extent and inter-label width. The process generally relies largely on the expertise of the pathologist, can be time-consuming, and can introduce subjectivity and inter- and intra- operator variability. As a result, histomorphometric reporting of bone biopsies has often been limited to research settings, while clinical diagnostics has relied on more on qualitative interpretation.

[0020] The present disclosure relates to systems and methods for digital phenotyping bone biopsy images to generate segment feature maps for static histomorphometry by delineating histologic primitives. In some examples, the systems and methods may identify one or more components within a digitized pathology image of a bone sample taken from a patient (e.g., bone biopsy image). In some examples, the one or more components may include but is not limited to one or more cell components, one or more tissue regions, among others, or any combination thereof. For example, the one or more components (also referred to as “regions”) may include but is not limited to mineralized bone, osteoid, osteoblasts, osteoclasts, bone marrow adipose tissue (BMAT), among others, or any combination thereof. In some examples, the digitized pathology image may be broken into a plurality of non-overlapping patches. One or more machine learning models may be operated upon the patches to segment one or more components from the digitized pathology image. In some examples, the segmented component(s) may be represented by component feature map (also referred to as “feature map”) for the one or more components.

[0021] In some examples, the component feature map for osteoid and bone may be used to generate an osteoid-bone mask. In some examples, for osteoblasts and / or osteoclasts, a machinelearning model for the respective component may each operate upon the group of patches and the osteoid-bone mask to generate a component feature map for that component. In some examples, the component feature map for that component and the osteoid-bone mask may then be processed through a computer vision model to generate an augmented component feature map for the respective component (e.g., osteoblasts and / or osteoclasts). The component feature map and the augmented feature map may be then combined to generate a (fused) feature map for thatcomponent. In some examples, one or more of the feature maps may be overlaid on the original pathology image to generate a reconstructed image.

[0022] In some examples, one or more analyses may be performed using the component feature maps and / or reconstructed image to determine one or more parameters (also referred to as “indices”), one or more medical predictions, among others, or any combination thereof. For example, one or more features may be extracted from the component feature map(s) and / or the reconstructed image, for example, using imaging processing. The one or more features may include but is not limited to texture features, tissue features, cell features, among others, or any combination thereof. The one or more features may be static or dynamic features. In some examples, one or more machine-learning models may operate upon the one or more features and / or one or more (other) parameters to generate one or more parameters and / or one or more medical predictions.

[0023] By way of example, the one or more static parameters may include but is not limited to one or more parameters that quantitatively and / or qualitatively describe tissue and / or cellular components, such as, bone’s architecture, bone mass, organization (e.g., arrangement of cells), spatial interactions (e.g., between cells of components and / or other structure / components), bone structure, bone formation, bone resorption, cell-level activities, bone mineralization, marrow composition metrics, among others, or any combination thereof, at a specific point in time.

[0024] For example, the one or more static parameters may include but is not limited to: bone volume fraction (Bone Volume / Tissue Volume); trabecular parameter(s) (e.g., trabecular number (Tb.N) (e.g., number of trabeculae within a given area of bone tissue), trabecular thickness (Tb.Th) (e.g., a measure of thickness of trabeculae), trabecular separation (Tb.Sp)(e.g., the distance between individual trabeculae, other trabecular metrics, among others, or any combination thereof)); cortical parameter(s) (e.g., cortical thickness, cortical porosity, other cortical metrics, among others, or any combination thereof); osteoblast parameter(s)(e.g., osteoblast number, osteoblast surface, osteoblast surface fraction (osteoblast surface / bone surface), other osteoblast metrics, among others, or any combination thereof); osteoclast parameter(s)(e.g., osteoclast number, osteoclast surface, osteoclast surface fraction (osteoclast surface / bone surface), other osteoclast metrics, among others, or any combination thereof); osteoid parameter(s)(e.g., osteoid area; osteoid volume fraction (Osteoid Volume / Bone Volume), osteoid surface fraction (e.g., Osteoid Surface / Bone Surface), osteoid thickness, other osteoid metrics, among others, or anycombination thereof); wall thickness (e.g., thickness of new bone formed during a remodeling cycle, among others, or any combination thereof); marrow composition parameters (e.g., outlines of BMAT, area of BMAT, BMAT number, number of hematopoietic cells, other marrow composition metrics, among others, or any combination thereof); spatial arrangement of cells distribution of one or more cells (e.g., distribution, pattern, separation (e.g., average distance between respective cells), interaction between cells, separation from other landmarks (e.g., how far the cell (e.g., osteoblast is from newly formed bone, etc.)); average nearest-neighbor distance between osteoclasts and osteoblasts; clustering parameters; co-localization along resorption or formation surfaces; osteoblast-osteoclast neighborhood networks; mineralization defects (e.g., derived from osteoid thickness); connectivity parameters (e.g., connectivity density); structure model index (SMI); bone mineral density (e.g., score value); bone strength; texture param eter(s); other static histomorphometric parameter(s); other microarchitecture parameter(s); other area parameter(s); other surface area metric(s); among others, or any combination thereof.

[0025] By way of another example, the one or more dynamic parameters may include one or more parameters that quantitatively and / or qualitatively describe bone formation over a period of time (e.g., for at least two points of time). For example, the one or more dynamic parameters may include but is not limited to mineral apposition rate (MAR); bone formation rate (BFR); activation frequency; spatial heterogeneity of turnover components; mineralization lag time; estimated bone turnover rate (e.g., low, normal, high); other dynamic surface metrics; other dynamic remodeling metrics; among others, or any combination thereof.

[0026] For example, the estimated bone turnover state may be inferred from one or more parameters. By way of example, the estimated bone turnover may be categorized as “high” if the estimated bone turnover is above a reference range of values defined as “normal.” “High” bone turnover may have histology showing osteoblast surface with high osteoid surface indicative of bone formation and high osteoclast surface with eroded surface indicative of resorption. This can correlate to fluorescent microscopy showing high mineralizing surface with mainly double tetracycline labels. Bone turnover may be categorized as “low” if the estimated bone turnover is below a reference range of values defined as “normal.” “Low” bone turnover may have histology showing low osteoblast surface with low osteoid surface indicative of decreased bone formation and low osteoclast surface. This can correlate to fluorescent microscopy showing low mineralizing surface with no, or few, mainly single tetracycline labels.

[0027] In some examples, the one or more analyses may additionally and / or optionally include generating one or more medical predictions related to bone health of the patient using the generated feature segmented maps, the reconstructed image, and / or the param eter(s). The one or more medical predictions may include but not limited to diagnosing, monitoring, prognosing, screening and / or evaluating bone quality and / or strength, musculoskeletal disease / condition (e.g., osteoporosis, renal osteodystrophy, etc.), treatment response, among others, or any combination thereof. In some examples, the one more medical predications may be provided as a qualitative (e g., low, normal, high) and / or quantitative value (e.g., score). For example, the one or more medical predications may include but is not limited to diagnosis, classification, prognosis, prediction and / or risk of a musculoskeletal disease / condition (e.g., osteoporosis, osteomalacia, renal osteodystrophy, other bone diseases, etc.), bone health (e.g., bone strength and / or quality), among others, or any combination thereof; diagnosis, classification, prognosis, prediction and / or risk of an outcome (e.g., fracture, change in one or more indices values (e.g., bone strength, etc.)); prediction of a treatment response (e.g., change in one or more indices and / or medical predictions) to one or more treatments for musculoskeletal disease / condition, other diseases (e.g., chronic kidney disease (CKD), among others, etc.)), among others, or any combination thereof; among others; or any combination thereof.

[0028] The generated component feature maps according to the disclosure are at a higher order or higher level that a human can resolve in the human mind or with pencil and paper. For example, the disclosed methods and systems enable identification of histologic primitives (structures) from bone biopsy images more quickly and accurately than the existing approaches. Additionally, the component feature map generated according to the disclosure includes features that are not discernable to a human eye and cannot be practically performed in the human mind. Additionally, the one or more parameters, such as texture, interaction between cells / spatial graphs, and / or morphological features, cannot be visually distinguishable in digitized pathology images and thus quantifiable by an unaided human eye or in the human mind.

[0029] In some examples, the systems and methods may be configured to generate one or more parameters and / or one or more medical predictions using component features extracted from radiological imaging. The radiological imaging may include but is not limited to DXA (Dualenergy X-ray Absorptiometry), CT (Computed Tomography)(e.g., HR-pQCT (high-resolution peripheral quantitative CT), MRI (Magnetic Resonance Imaging), and X-ray scanner, amongothers, or any combination thereof. In some examples, the one or more machine learning models may be trained to map the feature maps and / or parameters generated from digitized pathology images to radiological imaging for the same anatomical component so that one or more parameters and / or one or more medical predictions may be generated directly from the radiological imaging. The disclosed systems and methods can therefore provide a widely accessible, less invasive, and cost-effective method of accurately generation of one or more parameters and / or one or more medical predictions from musculoskeletal imaging. This can enable scalability for populationlevel screening. Additionally, the disclosed system sand methods can provide the ability to perform longitudinal monitoring of disease, therapy response, etc.

[0030] Figure 1 illustrates a block diagram of some embodiments of a digital phenotyping system 100 (also referred to as “ADAM”) configured to utilize (segmented) component feature maps generated from digitized pathology imaging data from a patient to generate one or more parameters and / or one or more medical predictions.

[0031] The system 100 can include a memory 110 configured to store one or more digitized pathology image data 112 from a patient. In some examples, the digitized pathology image data 112 may include one or more digitized images of a bone sample taken from a patient.

[0032] In some examples, the bone sample may be taken from the iliac area, trabecular bone, lumbar, vertebrae, sternum, cortical bone areas (e.g., tibial cortex, femoral neck, shaft, etc.), other areas of bone(s), other bone(s), or any combination thereof. For example, the digitized pathology image data 112 may include a digitized slide. In some examples, the image data 112 may be acquired using light microscopy, a fluorescence microscopy, among others, or any combination thereof. In some examples, the slide may be stained with Masson-Golder type stain, other stains, or any combination thereof. In some examples, the sample may be stained with only a Masson- Goldner type stain, such as a modified Masson-Goldner trichrome stain.

[0033] In some examples, the system 100 may include a component identification tool 120 configured to segment the digitized pathology image data 112 to generate to identify one or more components within the image data. For example, the one or more components may include bone, osteoid, osteoblasts, osteoclasts, BMAT, among others, or any combination thereof. In some examples, the component identification tool 120 may be configured to generate one or more component feature maps for one or more components and / or a reconstructed image generated from the generated component feature maps. In some examples, the component identification tool 120may be configured to delineate bone-osteoid, osteoblasts, osteoclasts, and BMAT from the image data.

[0034] In some examples, the component identification tool 120 may optionally include one or more pre-processing tools 130. In some examples, the pre-processing tool 130 may include a patch generator configured to segment each digitized pathology image 112 into one or more groups of non-overlapping patches. Each group may have different number and / or size of patches. In some examples, the pre-processing tool(s) 130 may include a tool configured to stain normalize each patch and / or the whole digitized pathology image.

[0035] In some examples, each group of patches 132 and / or the normalized whole digitized pathology image may be provided to a machine learning identification tool 140 to segment the imaging data 112 to identify one or more of the components 150, such as, by generating a component feature map 140 for the respective component(s). In some examples, the machine learning identification tool 140 may include one or more machine learning models 142i-142n implemented as computer code run on one or more processors (e.g., a central processing unit (CPU) including one or more transistor devices configured to operate computer code to achieve a result, a microcontroller, a graphics processing unit (GPU), and / or the like). In some examples, the models 142 may include but are not limited to deep learning models (e.g., self-configuring U- net-based segmentation model), such as neural network(s) (e.g., CNN)), computer vision model(s), other deep learning and / or machine learning models, or any combination thereof.

[0036] In some examples, the one or more models may be configured to identify a subset of one or more of the components 150. For example, the one or machine learning models may include but is not limited to: one or more machine learning models trained specifically to segment osteoid and bone from the patches 132 and / or whole image data to identify osteoid and bone in the image data 112; one or more machine learning models trained specifically to segment osteoclast cells from the patches 132 and / or whole image data to identify osteoclasts in the image data 112; one or more trained models specific osteoblasts configured to segment osteoblast cells from the patches 132 and / or whole image data to identify osteoblasts in the image data 112; one or more trained models specific to BMAT configured to segment BMAT from the patches 132 and / or whole image data to identify BMAT in the image data 112; among others; or any combination thereof.

[0037] In some examples, the one or more models 140 may be operated upon the normalized whole digitized pathology image and / or each group of patches 132. For example, a deep learningmodel may be operated upon the group of patches to generate a component feature map and a computer vision model may be operated upon the group of patches for that component to augment the component feature map.

[0038] In some examples, the generated component feature map may be used to generate a mask that can be used as an additional input for one or more other component’s model. For example, the feature map for osteoid-bone can be used to generate a mask that can be inputted into the model(s) for osteoblasts or osteoclasts. In some examples, more than one model may be used to generate the feature map for a component. For example, for osteoblasts and / or osteoclasts, a machine-learning model and a computer vision model may act upon the group of patches 142 and osteoid mask to generate the respective component feature map. Figure 2 shows an illustrative example of generating a feature map for osteoblasts.

[0039] For example, the machine learning identification tool 140 may include but is not limited to: one or more machine learning models 142 for osteoid-bone configured to segment bone and osteoid and generate the related component feature map 150 from the patches 132; one or more machine learning models 142 for osteoclasts configured to segment osteoclasts and generate the related component feature map 152 from the patches 132; one or more machine learning models 142 for osteoblasts configured to segment osteoblast and generate the related component feature map 152 from the patches 132; one or more machine learning models 142 for BMAT configured to segment BMAT and generate the related component feature map 152 from the patches 130; among others, or any combination thereof.

[0040] In some examples, each feature component map 150 (152i-152n) may be combined with the original pathology image 112 to generate a reconstructed image 160 specific to the component. In some examples, all of the feature maps 150 may be combined to generate a single, reconstructed image 150.

[0041] In some examples, the system 100 may include an analysis tool 170 configured to generate one or more parameters and / or one or more medical predications using the segmented feature map(s) 150 and / or reconstructed image 160.

[0042] In some examples, the analysis tool 170 may include one or more feature extraction tool 172 configured to extract one or more features for one or more components from the one or more feature components maps 150. By way of example, the one or more features may include but is not limited to pixel intensity features, Haralick, and Fourier shape descriptors; spatialarrangement features; graphical features; cellular features / parameters, such as cell count, area / or volume of the component; among others, or any combination thereof. In some examples, one or more features may be determined for each component. The one or more features for each component may be the same and / or different. In some examples, the feature extraction tool 114 may be implemented as computer code run on one or more processors (e.g., a CPU, a microcontroller, a GPU, and / or the like). For example, the feature extraction tool 172 may use image processing to extract the one or more features. In some examples, some of the features may be generated by measuring the one or more features of each component.

[0043] In some examples, the one or more extracted features may be provided to a machine learning stage 174, including one or more machine learning models, and / or a determination tool 176 configured to generate one or more parameters and / or one or more medical predictions 180. In some examples, the one or more machine learning models may include but is not limited to support vector classifiers, Random Forrest, linear regression, among others, or a combination thereof. By way of example, the one or more machine learning models may include one more models configured to generate MTA-related parameters (e.g., using the extracted texture features); one or more models configured to generate other static and / or dynamic parameter(s); one or more models configured to generate one or more medical predictions (e.g., using the one or more extracted features); among others; or a combination thereof.

[0044] In some examples, the determination tool 176 may be implemented as computer code run on one or more processors (e.g., a CPU, a microcontroller, a GPU, and / or the like). For example, the determination tool 176 may be configured to apply one or more algorithms to generate one or more parameters and / or one or more medical predictions using (i) the one or more features and / or (ii) the (other) parameter(s) and / or medical prediction (s) generated by the machine learning model(s) of the machine learning stage 174 and / or the determination tool 176. For example, the one or more algorithms may include one or more deterministic algorithm(s) that can calculate the one or more parameters and / the one or more medical predictions. In some examples, the one or more algorithms may include mathematical algorithms, statistical algorithms, among others, or any combination thereof.

[0045] In some examples, the one or more parameters and / or the one or more medical predictions generated by the operating one or more machine learning model(s) 174 on the extracted features may relate to generating one or more texture parameters (e.g., bone turnover rate / stateusing the disclosed textural analysis); one or more medical predictions (e.g., fracture risk, etc.); other parameters; among others; or any combination thereof.

[0046] In some examples, the one or more parameters and / or the one or more medical predictions generated by the determination tool 176 may relate to applying one or more algorithms to the extracted features. For example, the one or more algorithms may be configured to generate physical parameter(s) of the respective component (e.g., volume, density, surface area, ratios, etc.); spatial param eter(s) of the respective component (e.g., distribution, spacing, spatial clustering, nearest-neighbor distances, etc ); other parameters and / or medical predictions; among others; or any combination thereof. For example, the spatial parameter(s) may be generated using one or more graphical algorithms (e.g., Delaunay Triangulation, Minimum Spanning Tree, Voronoi diagram, etc.).

[0047] In some examples, the one or more parameters and / or the one or more medical predictions generated by the one or more machine learning models of the machine learning stage 174 can be used by the determination tool 176 and other machine learning model(s) of the machine learning stage 174 to generate one or more (other) parameters and / or one or more (other) medical predictions. Similarly, the one or more machine learning models of the machine learning stage 74 can operate on the one or more one or more parameters and / or the one or more medical predictions generated by the determination tool 176 to generate one or more (other) parameters and / or one or more (other) medical predictions.

[0048] In some examples, a report may be generated with the one or more component feature maps, the one or more parameters, the one or more medical predictions, among others, or any combination thereof. For example, the report may be transmitted, display, or otherwise outputted. In further examples, a treatment may be administered based on the generated param eter(s) and / or medical prediction(s).

[0049] Figure 2 illustrates a block diagram an example 200 of some additional embodiments of a system 120. This example 200 relates to the embodiments of the component identification tool 120 used to generate the feature map for osteoblasts.

[0050] The group 210 of (normalized) patches for osteoid and bone tissue generated at block 130 may be provided to a deep learning model 220. The deep learning model 220, for example, trained on osteoid and bone tissue to generate a feature map for osteoid-bone tissue, may operateon the group 210 to generate feature maps 230 (osteoid 232 and bone tissue 234). The feature map 230 may be used to generate an osteoid-bone mask 240.

[0051] The group 250 of (normalized) patches for osteoblasts generated at block 130 and the osteoid-bone mask 240 may be provided to models 260. The deep learning model 262, for example, trained on osteoblasts and osteoid-bone mask to generate a feature map for osteoblasts, may operate on the group 250 and the osteoid-bone mask 240 to generate a feature map. The computer vision model 264 may operate on the group 250 and the osteoid-bone mask 240 to generate a feature map. The feature maps generated by the models 262 and 264 may be compared to generate a fused feature map 270. For example, the feature map generated by the computer vision model 264 may be used to augment the feature map generated by the model 262 when those results are associated with a low confidence.

[0052] The operations performed at blocks 250-270 may be performed for other components using the component-specific model(s). For example, the operations performed at blocks 250-270 may additionally or alternatively performed using the feature maps for osteoclasts and model(s) trained on osteoclasts.

[0053] Example 310 shows an unannotated normalized bone biopsy image. The region of interest is labeled osteoid 312 for reference purposes. Example 320 shows the deep learning prediction binary mask 322 for the osteoid. Example 330 shows the region 332, adjacent to the osteoid, selected via the dilation morphological operation. Example 340 shows the osteoblasts 342 identified using a computer vision model. Example 350 the identified osteoblasts 342 superimposed on the normalized image 310. Example 360 shows a display of the osteoblasts 362 that are predicted by the deep learning model operated on the normalized image 310. Example 370 shows an example of augmentation of the osteoblasts 362 that are near the osteoid and display low confidence levels from the deep learning model predictions. In this example, the osteoblasts 362 with low confidence are augmented by the osteoblasts 342 identified by the computer vision model to generate the osteoblast region 372. Example 380 shows an example of the combining the predicted osteoblast region 372 with the normalized biopsy image 310 to identify the osteoblast region 372 on the normalized biopsy image 310. In this example, the osteoblast region 372 corresponds to the final set of osteoblasts detected by the component identification tool 120.

[0054] Figures 4A and 4B show examples of flow diagrams 400 and 450 illustrating methods of generating one or more parameters and / or one or more medical predictions using the generatedfeature maps according to embodiments. Operations described in diagrams 400 and 450 may be performed by a computing system, such as a computing system described below with respect to Figure 11.

[0055] Although the flow diagrams 400 and 450 may describe the operations as a sequential process, in various embodiments, some of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. An operation may have additional steps not shown in the figure. In some examples, some operations may be optional. Embodiments of the method may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the associated tasks may be stored in a computer-readable medium such as a storage medium.

[0056] As shown in Figure 4A, operations in flow diagram 400 may begin at block 410 when, such as accessing one or more digitized pathology image data of a bone sample of a patient from an imaging system and / or memory. In some examples, the digitized pathology image(s) may be of a bone sample stained with a Masson-Goldner type stain.

[0057] Next, at block 420, the image data may be processed to identify one or more components (e.g., osteoid and bone, osteoclasts, osteoblasts, and / or BMAT) within the pathology image data. For example, one or more feature maps for one or more bone components and / or a reconstructed image including the one or more feature maps may be generated. In some examples, the one or more bone components may be identified according to one or more blocks 422-426.

[0058] At block 422, the imaging data may be pre-processed to generate normalized patches from the digitized pathology image data. In some examples, one or more groups of patches may be generated for one or more regions (e.g., osteoid and bone, osteoclasts, osteoblasts, and / or BMAT).

[0059] At block 424, the respective model(s) may be operated upon the group of patches to generate the respective feature map of the respective component, thereby identifying the respective component within the image data. In some examples, the one or more models may include a deep learning model specific to each component (e.g., osteoid-bone, osteoblast, osteoclast, and BMAT). In some examples, the one or more models for one or more components may also include a computer vision model configured to augment the respective feature map. In some examples,osteoid-bone feature map, an osteoblast feature map, osteoclast feature map, and a BMAP map may be generated using the respective model(s).

[0060] For example, the group of patches may be operated upon by a first deep learning model trained for osteoid-bone to identify the osteoid-bone and a second deep learning model for BMAT to identify BMA. The first and second deep learning models may also be configured to generate the osteoid-bone and BMAT feature maps, respectively. By way of further examples, the group of patches may be operated upon by a (third) deep learning model and a computer vision model for osteoblasts to identify osteoblasts and generate the osteoblast feature map. In some examples, the osteoclast may be identified using a (fourth) deep learning model for osteoclasts to identify osteoclasts and generate the osteoclast feature map within the image data. In some examples, the one or more machine learning models for osteoclasts may include a computer vision model. By way of example, the osteoid-bone feature may be used to generate the osteoblast feature map and / or the osteoclast feature map, for example, as provided in the system 200. In some examples, the osteoblast feature map and optionally the osteoclast feature map may be generated using the osteoid-bone feature map and generated mask as provided in the system 200. In some examples, the osteoid-bone feature map may be further processed to generate an osteoid feature map and a bone feature map.

[0061] At block 426, in some examples, a reconstructed image may be generated using each feature map(s) and the digitized pathology image at block 410. In some examples, the feature map(s) and / or reconstructed image for the patient may optionally be stored in an electronic memory.

[0062] In some examples, at block 430, one or more features may be extracted from the one or more component feature maps. By way of example, the one or more features may include but is not limited to pixel intensity features, Haralick, and Fourier shape descriptors; spatial arrangement features; graphical features; cellular features, such as cell count, area / or volume of the component; among others, or any combination thereof. In some examples, one or more features extracted may be component and / or parameter and / or medical prediction specific.

[0063] In some examples, at block 440, one or more parameters and / or one or more medical predictions may be generated using the extracted features. In some examples, the one or more parameters and / or medical predictions may be generated by (i) operating machine learning model(s) onto one or more of the extracted features 442 and / or (ii) applying one or more algorithmsto one or more of the extracted features 444. In some examples, one or more of the parameters determined at blocks 442 or 444 may be used as an input for the analysis performed at the other block. For example, one or more machine learning models may operate on at least one parameter generated at block 444 to determine other parameter(s) and / or medical prediction(s).

[0064] In some examples, based upon the one or more parameters and / or the one or more medical predictions, a treatment may be provided to the patient.

[0065] In some examples, a report with the feature map(s) and / or the parameter(s) and / or medical prediction(s) may be generated and outputted (e.g., transmitted, displayed, etc.).

[0066] In some examples, Morphological Texture Analysis (MTA) may be performed at block 440 to generate an MTA index. In some examples, the MTA index may relate to bone turnover status (e.g., low, normal, or high). Figure 4B shows an examples of a flow diagram 450 illustrating a method of generating an MTA index using one or more extracted features (block 430) according to embodiments.

[0067] As shown in Figure 4B, operations in flow diagram 440 may begin at block 452 by accessing one or more of the extracted features for one or more components (block 430). In some examples, the one or more of the extracted features may include texture features for one of the components (e.g., bone, osteoid, and / or BMAT). In some examples, the one or more features may be extracted for one of the components that had been identified as the best compartment to predict turnover during training of one or more machine learning model. In some examples, the one or more texture features may be extracted from the BMAT feature map. For example, the one or more texture features may include but is not limited to intensity, Haralick, Fourier shape descriptors, among others, or any combination thereof.

[0068] Next, at block 454, one or more machine learning model(s), trained using BMAT texture features, may be operated onto one or more of the extracted features 452 to generate the bone turnover status / MTA index. In some examples, the one or more machine learning model(s) may include but is not limited to SVC.

[0069] In some examples, during training, to identify the component that is best to predict turnover, machine learning models may be trained using extracted texture features for bone, osteoid, and BMAT. A machine learning model may be trained for each component. The extracted texture features may be analyzed to determine the best compartment to predict turnover, for example using area under the receiver operating characteristic (AUC-ROC) curve.

[0070] Figure 5 illustrates a block diagram of some embodiments of a digital phenotyping system 500 configured to utilize (segmented) component feature maps generated from digitized pathology imaging data to generate one or more parameters and / or one or more medical predictions.

[0071] The system 500 can include a memory 110 configured to store one or more digitized pathology image data 112 including a plurality of digitized pathology images of a bone sample from a plurality of patients. In some examples, the plurality of digitized pathology images may be obtained by an image generation stage 502 and / or from an on-line database and / or archive containing digitized pathology images of bone samples generated at different sites (e.g., different hospitals, research laboratories, and / or the like). Prior to including digitized pathology images within the digitized pathology imaging data, the digitized pathology images may be subjected to a pre-processing stage 504. The pre-processing stage 504 may be configured to normalize image characteristics (e.g., color, brightness, contrast) so as to mitigate batch effects (e.g., differences between images obtained from different sites).

[0072] In some examples, the digitized pathology imaging data 112 may include a training set 112t and a validation set 112v. The training set 112t may include digitized pathology images of a bone sample from a first plurality of patients. The validation set 112v may include digitized pathology images of a bone sample from a second plurality of patients. In some examples, the plurality of patients may include but are not limited to patients with morphologic diversity, osteoporosis, osteomalacia, renal osteodystrophy, Paget’s disease of bone, metabolic disease (e.g., hyperparathyroidism, hypoparathyroidism, etc.), vitamin deficiency (e.g., vitamin D deficiency, etc.), hematologic disorders (e.g., multiple myeloma, aplastic anemia, etc.), other bone diseases / conditions, other diseases / conditions / deficiencies, among others, or any combination thereof.

[0073] In some examples (not shown), the memory 110 may also be configured to store ground truth data associated with the training set 112t. For example, the ground truth data may include but is not limited to: segmentation results (e.g., manual traced boundaries) and / or diagnostic report provided by an expert human pathologist, one or more parameters measured by histomorphometry software and / or human pathologist using histomorphometry software; other clinical data (e.g., medical diagnoses, treatment, etc.); among others; or any combination thereof.

[0074] In some examples, the memory 1 10 may also include multi-modal data for each patient. For example, the multi-modal data may include a training set for the first plurality of patients, and a validation set for the second plurality of patients. In some examples, the multi-modal data may include but is not limited to other imaging, serum biomarkers, clinical metadata, among others, or any combination thereof.

[0075] In some examples, the training set 112t may be used to train a downstream machine learning identification tool 140 to perform segmentations that identify each bone component. In some examples, the training set 112t may also be used to train a downstream machine learning stage 174 to generate one or more parameters related to morphological texture (e.g., size, shape, and spatial arrangement of bone tissue), such as bone turnover status.

[0076] The validation set 112v may be used to validate the results of the machine learning identification tool 140 to perform segmentations that identify cell components. The validation set 112v may also be used to validate the results of the machine learning stage 174 and the one or more parameters and / or one or more medical predictions 180 generated by the one or more machine learning models of the machine learning stage 174.

[0077] In some examples, the machine learning stage 174 may include a feature selection element 574 configured to select a set of features for each component to generate the one or more parameters and / or one or more medical predictions 180. For example, the feature extraction tool 172 may extract a first number of features for each component and then the feature selection element 574 may select a smaller second number of the features for each component that are most prognostic (e.g., that have a most significant impact in determining parameter(s) and / or medical prediction(s). In some examples, the second number of features may be used to train and validate the machine learning stage 174. In some examples, the feature selection element 574 may be configured to employ feature selection based on statistical filtering (e.g., correlation analysis, nonparametric significance testing, dimensionality reduction etc.) followed by one or more machine learning models of the machine learning stage 174 (e.g., support vector machines, random forest). In other examples, the feature selection may employ regression models, regularization-based methods such as LASSO regression, survival analysis-based approaches such as Cox proportional hazards models, where longitudinal outcome data (e.g., fracture events) are available, among others, or any combination thereof.

[0078] Figures 6A-E illustrate some exemplary images of components segmented from digitized pathology imaging data of a patient by the component identification tool 120. Figure 6A shows example 600 of exemplary images of segmented bone regions generated by the component identification tool 120. For example, the example 600 shows four examples 602-608 of biopsy images and the segmented bone region generated from the respective biopsy image using the machine learning identification tool 120.

[0079] Figure 6B shows example 610 of some exemplary images of segmented osteoid regions generated using the component identification tool 120. For example, the example 610 shows four examples 612-618 of biopsy images and the segmented osteoid region generated from the respective biopsy image using the machine learning identification tool 120.

[0080] Figure 6C shows example 620 of some exemplary images of segmented osteoclasts regions using the component identification tool 120. For example, the example 620 shows biopsy images 622 that have been manually annotated and images 624 showing the segmented osteoclasts generated from the respective biopsy image 622 using the machine learning identification tool 120.

[0081] Figure 6D shows example 630 of some exemplary images of segmented osteoblasts regions using the component identification tool 120. For example, the example 630 shows biopsy images 632 that have been manually annotated and images 634 showing the segmented osteoblasts generated from the respective biopsy image 632 using the machine learning identification tool 120.

[0082] Figure 6E shows example 640 of some exemplary images of segmented BMAT regions. For example, the example 640 shows biopsy images 642 that have been manually annotated and images 644 showing the segmented BMAT regions generated from the respective biopsy image 642 using the machine learning identification tool 140.

[0083] Figures 7A-7E shows examples of correlation plots of phenotyped regions and cellular structures from manual annotations and automated segmentation performed by the machine learning identification tool 120. Figure 7A shows a correction plot 700 of bone region for seventyseven test images between manual annotated biopsy images and segmented regions (e.g., Figure 6 A) generated by the machine learning identification tool 120. In this example, the Spearman correlation coefficients (p) between model and manual annotations for bone region were reported to be 0.99 for bone (p = .0000). Dice score for bone region was 0.96. The Spearman correlation between model and OsteoMeasure-derived areas was 0.92 for mineralized bone.

[0084] Figure 7B shows a correction plot 710 of osteoid region for seventy-seven test images between manual annotated biopsy images and segmented regions (e.g., Figure 6B) generated by the machine learning identification tool 120. In this example, the Spearman correlation coefficients (p) between model and manual annotations for osteoid were reported to be 0.99 (p = 2.89e'5:>). Dice score for osteoid region was 0.73. The Spearman correlation between model and OsteoMeasure-derived areas was 0.8 for osteoid region.

[0085] Figure 7C shows a correction plot 720 of osteoclast region for thirty-one test images between manual annotated biopsy images and segmented regions (e.g., Figure 6C) generated by the machine learning identification tool 120. In this example, the Spearman correlation coefficients (p) between model and manual annotations for osteoid were reported to be 0.60 (p = 9.82e'6). The Spearman correlation between model and OsteoMeasure-derived areas was 0.66 for osteoclast region. Bland-Altman analysis showed a mean difference or offset of <2 between manual and automated counts for osteoclasts, with most counts lying within the 95% confidence interval. Additionally, the inter-operator correlation for osteoclasts count was p =0.62.

[0086] Figure 7D shows a correction plot 730 of obsteoblast region for thirty test images between manual annotated biopsy images and segmented regions (e.g., Figure 6D) generated by the machine learning identification tool 120. In this example, the correlation coefficients (p) between model and manual annotations for osteoblasts were reported to be 0.69 (p = 1.06e'3). The Spearman correlation between model and OsteoMeasure-derived areas was 0.53 for osteoblast region. Bland-Altman analysis showed a mean difference or offset of -5 between manual and automated counts for osteoclasts, with most counts lying within the 95% confidence interval. Additionally, the inter-operator correlation for osteoblasts count was p =0.84.

[0087] Figure 7E shows a correction plot 740 of BMAT region for twenty-nine images between manual annotated biopsy images and segmented regions (e.g., Figure 6E) generated by the machine learning identification tool 120. In this example, the correlation coefficients (p) between model and manual annotations for BMATS were reported to be 0.95 (p = 5.99e'8). Dice score for osteoid region was 0.65.

[0088] Figures 8A-C show examples of representative images for high and low bone turnover and associated violin plots. Figure 8A shows representative images of BMAT regions 810 with heatmaps of pixel intensities 820 from biopsies labeled as “high” bone turnover. Figure 8B shows a representative images of BMAT regions 820 with heatmaps of pixel intensities 830 from biopsieslabeled as “low” bone turnover. Figure 8C shows violin plots 850-870 for MTA features showing values for “low” (x-axis label “0”) and “high” bone turnover images (x-axis label “1”). As shown in Figure 8C, high turnover images respectfully exhibited a high median value for the energy feature (Intensity .Hist. Energy (e.g., measure of the uniformity of intensity values, computed from the histogram of the image) as shown in plot 850), while low turnover images had high median values for intensity-based features (Intensity .Min (represents the minimum intensity value within the BMAT area in bone biopsies) as shown in plot 860) and Intensity. Skewness (measure of the asymmetry of the intensity distribution) as shown in plot 870).

[0089] In this example, computationally derived image texture and shape features demonstrated a performance of [AUC-ROC, Fl score] of [0.72,0.61] for bone, [0.78, 0.67] for osteoid and [0.86,0.74] for BMAT to classify images labelled as low and high turnover using diagnostic reports. Furthermore, three classifiers yielded performance metrics [AUC-ROC, Fl score] of [0.71,0.70] for a SVM classifier, [0.67, 0.61] for linear regression and [0.71, 0.71] for a random forest classifier (Figure SI 1) when trained on the top significant BMAT texture features (p < .002).

[0090] By leveraging the potential of deep-learning models, the disclosed systems and methods can generate approximately 20 feature maps in under a minute, providing support for clinical triage and decision-making. The feature maps can be generated without a requirement for manual annotation or additional staining such as TRAcP to identify osteo-clasts. The disclosed system and methods can use a sematic segmentation-based DL approach, using spatial information for training as opposed to naive pixel-level DL classifiers. For instance, incorporation of bone and osteoid masks into the osteoblast segmentation training data provided essential spatial cues for their detection.

[0091] The results showed excellent correlation between predictions from the disclosed tool 120 and the ground truth annotations (p > 0.9) for tissue region segmentation such as mineralized bone, osteoid and BMAT. The tool’s predictions accurately recognized folding artifacts in bone areas, marking them as non-bone regions (Figure 6A). In a subset of images (zz=22) with available OsteoMeasure histomorphometry values, alignment between the disclosed system-derived and OsteoMeasure-derived measurements has been. Spearman correlation coefficients of p >0.8 were obtained for bone and osteoid areas, and 0.66 and 0.53 for osteoclast count and osteoblast count respectively. The ability to automate precise measurements of mineralized bone and osteoid canfacilitate easier, quicker, and more accurate computation of parameters such as bone volume / tissue volume and osteoid volume / bone volume, using a 3D correction factor. The disclosed systems and methods can also enable BMAT segmentation in undecalcified bone biopsy images.

[0092] As shown in the examples of Figures 6A-7E, the disclosed systems and methods can compartmentalize five tissue components in undecalcified bone histologic images, generate accurate feature maps and provides quantitative parameters for static histomorphometry. The disclosed system and methods also can enable texture analysis to be performed for each histologic primitive and the relationship of non-traditional features, like BMAT to normal and pathological tissue states to be explored.

[0093] Figure 9 illustrates a block diagram of some embodiments of a digital phenotyping system 100 configured to generate one or more parameters and / or one or more medical predictions from radiology imaging data from a patient.

[0094] The system 910 can include a memory 110 configured to store radiology image data 912 from a patient. In some examples, the radiology image data 912 may include one or more digitized images of a bone sample taken from a patient. In some examples, the radiology image data may include but is not limited to Dual-Energy X-ray Absorptiometry (DEXA) data, High- Resolution Peripheral Quantitative Computed Tomography (HR-pQCT) data, Micro-Computed Tomography (microCT) data, other CT image data, magnetic resonance imaging (MRI) image data, other bone diagnostic radiology image data, or any combination thereof.

[0095] In some examples, the system 900 may include a feature extraction tool 920 configured to extract one or more features for one or more components using the radiology image data 912. By way of example, the one or more features may depend on the radiology image data. For example, for lumbar spine DXA scans, the one or more features may include areal BMD, T-score, Z-score, vertebral texture metrics, among others, or any combination thereof. For example, for HR-pQCT image data of the distal radius or tibia, the one or more features may include trabecular morphometries (e.g., trabecular number, thickness, separation), cortical metrics (e.g., cortical thickness, porosity), radiomics-based texture features (e.g., entropy, energy, anisotropy), among others, or any combination thereof.

[0096] In some examples, the feature extraction tool 920 may be implemented as computer code run on one or more processors (e.g., a CPU, a microcontroller, a GPU, and / or the like).

[0097] In some examples, the one or more features may be provided to a machine learning stage 930 including one or machine learning models configured to generate one or more parameters and / or one or more medical predictions 180. In some examples, the machine learning stage 930 may be trained to generate the one or more parameters and / or one or more medical predictions! 80 using test data and / or validation data that includes digitized pathology image data. Figure 10 shows an example of training the machine learning stage 930 according to some embodiments. In some examples, the one or more machine learning models may include but is not limited to CNN, transformers, radiomics pipelines support vector classifiers, Random Forrest, linear regression, among others, or a combination thereof. For example, the machine learning stage 174 can be trained to generate one or more parameters and / or one or more medical predictions using the extracted features.

[0098] Figure 10 illustrates a block diagram of some embodiments of a digital phenotyping system 1000 configured to generate one or more parameters and / or one or more medical predictions from the radiology image data from a patient.

[0099] The system 1000 can include a memory 110 configured to store one or more image data including a plurality of radiology images 914 of the anatomical site as the bone sample for the plurality of patients and plurality of digitized pathology images 912 of a bone sample from a plurality of patients. In some examples, the plurality of patients may include but are not limited to patients with morphologic diversity, osteoporosis, osteomalacia, renal osteodystrophy, Paget’s disease of bone, metabolic disease (e.g., hyperparathyroidism, hypoparathyroidism, etc.), vitamin deficiency (e.g., vitamin D deficiency, etc.), hematologic disorders (e.g., multiple myeloma, aplastic anemia, etc.), other bone diseases / conditions, other diseases / conditions / deficiencies, among others, or any combination thereof.

[0100] By way of example, the plurality of digitized pathology images and the (paired) plurality of radiology images may be obtained by an image generation stage 1002 and / or from an on-line database and / or archive containing the image data generated at different sites (e.g., different hospitals, research laboratories, and / or the like). Prior to including the image data within the image data 912 and / or 914, the imaging data may be subject to a pre-processing stage 1004. In this example, the pre-processing stage 1004 may be configured to normalize image characteristics so as to mitigate batch effects (e.g., differences between images obtained from different sites). In some examples, prior to the digitized pathology images within the digitized pathology imagingdata 914, the digitized pathology images may also be subjected to a segmentation, for example, using the system 100 in Figure 1. In some examples, one or more parameters and / or medical predictions may be also generated using the imaging data 914, for example, using the system 100 in Figure 1.

[0101] In some examples, the digitized pathology imaging data 914 may be segmented to identify one or more components, for example, using the system 100. In some examples, digitized pathology imaging data 914 may include a training set 914t and a validation set 914v of the segmented digitized pathology imaging data (e.g., component feature maps 150 and / or reconstructed image 160). The training set 914t may include segmented, digitized pathology images of a bone sample from a first plurality of patients. The validation set 914v may include segmented, digitized pathology images of a bone sample from a second plurality of patients.

[0102] In some examples, the radiology imaging data 912 may include a training set 912t and a validation set 912v. The training set 912t may include radiology image data of the same anatomical site as the bone sample from the first plurality of patients. The validation set 912v may include radiology image data of the same anatomical site as the bone sample from the second plurality of patients.

[0103] In some examples (not shown), the memory 110 may also be configured to store ground truth data associated with the training sets 912t and / or 914t. For example, the ground truth data may include the generated component feature maps and generated parameter(s) and / or one or more medical predictions generated for each training set of digitized pathology image data for a patient using the system 100; segmentation results (e.g., manual traced boundaries) and / or diagnostic report provided by an expert human pathologist; one or more parameters measured by histomorphometry software and / or human pathologist using histomorphometry software; other clinical data (e g., medical diagnoses, treatment, etc.); among others; or any combination thereof.

[0104] Each respective set of training data (912t and 914t) and validation data (912v and 914v) may be aligned, for example, by performing spatial and / or anatomical alignment, so that the radiology features correspond to the biopsy ground truth.

[0105] In some examples, the training sets 912t and 914t may be used to train a downstream machine learning stage 930 to generate one or more parameters and / or one or more medical predictions from radiology imaging data. The validation sets 912t and 914t may be used to validatethe results of the machine learning stage 930 and the one or more parameters and / or one or more medical predictions 180.

[0106] In some examples, the memory 110 may also include multi-modal data for each patient. For example, the multi-modal data may include a training set for the first plurality of patients, and a validation set for the second plurality of patients. In some examples, the multi-modal data may include but is not limited to other imaging, serum biomarkers, clinical metadata, among others, or any combination thereof.

[0107] In some examples, the machine learning stage 930 may include a feature selection element 1030 configured to select a set of radiology features for the radiology imaging data 912 to generate the one or more parameters and / or one or more medical predictions 180. For example, the feature extraction tool 1030 may extract a first number of features and then the feature selection element 1030 may select a smaller second number of the features. In some examples, the second number of features may be used to train and validate the machine learning stage 930.

[0108] In some examples, the set of radiology features may include but is not limited to trabecular thickness, cortical porosity, texture, among others, or any combination thereof. In some examples, the feature selection element 1030 may be configured to employ feature selection based on statistical filtering (e.g., correlation analysis, non-parametric significance testing, dimensionality reduction, etc.) followed by one or more machine learning models of the machine learning stage 930 (e.g., support vector machines, random forest). In alternative embodiments, feature selection may employ regression models, regularization-based methods such as LASSO regression, survival analysis-based approaches such as Cox proportional hazards models, where longitudinal outcome data (e.g., fracture events) are available, among others, or any combination thereof.

[0109] In some examples, the machine learning stage 930 may be configured to generate one or more parameters and / or one or more medical predictions 180. In some examples, the machine learning stage 930 may be trained by mapping the radiology image data 912t to the segmented components of the digitized pathology image data 914t for the same patient, using a classifier, such as supervised regression.

[0110] Fig. 11 illustrates some embodiments of a block diagram of a system 1100 configured to generate one or more parameters and / or one or more medical predictions associated with bone health from digitized pathology imaging data and / or radiology imaging data from a patient.

[0111] The system 1100 comprises a digital phenotyping system 1110. The system 1110 may be coupled to an image generation stage 1106, which is configured to generate a digitized pathology image of bone sample collected from a patient 1102 and / or a radiology image of a bone site from a patient 1102.

[0112] The system 1110 may include a processor 1120 and a memory 1130. The processor 1120 can, in various embodiments, comprise circuitry such as, but not limited to, one or more single-core or multi-core processors. The processor 1120 can include any combination of general- purpose processors and dedicated processors (e.g., graphics processors, application processors, etc.). The processor(s) 1120 can be coupled with and / or can include memory (e.g., memory 1130) or storage and can be configured to execute instructions stored in the memory 1130 or storage to enable various apparatus, applications, or operating systems to perform operations and / or methods discussed herein. In some examples, the memory 1130 may include electronic memory (e.g., solid state memory, SRAM (static random-access memory), DRAM (dynamic random-access memory), and / or the like).

[0113] The memory 1130 can be configured to store digitized pathology imaging data 1132 including digitized pathology images. The digitized pathology images may include digitized bone specimen images having a plurality of pixels, each pixel having an associated intensity. In some additional embodiments, the digitized pathology images 1132 may be stored in the memory 1130 as one or more training sets of digitized images for training a classifier and / or one or more test sets (e.g., validation sets) of digitized images.

[0114] The system 1110 may also include an input / output (I / O) interface 1122 (e.g., associated with one or more I / O devices), a display 1124, and an interface 1140 that connects the processor 1120, the memory 1130, and the I / O interface 1140. The I / O interface 1120 can be configured to transfer data between the memory 1130, the processor 1120, and external devices, for example, the image generation stage 303.

[0115] In some examples, the system 1110 may further include one or more circuits 1150 that include one or more of a machine learning identification circuit 1152, a feature extraction circuit 1154, and a machine learning circuit 1156. In some examples, the one or more circuits 1150 may operate according to machine learning algorithms stored in the memory 1130.

[0116] In some examples, the machine learning identification circuit 1152 may be configured to segment the plurality of digitized pathology images 1132 to generate component feature maps1134 that identify each component within the digitized pathology images. In such examples, the feature extraction circuit 1154 may be configured to extract a plurality of features 1136 from the one or more component feature maps 1134. The machine learning circuit 1154 may be configured to utilize the plurality of features 1136 to generate one or more parameters and / or one or more medical predictions 1138.

[0117] In some examples, the memory 1130 may be configured to store radiology imaging data 1140. The radiology imaging data 1140 may include radiology images of a region of bone of interest having a plurality of pixels, each pixel having an associated intensity. In some additional embodiments, the radiology imaging data 1140 may be stored in the memory 1130 as one or more training sets of radiology imaging for training a classifier and / or one or more test sets (e.g., validation sets) of radiology imaging.

[0118] In such examples, the feature extraction circuit 1154 may be configured to extract a plurality of features 1136 from the radiology imaging data 1140. The machine learning circuit 1154 may be configured to utilize the plurality of features 1136 to generate one or more parameters and / or one or more medical predictions 1138.

[0119] In some examples, the display 1110 is configured to output or display the one or more parameters and / or one or more medical predictions 1138 generated by the digital phenotyping system 1110.

[0120] In some examples, the disclosure may relate to methods, systems, and computer- readable media configured configured to identify one or more components from digitized pathology imaging data to, optionally, generate one or more component feature maps. In some examples, the methods and / or systems may be configured to generate one or more parameters and / or one or more medical predictions using the component feature maps.

[0121] In some examples, the disclosure may relate to a computer-implemented method. The method may include operating upon digitized pathology image data of a bone sample from a patient with one or more models to generate one or more feature maps for one or more components. The one or models may include one or more deep learning models, one or more computer vision models, among others, or any combination thereof. The one or more components may include osteoclast, osteoblast, bone marrow bone marrow adipose tissue (BMAT), mineralized bone, and / or osteoid. In some examples, the patient may have or suspect to have a bone disorder, such as renal osteodystrophy, osteoporosis, etc.

[0122] In some examples, the one or more models may include a first set of one or more models trained to generate an osteoclast feature map; a second set of one or more models trained to generate an osteoblast feature map; a third set of one or more models trained to generate a BMAT feature map; and a fourth set of one or more models trained to generate an osteoid-bone feature map.

[0123] In some examples, the method may further include extracting one or more features for one or more of the feature maps and generating one or more parameters and / or one or more medical predictions using the one or more extracted features.

[0124] In some examples, the generating may include: operating upon one or more of one or more features with one or more models to generate the one or more parameters and / or the one or more medical predictions; and / or processing the one or more features using one or more algorithms to generate the one or more parameters and / or the one or more medical predictions.

[0125] In some examples, the one or more extracted features may include one or more texture features. The one or more parameters may include bone turnover status. In some examples, the one or more texture features may be extracted from the BMAT feature map. The bone turnover status may be generated by operating a machine learning model on the one or more extracted texture features for the BMAT. In some examples, the one or more models may classify the bone turnover status as low, normal, or high.

[0126] In some examples, the one or more parameters may include surface area, cell count, cell interaction -related parameters, among others, or any combination thereof. In some examples, the one or more medical predictions may include fracture loss, among others, or any combination thereof.

[0127] In some examples, the generating may include operating upon the component feature maps, extracted features, and / or a first set of one or more parameters and / or one or more medical predictions with one or more deep learning models to generate a second set of one or more parameters and / or one or more medical predictions.

[0128] In some examples, the disclosure may relate to methods, systems, and computer- readable media configured generate one or more parameters and / or one or more medical predictions from radiology imaging data of a patient. For example, the radiology imaging data may include but is not limited to Dual-Energy X-ray Absorptiometry (DEXA) data, High-ResolutionPeripheral Quantitative Computed Tomography (HR-pQCT) data, other radiological image data, among others, or any combination thereof.

[0129] In some examples, the disclosure may relate to a method that includes extracting one or more radiology features from the radiology imaging data. In some examples, the method may include operating upon one or more of the one or more radiology features with one or more models to generate the one or more parameters and / or the one or more medical predictions; and / or processing the one or more radiological features using one or more algorithms to generate the one or more parameters and / or the one or more medical predictions.

[0130] In some examples, the one or more models may be trained using the digitized pathology imaging data and the one or more parameters and / or one or more medical predictions generated using the digitized pathology imaging data.

[0131] It will be appreciated that the disclosed methods and / or block diagrams may be implemented as computer-executable instructions, in some examples. Thus, in one example, a computer-readable storage device (e.g., a non-transitory computer - readable medium) may store computer executable instructions that if executed by a machine (e.g., computer, processor) cause the machine to perform the disclosed methods and / or block diagrams. While executable instructions associated with the disclosed methods and / or block diagrams are described as being stored on a computer-readable storage device, it is to be appreciated that executable instructions associated with other example disclosed methods and / or block diagrams described or claimed herein may also be stored on a computer-readable storage device.

[0132] Various embodiments may omit, substitute, or add various procedures or components as appropriate. For instance, in alternative configurations, the methods described may be performed in an order different from that described, and / or various stages may be added, omitted, and / or combined. Also, features described with respect to certain embodiments may be combined in various other embodiments. Different aspects and elements of the embodiments may be combined in a similar manner. Also, technology evolves and, thus, many of the elements are examples that do not limit the scope of the disclosure to those specific examples.

[0133] The foregoing method descriptions and the process flow diagrams are provided merely as illustrative examples and are not intended to require or imply that the operations of various embodiments must be performed in the order presented. As will be appreciated by one of skill in the art the order of operations in the foregoing embodiments may be performed in any order. Wordssuch as “thereafter,” “then,” “next,” etc. are not intended to limit the order of the operations; these words are simply used to guide the reader through the description of the methods. Further, any reference to claim elements in the singular, for example, using the articles “a,” “an” or “the” is not to be construed as limiting the element to the singular.

[0134] References to “one embodiment”, “an embodiment”, “one example”, and “an example” indicate that the embodiment(s) or example(s) so described may include a particular feature, structure, characteristic, property, element, or limitation, but that not every embodiment or example necessarily includes that particular feature, structure, characteristic, property, element or limitation. Furthermore, repeated use of the phrase “in one embodiment” does not necessarily refer to the same embodiment, though it may.

[0135] To the extent that the term “includes” or “including” is employed in the detailed description or the claims, it is intended to be inclusive in a manner similar to the term “comprising” as that term is interpreted when employed as a transitional word in a claim.

[0136] Throughout this specification and the claims that follow, unless the context requires otherwise, the words 'comprise' and 'include' and variations such as 'comprising' and 'including' will be understood to be terms of inclusion and not exclusion. For example, when such terms are used to refer to a stated integer or group of integers, such terms do not imply the exclusion of any other integer or group of integers.

[0137] Terms, “and” and “or” as used herein, may include a variety of meanings that also is expected to depend at least in part upon the context in which such terms are used. Typically, “or” if used to associate a list, such as A, B, or C, is intended to mean A, B, and C, here used in the inclusive sense, as well as A, B, or C, here used in the exclusive sense. In addition, the term “one or more” as used herein may be used to describe any feature, structure, or characteristic in the singular or may be used to describe some combination of features, structures, or characteristics. However, it should be noted that this is merely an illustrative example and claimed subject matter is not limited to this example. Furthermore, the term “at least one of’ if used to associate a list, such as A, B, or C, can be interpreted to mean any combination of A, B, and / or C, such as A, AB, AC, BC, AA, ABC, AAB, AABBCCC, and the like.

[0138] It will be appreciated that the disclosed methods and / or block diagrams may be implemented as computer-executable instructions, in some examples. Thus, in one example, a computer-readable storage device (e.g., a non-transitory computer-readable medium) may storecomputer executable instructions that if executed by a machine (e.g., computer, processor) cause the machine to perform the disclosed methods and / or block diagrams. While executable instructions associated with the disclosed methods and / or block diagrams are described as being stored on a computer-readable storage device, it is to be appreciated that executable instructions associated with other example disclosed methods and / or block diagrams described or claimed herein may also be stored on a computer-readable storage device.

[0139] Examples herein can include subject matter such as an apparatus, including a digital whole slide scanner, a light microscopy system, a fluorescence microscopy system, a personalized medicine system, a CADx system, a processor, a system, circuitry, a method, means for performing acts, steps, or blocks of the method, at least one machine-readable medium including executable instructions that, when performed by a machine (e.g., a processor with memory, an applicationspecific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like) cause the machine to perform acts of the method or of an apparatus or system, according to embodiments and examples described.

[0140] Various illustrative logical blocks, modules, circuits, and algorithm operations described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and operations have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such embodiment decisions should not be interpreted as causing a departure from the scope of the claims.

[0141] The hardware used to implement various illustrative logics, logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general -purpose processor may be a microprocessor, but, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as acombination of computing systems, (e. ., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some operations or methods may be performed by circuitry that is specific to a given function.

[0142] In one or more example embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer readable medium or non-transitory processor-readable medium. The operations of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a non-transitory computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable storage media may be any storage media that may be accessed by a computer or a processor. By way of example but not limitation, such non-transitory computer-readable or processor-readable media may include RAM, ROM, EEPROM, FLASH memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of non- transitory computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0143] Those skilled in the art will appreciate that information and signals used to communicate the messages described herein may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0144] Further, while certain embodiments have been described using a particular combination of hardware and software, it should be recognized that other combinations of hardware andsoftware are also possible. Certain embodiments may be implemented only in hardware, or only in software, or using combinations thereof. In one example, software may be implemented with a computer program product containing computer program code or instructions executable by one or more processors for performing any or all of the steps, operations, or processes described in this disclosure, where the computer program may be stored on a non-transitory computer readable medium. The various processes described herein can be implemented on the same processor or different processors in any combination.

[0145] Where devices, systems, components or modules are described as being configured to perform certain operations or functions, such configuration can be accomplished, for example, by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation such as by executing computer instructions or code, or processors or cores programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes can communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communications, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0146] “ Circuit”, as used herein, includes but is not limited to hardware, firmware, software in execution on a machine, or combinations of each to perform a function(s) or an action(s), or to cause a function or action from another logic, method, or system. A circuit may include a software controlled microprocessor, a discrete logic (e.g., ASIC), an analog circuit, a digital circuit, a programmed logic device, a memory device containing instructions, and other physical devices. A circuit may include one or more gates, combinations of gates, or other circuit components. Where multiple logical circuits are described, it may be possible to incorporate the multiple logical circuits into one physical circuit. Similarly, where a single logical circuit is described, it may be possible to distribute that single logical circuit between multiple physical circuits.While the disclosure has been described in detail with reference to exemplary embodiments, those skilled in the art will appreciate that various modifications and substitutions may be made thereto without departing from the spirit and scope of the disclosure as set forth in the appended claims. For example, elements and / or features of different exemplary embodiments may be combined with each other and / or substituted for each other within the scope of this disclosure and appended claims..

Claims

CLAIMSWhat is claimed:1 . A computer-implemented method, comprising: operating upon digitized pathology image data of a bone sample from a patient with one or more models to generate one or more feature maps for one or more components; wherein the one or models includes one or more deep learning models, one or more computer vision models, among others, or any combination thereof; and wherein the one or more components includes osteoclast, osteoblast, bone marrow bone marrow adipose tissue (BMAT), mineralized bone, and / or osteoid.

2. The method according to claim 1 , wherein the one or more models include a first set of one or more models trained to generate an osteoclast feature map; a second set of one or more models trained to generate an osteoblast feature map; a third set of one or more models trained to generate a BMAT feature map; and a fourth set of one or more models trained to generate a osteoid-bone feature map.

3. The method according to claims 2 or 3, further comprising: generating an osteoid-bone mask using the osteoid-bone feature map.

4. The method according to claim 3, wherein the osteoblast feature map and / or the osteoclast feature map are generated by operating a deep learning model of the respective set of one or more models and the osteoid-bone mask with a computer fusion model.

5. The method according to claim 4, further comprising: generating a fused feature map for the osteoblast and / or osteoclast using the respective feature map generated by the respective computer fusion model and the respective feature map generated by the respective deep learning model based on confidence level of the respective feature map generated by the respective deep learning model.

6. The method according to any of claims 1-5, further comprising:generating a reconstructed image using the digitized pathology image data and the one or more feature maps.

7. The method according to any of claims 1-6, further comprising: extracting one or more features for one or more of the feature maps; and generating one or more parameters and / or one or more medical predictions using the one or more extracted features.

8. The method according to claim 7, wherein the generating includes: operating upon one or more of one or more features with one or more models to generate the one or more parameters and / or the one or more medical predictions; and / or processing the one or more features using one or more algorithms to generate the one or more parameters and / or the one or more medical predictions.

9. The method according to claims 7 or 8, wherein: the one or more extracted features includes one or more texture features; and the one or more parameters includes bone turnover status.

10. The method according to claim 9, wherein: the one or more texture features is extracted from the BMAT feature map; and the bone turnover status is generated by operating a machine learning model on the one or more extracted texture features for the BMAT.

11. The method according to claims 9 or 10, wherein the one or more models classifies the bone turnover status as low, normal, or high.

12. The method according to claim 11, wherein the generating includes: operating upon the component feature maps, extracted features, and / or a first set of one or more parameters and / or one or more medical predictions with one or more deep learning models to generate a second set of one or more parameters and / or one or more medical predications.

13. The method according to any of claims 1-12, wherein: the digitized pathology image is an image of a bone sample stained only with a Masson- Golder type stain; and the one or more parameters include one or more static parameters and / or one or more dynamic parameters.

14. A non-transitory computer-readable medium storing computer-executable instructions that, when executed, cause a processor to perform operations, comprising: the method as recited in any of claims 1-13.

15. An apparatus, comprising: one or more processors; and a memory containing instructions which when executed on the one or more processors, cause the one or more data processors to perform the method of any of claims 1 to 13.

Images