Machine learning model compression for digital pathology analysis

Abstract

Techniques for compressing machine learning (ML) models for digital pathology analysis are described. More specifically, the techniques are applicable to ML models having skip connections, such as skip connections within residual units of a ML model and / or skip connections between an encoder branch and a decoder branch of a ML model. Initially, a ML model is trained. Subsequent to training the ML model, a skip connection within the ML model is identified (e.g., by identifying interconnections between various layers within the model). Subsequently, one or more filters of the ML model are pruned, while maintaining dimensional integrity of the skip connection within the ML model.

Description

MACHINE LEARNING MODEL COMPRESSION FOR DIGITAL PATHOLOGYANALYSISCROSS-REFERENCE OF RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 549,932, filed on February 5, 2024, titled “MACHINE LEARNING MODEL COMPRESSION FOR DIGITAL PATHOLOGY ANALYSIS,” which is incorporated by reference in its entirety for all purposes.BACKGROUND

[0002] Digital pathology is often used to guide a medical professional in diagnosis, prognosis and treatment assessment of a patient's disease. Stained biological samples (e.g., stained tissue slices) are mounted on glass microscope slides and are digitally imaged. A human pathologist or computational algorithm analyzes individual sections (referred to as patches) and characterizes cells in each patch. For example, it may be determined, for each patch, whether tumor cells are depicted in the patch, a quantity of tumor cells, a spatial distribution of tumor cells, etc. Such results can inform a diagnosis, treatment selection, treatment efficacy assessment, and so on.

[0003] Technological advances have enabled the digitization of histopathology slides into high-resolution images, providing opportunities to develop computer vision tools for a wide range of clinical applications. High-resolution, digital images of microscope slides make it possible to use computer-based analysis of slides. For example, machine learning models have shown promise as a tool in medical diagnostic applications and in predicting treatment outcomes.

[0004] In recent years, Artificial Intelligence has made significant progress and has achieved state-of-the-result in a variety of tasks from different domains / modalities. Much of this success comes from the availability of large training dataset and ever-increasing size of the model(s) employed. Massive models such as vision models come at an increase in computational costs and often require the use of multiple high-end GPUs. Hospitals and the healthcare industry may lack the infrastructure required to fully leverage the increasingly performant yet computationally-costly models due to these increased computational costs.SUMMARY

[0005] In various embodiments, a non-transitory computer-readable medium includes instructions that when executed by one or more processors, cause the one or more processors to perform operations including: training a machine learning (ML) model; subsequent to training the ML model, identifying a skip connection within the ML model; and pruning one or more fdters of the ML model, while maintaining dimensional integrity of the skip connection within the ML model.

[0006] In an example, pruning the one or more filters of the ML model while maintaining dimensional integrity of the skip connection comprises: configuring a first layer of the ML model to have a first output including a first number of channels, wherein a second layer and a third layer are in series with the first layer of the ML model; pruning the second layer of the ML model, such that second layer has a second output including a second number of channels, wherein the second number of channels is less than the first number of channels; pruning the third layer of the ML model, such that third layer has a third output including the first number of channels; and summing or concatenating (i) the first output having the first number of channels and (ii) the third output having the first number of channels. In an example, the operations further include: prior to summing or concatenating the first output and the third output, processing each of the first output and the third output, wherein processing the first output comprises prior to summing or concatenating, performing, without altering a dimensionality of a number of channels of the first output, one or more of (i) an activation operation, (ii) a batch normalization operation, and / or (iii) a convolution operation on the first output; and wherein processing the third output comprises prior to summing or concatenating, performing, without altering a dimensionality of a number of channels of the third output, one or more of (i) another activation operation, (ii) another batch normalization operation, and / or (iii) another convolution operation on the third output.

[0007] In an example, the first output by the first layer has N number of channels, where N is a positive integer that is the first number; the second output by the second layer has N / a number of channels, where N / a is the second number, and wherein “a” is a positive integer pruning ratio implemented by the second layer; and the third output by the third layer has the N number of channels. In an example, prior to pruning the ML model, the second output by the second layer has the N number of channels. In an example, the operations further include: prior to summing orconcatenating the first output and the third output, processing the third output by a fourth layer, wherein the fourth layer does not change a dimensionality of the first output, such that the output of the fourth layer includes the first number of channels, wherein the first layer, the second layer, the third layer, and the fourth layer implement corresponding convolution operations.

[0008] In an example, the operations further include: subsequent to summing or concatenating the first output and the third output, providing a summed or concatenated output to a fourth layer of the ML model. In an example, the skip connecting is from the first layer to a layer performing the summing or concatenating; and maintaining the dimensional integrity of the skip connection within the ML model comprising ensuring that the first output and the third output being summed or concatenated have the same first number of channels. In an example, pruning the one or more filters of the ML model comprises: configuring a fourth layer of the ML model to receive a combined output, wherein the combined output is the sum or concatenation of the first output and the third output, and wherein the combined output has a third number of channels; pruning the fourth layer such that the fourth layer has a fourth output including a fourth number of channels that is less than the third number of channels, wherein a fifth layer is in series with the fourth layer of the ML model; pruning the fifth layer of the ML model, such that fifth layer has a fifth output including the third number of channels; and summing or concatenating (i) the combined output having the third number of channels and (ii) the fifth output having the third number of channels.

[0009] In an example, the operations further include: identifying the second layer of the ML model as a first candidate for pruning, so as to reduce a size of the ML model; identifying the third layer of the ML model as a second candidate for pruning, so as to maintain the dimensional integrity of the skip connection within the ML model; calculating a first pruning ratio for the second layer; calculating a second pruning ratio for the third layer, the second pruning ratio is calculated based on the first pruning ratio, such that the third layer reverses the channel reduction achieved by the second layer; and pruning one or more filters of the ML model comprises pruning the second layer and the third layer in accordance with the first pruning ration and the second pruning ration, respectively. In an example, summing or concatenating the first output and the third output comprises summing, in a summer, the first output and the third output; the second layer, the third layer, and the summer are within a residual unit of the ML model; and the first output from the first layer forms an input to the residual unit. In an example, summing or concatenating the first output and the third output comprises concatenating, in a concatenation layer, the first outputand the third output; the ML model is a U-Net model; the first layer, the second layer, and the third layer are within an encoder of the U-Net model; and the concatenation layer is within a decoder of the U-Net model.

[0010] In an example, the operations further include: subsequent to pruning the one or more filters of the ML model, fine tuning the ML model by at least in part retraining the ML model. In an example, the ML model is usable to process digital pathology images. In an example, the ML model has one of a HoVer-Net architecture, a ResNet architecture, or a U-Net architecture.

[0011] In various embodiments, a method comprises: training a machine learning (ML) model; subsequent to training the ML model, identifying a skip connection within the ML model; and pruning one or more filters of the ML model, while maintaining dimensional integrity of the skip connection within the ML model. In an example, pruning the one or more filters of the ML model while maintaining dimensional integrity of the skip connection comprises: configuring a first layer of the ML model to have a first output including a first number of channels, wherein a second layer and a third layer are in series with the first layer of the ML model; pruning the second layer of the ML model, such that second layer has a second output including a second number of channels, wherein the second number of channels is less than the first number of channels; pruning the third layer of the ML model, such that third layer has a third output including the first number of channels; and summing or concatenating (i) the first output having the first number of channels and (ii) the third output having the first number of channels. In an example, the method further comprises: using the pruned ML model to process digital pathology images.

[0012] In various embodiments, a system comprises: one or more data processors; and a non- transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform a set of actions including: training a machine learning (ML) model; subsequent to training the ML model, identifying a skip connection within the ML model; and pruning one or more filters of the ML model, while maintaining dimensional integrity of the skip connection within the ML model. In an example, the set of actions further comprises: configuring a first layer of the ML model to have a first output including a first number of channels, wherein a second layer and a third layer are in series with the first layer of the ML model; pruning the second layer of the ML model, such that second layer has a second output including a second number of channels, wherein the secondnumber of channels is less than the first number of channels; pruning the third layer of the ML model, such that third layer has a third output including the first number of channels; and summing or concatenating (i) the first output having the first number of channels and (ii) the third output having the first number of channels.

[0013] In some embodiments, a system is provided that includes one or more data processors and a non-transitoiy computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0014] In some embodiments, a computer-program product tangibly embodied in a non- transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform part or all of one or more methods or processes disclosed herein.

[0015] In some embodiments, a system is provided that includes one or more means to perform part or all of one or more methods or processes disclosed herein.

[0016] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention as claimed has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0018] The present disclosure is described in conjunction with the appended figures:

[0019] Fig. 1 illustrates a machine learning (ML) model depicting a HoVer-Net architecture, where the ML model is compressed using techniques described herein, in accordance with an embodiment of the present disclosure.

[0020] Fig. 2 illustrates a section of a ML model, where the section of the ML model comprises a series of residual units, where the section of the ML model is compressed using techniques described herein.

[0021] Fig. 3 illustrates a ML model comprising a U-Net architecture, where the ML model is compressed using techniques described herein.

[0022] Fig. 4 illustrates a table depicting example input and output channel dimensions of various layers of an example ML model, wherein various layers within each residual block or convolutional block are pruned independently to ensure dimensional integrity of the skip connections within the ML model.

[0023] Fig. 5 illustrates a table depicting another example input and output channel dimensions of various layers of a ML model, where pruning ratios of different residual blocks or convolutional blocks are set in unison, to ensure dimensional integrity of the skip connections within the ML model.

[0024] Fig. 6 illustrates example pseudo-codes for specifying model pruning settings and model components.

[0025] Fig. 7 illustrates an example flow chart of a computer-implemented method for pruning a ML model including one or more skip connections.

[0026] Fig. 8 illustrates example whole slide images with pen-mark artifacts.

[0027] Fig. 9 illustrates a graph depicting validation loss, e.g., when a LI pruner is used to uniformly prune one or more layers of HoverNet with only a segmentation branch.

[0028] Fig. 10 illustrates a graph depicting a reduction of memory footprint as a result of pruning.

[0029] Fig. 11A illustrates a graph depicting an effect of pruning on segmentation validation loss for the HoverNet model.

[0030] Fig. 11B illustrates a graph depicting an effect of pruning on classification validation loss for the HoverNet model.

[0031] Fig. 12 illustrates a graph depicting an effect of pruning on latency for the HoverNet model.

[0032] Fig. 13 illustrates a graph depicting a comparison between an iterative magnitude pruning versus a one shot pruning for a segmentation branch.

[0033] Fig. 14 illustrates a graph depicting a comparison between an iterative magnitude pruning versus an one shot pruning for a classification branch.

[0034] Fig. 15A illustrates a graph depicting classification model performance (e.g., Fl) at various pruning levels.

[0035] Fig. 15B illustrates a graph depicting classification model performance (e.g., sensitivity) at various pruning levels.

[0036] Fig. 15C illustrates a graph depicting classification model performance (e.g., specificity) at various pruning levels.

[0037] Fig. 16 illustrates a graph depicting a classification model parameter (e.g., memory footprint) reduction at various pruning levels.

[0038] Fig. 17 illustrates a graph depicting a reduction in latency of classification inference at various pruning levels.

[0039] Fig. 18 is a block diagram illustrating an example overview of a system for processing digital pathology images, where ML models are used for such processing of digital pathology images, and where the ML models may be pruned using techniques described herein.

[0040] Fig. 19 shows an exemplary network of a digital pathology image generation system from Fig. 18.DETAILED DESCRIPTION

[0041] Model complexity and the required computing power have become an issue across many machine learning applications. For example, machine learning (ML) models have shown promise as a tool in medical diagnostic applications and in predicting treatment outcomes, and used in processing digital pathology images, and such ML models are massive, resulting in an increase in computational costs and use of multiple high-end Graphic Processing Unit(s) (GPUs). To reduce the model size and thus reduce the computational costs of the ML models, model compression techniques may be used.

[0042] Model pruning is a type of model compressing technique. Tn model pruning, a ML model is trained initially. Subsequently, unnecessary or relatively unimportant model parameters are removed from the trained ML model, thereby decreasing a size of the ML model, which in turn reduces cost of storage and / or execution of the ML model.

[0043] However, many ML models used in digital pathology image processing applications have shortcuts or skip connections, which pose additional challenges in applying model pruning techniques to such ML models. Skip connections are used in deep learning architectures, and involve bypassing one or more layers by adding shortcut or skip connections. For example, in a ML model, a layer may feed into the next layer, and also feed into another layer few hops away (such as about 2-3 hops away). Thus, in a skip connection, output of a first layer is fed to a downstream second layer. Furthermore, a third layer (which is downstream of the first and second layers) receives (i) an output of the second layer and also (i) the output of the first layer. Thus, the output of the first layer is transmitted to the third layer (i) via the second layer, and (ii) directly to the third layer by bypassing or skipping the second layer. For example, a summer may add the outputs of the second layer and the first layer, and the summed output may be provided to the third layer. For example, a residual unit is a building block of many ML models, where a residual unit may include skip connections, in which a direct connection between the first layer and the third layer may skip one or more intermediate layers (e.g., including the above-described second layer).

[0044] In addition to residual units, skip connections are also observed in other types of ML models used in digital pathology image processing applications. For example, a HoVer-Net architecture or a U-Net architecture may include an encoder branch and at least one decoder branch. There may be several layers within each of the encoder branch and the decoder branch. A skip connection (also referred to herein as a shortcut connection) directly provides a connection from a layer of the encoder to a corresponding layer of the decoder. For example, as will be described below in further detail, at the decoder, a concatenation operation is performed, where an output from a corresponding layer of the encoder and an output from another corresponding layer of the decoder is concatenated, and the concatenated feature maps are then processed at the decoder. In this example, the output from the layer of the encoder to the concatenation layer is the skip or shortcut connection. Multiple such skip connections may exist between the encoder and the decoder.

[0045] Thus, some ML models used in digital pathology image processing applications may include one or both of the following two types of skip connections - (i) skip connections within individual residual units, and (ii) skip connections from the encoder to the decoder. However, applying model pruning to some ML models, such as those ML models in which skip connections are included, poses additional challenges. For example, it may be challenging to prune an ML model including a skip connection (such as a model including a residual unit, and / or a model including a skip connection between an encoder and a decoder path), e.g., because the skip connections in a layer impact model dimensions downstream, thereby possibly making dimensions for skip connections incompatible. For example, because in a skip connection two inputs are to be summed or concatenated, any incompatible dimensions due to the pruning may render such a model non-operational. Thus, while implementing pruning of a ML model including one or more skip connections, additional mitigating strategies has to be implemented. For example, pruning strategies undertaken has to maintain dimensional integrity of the skip connections within the ML model, as described below in detail.

[0046] For example, as will be described below in further detail, a layer of a ML model may be pruned actively or passively. An “active” pruning refers to pruning output channel dimension of a layer of a ML model, so as to compress the ML model. For convolutional layers, the output channel dimension may be changed (such as reduced), e.g., by changing (such as reducing) a number of filters used in the convolution layer. However, an actively pruned layer impacts the pruning of a subsequent layer (such as an immediate subsequent layer), and may cause incompatibility in the summer or concatenator associated with a skip connection.

[0047] Thus, to allow for active pruning of a layer and yet maintain dimensional integrity of the skip connection within the ML model, one or more downstream layers have to be passively pruned. In a passive pruning, an output channel dimension of a convolution layer “has” to be pruned, to accommodate the active pruning of an inter-connected previous convolution layer and to maintain dimensional integrity of skip connections.

[0048] For example, assume a residual unit having at least three convolution layers (a first, a second, and a third convolution layer in series), and one or more batch normalization layers and one or more rectified linear units (ReLU) layers. Note that the batch normalization layers and the ReLU activation layers do not affect dimensionality, and hence, in this disclosure, these layersmay be ignored when discussing channel dimensionality of inputs and outputs of convolution layers. The residual unit has an input, which is received by the first convolution layer from an upstream layer. Output of the first convolution layer is provided to the second convolution layer, and output of the second convolution layer is provided to the third convolution layer. A summer at the end of the third convolution layer receives (i) the input to the residual unit (via a skip connection) and (ii) the output of the third convolution layer. Thus, to maintain dimensional integrity of the skip connection, the channel dimensions of the following should match: (i) the input to the residual unit and (ii) the output of the third convolution layer. Now assume (merely as an example and without limiting the scope of this disclosure) that prior to pruning the ML model (e.g., in the unpruned and trained ML model), each of the first, second, and third convolution layers, as well as the input to the residual unit, have an output channel dimension of 64. Also assume that due to upstream pruning (e.g., upstream of the residual unit described here), in the pruned ML model, the input to the residual unit now has a channel dimension of 64 / a, where “a” is a pruning ratio and is a positive integer (e.g., can be 1, 2, 4, 8, 16, or the like). Further assume that the first convolution layer of the residual unit is pruned (such as an “active” pruning is performed on the first convolution layer), such that the output of the pruned first convolution layer has now a channel dimension of 64 / b, where “b” is a pruning ratio different from “a” and is a positive integer. If this channel dimension is left unchanged in the second and third layer, there would be channel dimension incompatibility in the summer. Accordingly, to maintain channel dimensional integrity of the skip connection at the summer, the second convolution layer (or the third convolution layer) is also pruned (such as a “passive” pruning is performed on the second convolution layer), such that the output of the second convolution layer is back to 64 / a. In an example, the third convolution layer may not alter a dimensionality, and hence, the third convolution layer also outputs feature maps with a channel dimension of 64 / a. Now, the summer receives (i) the input to the residual unit (via a skip connection) with a channel dimension of 64 / a, and (ii) the output of the third convolution layer with a channel dimension of 64 / a. This ensures that the channel dimensions of the two inputs to the summer matches, and dimensional integrity of the skip connection is maintained. This example use case is discussed in further detail with respect to Fig. 5 herein below.

[0049] Note that similar to maintaining the dimensional integrity of the skip connection within a residual unit, similar strategies can also be adapted for skip connections from an encoder to adecoder of a ML model, as also described below in further detail. Numerous configurations and embodiments will be apparent in light of this disclosure.ML models with skip connections (such as HoVer-Net architecture. Residual Units, U- Net architecture)

[0050] As described above, techniques are described herein for compressing ML Models. As also described above, applying compression techniques for ML models employing skip connections pose additional challenges. This section of this disclosure describes various ML models that employ skip connections, where various compression techniques (such as pruning techniques) described herein may be applied to one or more of these models but is not limited thereto. The described techniques may be applied to other models with similar architectures.

[0051] Fig. 1 illustrates a ML model 100 depicting a HoVer-Net architecture, where the ML model 100 is compressed using techniques described herein. In an example, HoVer-Net may be used for digital pathology tasks, such as detecting and classifying cell nuclei in whole-slide images. The HoVer-Net ML model 100 is used in the field of digital pathology, e.g., to process digital pathology images. For example, the ML model 100 may be used for nuclei segmentation, classification, and extraction of nuclear boundary maps in histopathology images.

[0052] The ML model 100 receives an input Image 102. The ML model 100 processes, for example, an RGB (red, blue, green) image or an image of another type and of an appropriate dimension, such as a 270 x 270 x 3 RGB image.

[0053] The ML model 100 comprises a feature extraction branch 103 (also referred to as a feature extraction backbone) to extract features from the input image 102. The feature extraction branch 103 includes a convolution layer 104 processing the input image 102. Note that convolution layers are labelled as “conv” in at least some of the figures of this disclosure. Merely as an example, a 7x7 kernel of the convolution layer 104 processes the input image 102. The output of the convolution layer 104 is processed by a series of residual units 108a, 108b, 108c, 108d. In an example, residual units are a part of a ResNet architecture and facilitate in training deep networks effectively by introducing skip connections, as described below in further detail (see Fig. 2 for skip connections within a residual unit). An output of the residual unit 108d is processed by another convolution layer 112.

[0054] The output of the convolution layer 1 12 is branched in three different branches, such as a nuclear pixel branch 120, a HoVer (Horizontal and Vertical) branch 130, and a nuclear classification branch 140. The branches 120, 130, and 140 are task specific, each configured to perform a specific task.

[0055] The nuclear pixel branch 120, in an example, predicts a binary segmentation map of one or more nuclei (such as differentiates between foreground versus background). The nuclear pixel branch 120 comprises a plurality of up-sampling layers 121a, 121b, 121c, a plurality of convolution layers 122a, 122b, 122c, 122d, 122e, 122f, and a plurality of decoder units 124a, 124b. In an example, the nuclear pixel branch 120 outputs a binary mask of nuclei.

[0056] The HoVer branch 130 predicts horizontal and vertical maps, which encode spatial displacement of nuclear pixels from the center of each nucleus. The HoVer branch 130 comprises a plurality of up-sampling layers 131a, 131b, 131c, a plurality of convolution layers 132a, 132b, 132c, 132d, 132e, 132f, and a plurality of decoder units 134a, 134b. In an example, the HoVer branch 130 outputs one or more horizontal and vertical displacement maps.

[0057] The nuclear classification branch 140 classifies nuclei into categories based on their type (e.g., epithelial, lymphocyte, etc ). The nuclear classification branch 140 comprises a plurality of up-sampling layers 141a, 141b, 141c, a plurality of convolution layers 142a, 142b, 142c, 142d, 142e, 142f, and a plurality of decoder units 144a, 144b. In an example, the nuclear classification branch 140 outputs categorical maps for nuclear types.

[0058] The feature extraction branch 103 forms an encoder of the ML model 100, and the nuclear pixel branch 120, the HoVer branch 130, and the nuclear classification branch 140 form three decoder branches of the ML model 100. Thus, the ML model 100 is a multi -branch encoderdecoder network. As described above, the encoder (e.g., the feature extraction branch 103) extracts features from the input image 102, which will be used by multiple decoder branches. After feature extraction, the ML model 100 splits into three separate decoder branches to perform different tasks, as described above.

[0059] As illustrated in Fig. 1, the ML model 100 comprises a series of residual units 108a, 108b, 108c, 108d, where the number of such residual units is implementation specific. Example structure of the residual units 108a, . .. , 108d are described below in further detail, e.g., with respectto Fig. 2. As will be described below with respect to Fig. 2, each residual unit 108 includes a corresponding skip connection.

[0060] Furthermore, there are one or more corresponding skip connections between the feature extraction branch 103 (e.g., the encoder branch) and each of the decoder branches (such as the nuclear pixel branch 120, the HoVer branch 130, and the nuclear classification branch 140). These skip connections are illustrated using dotted lines in Fig. 1. For example, skip connections 180 are between the feature extraction branch 103 and the nuclear pixel branch 120; skip connections 181 are between the feature extraction branch 103 and the HoVer branch 130; and skip connections 182 are between the feature extraction branch 103 and the nuclear classification branch 140. Skip connections between an encoder branch and a decoder branch are described below in further detail with respect to Fig. 3.

[0061] Thus, the ML model 100 includes two types of skip connections - (i) skip connections within the residual units 108a, ..., 108d, and (ii) skip connections between the feature extraction branch 103 and each of the three decoder branches. These skip connections pose additional challenges in compressing (such as pruning) the ML model 100, and techniques for compressing the ML model 100, while maintaining dimensional integrity of the skip connection within the ML model 100, are described below in further detail.

[0062] Fig. 2 illustrates a section 200 of a ML model, where the section 200 of the ML model comprises a series of residual units 204a, 204b, 204c, where the section 200 of the ML model is compressed using techniques described herein. Although three residual units 204a, 204b, 204c are illustrated in Fig. 2, the section 200 may include higher (or lower) number of such residual units.

[0063] In an example, the section 200 is a part of a ResNet architecture. In an example, the section 200 is a part of a HoVer-Net architecture (such as part of the feature extraction branch 103 of the ML model 100 of Fig. 1). In an example, the section 200 is a part of a U-Net architecture (such as the ML model 300 of Fig. 3 described below).

[0064] In a network with residual units, a layer may feed into the next layer, and also into another layer few hops away (such as about 2-3 hops away in Fig. 3). Thus, skip connections are introduced within the residual unit. The residual units 204a, 204b, 204c of Fig. 2 may be the residual units illustrated in Fig. 1, or maybe one or more other residual units described herein.

[0065] In an example, the residual units 204a, 204b, 204c are bottleneck residual units. For example, a bottleneck residual unit is a type of residual unit, where a bottleneck residual unit uses relatively fewer parameters, e.g., by introducing a 1x1 convolution for dimensionality reduction before applying heavier 3x3 convolutions.

[0066] The section 200 comprises a convolution layer 203a processing an input 201, followed by a batch normalization layer 203b. An output of the batch normalization layer 203b is provided to the residual unit 204a.

[0067] The residual units 204a, 204b are illustrated in further detail in Fig. 2. For purposes of illustrative clarity, the boundary of the residual unit 204a is illustrated using solid lines and the boundary of the residual unit 204b is illustrated using dotted lines.

[0068] As illustrated in Fig. 2, the residual unit 204a receives an output 202 of the batch normalization layer 203b, and processes the output 202 (considered as input 202 for the residual unit 204a) using convolution layers 210, 213, 216, batch normalization layers 211, 214, 217, and rectified linear units (ReLU) 212, 215, 219, where these layers are arranged in the manner illustrated in Fig. 2.

[0069] Note that in an example where the residual units 204a, 204, 204c are bottleneck residual units, a first convolution layer of the residual unit (such as the convolution layer 210) and / or a last convolution layer (such as the convolution layer 216) of the residual unit may be a 1x1 convolution layer, whereas an intermediate convolution layer of the residual unit (such as the convolution layer 213) may be a higher dimension convolution layer, such as a 3x3 convolution layer.

[0070] In addition to the input 202 propagating through various layers of the residual unit 204a, the input 202 also traverses through a residual branch 220 (also referred to as a skip connection branch 220), and is summed at a summer 218 with an output of the batch normalization layer 217, as illustrated in Fig. 2.

[0071] Although Fig. 2 does not illustrate any convolution layer within the skip connection branch 220, the skip connection branch 220 may include one or more additional convolution layers (such as a 1x1 convolution layer).

[0072] Similarly, the residual unit 204b receives an input 222 from the residual unit 204b (such as from the summer 218, through a ReLu layer 219), and processes the input 222 using convolutionlayers 230, 233, 236, batch normalization layers 231, 234, 237, and ReLU layers 232, 235, 239, where these layers are arranged in the manner illustrated in Fig. 2. Note that in an example where the residual units 204a, 204, 204c are bottleneck residual units, the convolution layer 230 and / or the convolution layer 236 may be a 1x1 convolution layer, whereas the convolution layer 233 may be a higher dimension convolution layer, such as a 3x3 convolution layer, as described above with respect to the residual unit 204a.

[0073] In addition to the input 222 propagating through various layers of the residual unit 204b, the input 222 also traverses through a residual branch 240 (or a skip connection branch 240), and is summed at a summer 238 with an output of the batch normalization layer 237, as illustrated in Fig. 2.

[0074] Fig. 3 illustrates a ML model 300 comprising a U-Net architecture, where the ML model 300 is compressed using techniques described herein. U-Net is a convolutional neural network (CNN) architecture which may be used for many different purposes, such as for image segmentation tasks. In an example, U-Net is used extensively in image segmentation in biomedical imaging applications, and for processing digital pathology images.

[0075] The U-Net ML model 300 of Fig. 3 has a U-shaped structure comprising two sections: (i) a contracting path, also referred to as an encoder 362 in Fig. 3, and (ii) an expanding path, also referred to as a decoder 364 in Fig. 3. The encoder 362 captures context of an input image tile 302 through downsampling. The decoder 364 recovers spatial information through upsampling.

[0076] Various layers 304a, 304b, ..., 340o of the encoder 362 are illustrated, and various layers 304p, 304q, ..., 304z, 304aa of the decoder 364 are illustrated in Fig. 3. These layers are generally referred to as layers 304 in general. One or more of these layers 304 comprises a corresponding convolution layer, followed by a ReLU activation. Although individual ones of the layers 304a, .. ., 304z, 304aa comprise one or more convolution layers and ReLu activations, in an example, one or more of these layers may also comprise residual unit architecture. For example, a layer 304 may include a skip connection therewithin, e.g., as described above with respect to Fig. 2.

[0077] In the encoder 362, the convolution of the layers 304a, ..., 304o reduces spatial dimensions of the input, and increases feature depth, and facilitates in understanding the globalcontext of the input image. In the encoder 362, after each layer 304c, 304f, 304i, 3041, corresponding max pooling operations 308a, 308b, 308c, 308d, respectively, are performed.

[0078] In the decoder path 364, the convolution of the layers 304s, . . ., 304z, 304aa decreases feature depth. Furthermore, up-convolutions (or transposed convolutions) of the input are performed at 312a, .. ., 312d, as illustrated in Fig. 3.

[0079] Furthermore, there are a plurality of skip connections 320a, 320b, 320c, 320d. Each skip connection 320 is from a corresponding layer of the encoder 362 to a corresponding layer of the decoder 364. For example, the skip connection 320a servers as a bridge between the layer 3041 of the encoder 362 and the layer 304p of the decoder 364. For example, output of the layer 3041 and output of the layer 304o are concatenated and then convoluted at the layer 304p. Similarly, output of the layer 304i and output of the layer 304r are concatenated and then convoluted at the layer 304s. Similarly, output of the layer 304f and output of the layer 304u are concatenated and then convoluted at the layer 304c. Finally, output of the layer 304c and output of the layer 304x are concatenated and then convoluted at the layer 304y. The concatenation operations are labelled as “concat” in Fig. 3. Thus, concatenation (concat) operations 316a, 316b,316c, 316d are respectively performed at layers 304p, 304s, 304v, and 304y.Model compression

[0080] Various models (such as the HoVer-Net ML model 100, the ResNet ML model 200, and / or the U-Net model 300 described above) may be relatively large in size, and such a massive sized model increases the computational costs and requires high processing power to execute, thereby increasing cost of organizations employing these models. Accordingly, in an example, it may be intended to reduce the model size and reduce the computational costs of the one or more of these models. Techniques available for such compression of ML models include model pruning, knowledge distillation, quantization, etc.

[0081] In model pruning, unnecessary or relatively unimportant model parameters are removed from a ML model, thereby decreasing a size of the ML model, which in turn reduces cost of storage and / or execution of the ML model. However, applying model pruning to some ML models, such as those ML models in which skip connections are included, present additional challenges. For example, it may be challenging to prune an ML model including a skip connection (such as a model including a residual unit, and / or a model including a skip connection between anencoder and a decoder path), e.g., because the skip connections in a layer impact model dimensions downstream, thereby possibly making dimensions for skip connections incompatible. For example, because in a skip connection two inputs are to be summed or concatenated, any incompatible dimensions due to the pruning may render such a model non-operational. Thus, while implementing pruning of a ML model including one or more skip connections, additional mitigating strategies has to be implemented for (i) skip connections between an encoder and a decoder and / or (ii) the skip connections within a residual unit. For example, pruning strategies undertaken has to maintain dimensional integrity of the skip connections within the ML model, as described below in detail.

[0082] Accordingly, in various embodiments and as will be described below in further detail, for pruning a ML model including skip connections, two layers that are interconnected through a skip connection are identified. In an example, this may be the layers 304i and 304r between the encoder 362 and the decoder 364, respectively, in Fig. 3 (e.g., output of these layers 304i and 304r concatenated in layer 304s). In another example, this may be the layers 216 and 236 in Fig. 2, as output of these layers are summed at the summer 238 (note that the batch normalization layers and the ReLU layers of residual units do not change dimensionality of feature maps).

[0083] In an example, once the two layers (e.g., that are interconnected through a skip connection) are identified, the pruning of the ML model is performed such that the pruning does not lead to incompatibility among the inter-connected layers. For example, pruning strategies undertaken maintain dimensional integrity of the skip connections within the ML model.

[0084] For example, the ML model is pruned such that the same number of filters in convolutional layers and / or the convolution filters with the same indexing are pruned, ensuring that the skip-connected layers (e.g., layers connected via skip connections) have the same filter numbers.

[0085] For example, referring to Fig. 2, for the summation operation at the summer 238 within the residual unit 204b, the dimension of the output of the conv 216 and the dimension of the output of the conv 236 has to be compatible. Note that the batch normalization layers and the ReLU layers of residual units do not change dimensionality of feature maps, and hence, presence of these layers is ignored when computing dimensionalities. For example, assume that the dimension of the output of the conv 216 is B x C x H x W, where B is the batch size, C is the number of channels, and Hand W are the height and width, respectively, of the feature maps. Then, for ensuring dimensional integrity of the skip connection 240 (and compatibility in the summer 238), the dimension of the output of the conv 236 also has to be B x C x H x W.

[0086] Similarly, when concatenating in a U-Net (such as concat operations 316a, 316b, 316c, 316d), the height and width of the feature maps to be concatenated have to match. Also, in an example, the number of channels of the feature maps to be concatenated may have to match. To account for any mismatch in height and width of the feature maps, cropping and / or padding may be employed. However, in an example, any pruning of the channel size of the features maps has to be taken into account, to ensure matching of the channel sizes of the feature maps to be concatenated.

[0087] Fig. 4 illustrates a table 400 depicting example input and output channel dimensions of various layers of a ML model, such as any of the ML models described herein, wherein various layers within each residual block or convolutional block are pruned independently (e.g., pruning ratios of different residual units can be set independently) to ensure dimensional integrity of the skip connections within the ML model. Note that in table 400, only those layers, which affect dimensionality of the feature maps, are illustrated. For example, as the batch normalization layers and ReLU layers do not affect dimensionality of the feature maps, the batch normalization layers 203b, 211, 214, 217, 231, 234, 237 and ReLU activation layers 212, 215, 219, 232, 235, 239 of Fig. 2 are not illustrated in Fig. 4. Column 1 of table 400 depicts the various layers (e g., layers which can possibly affect dimensions of the feature maps) of a ML model, such as the section 200 of the ML model of Fig. 2. Columns 2 and 3 respectively illustrate number of input channels and number of output channels in the original version of the ML model (e.g., prior to pruning the ML model). For example, the original ML model is initially trained with the channel dimensions illustrated in columns 2 and 3. Thus, for example, an input to the conv 210 has 63 channels and an output of the conv 210 has 64 channels in the original trained and unpruned ML model.

[0088] Columns 4 and 5 respectively illustrate number of input channels and number of output channels in the pruned version of the ML model (e.g., subsequent to pruning). Thus, for example, pruning of the ML model changes the output channels of the layer 210 from 64 in the original ML model to 64 / a in the pruned ML model.

[0089] The pruning settings for two example residual units 204a and 204b of Fig. 2 are illustrated in Fig. 4. In the example of Fig. 4, only certain layers within each residual unit or convolutional block, which do not impact the subsequent layers, are pruned. For example, in Fig. 4, as the output of convolution layers 216 and 236 are to be summed at the summer 238, output channel dimensions of these convolution layers 216 and 236 remain same as the original unpruned version (which in this example is 64). The layers chosen to prune are the convolution layers 210, 213, 230, 233.

[0090] For example, actively pruning (“active” pruning is described below in further detail) the convolutional layer 210 results in an output channel dimension of 64 / a, where “a” is the pruning ratio setting, which may be an appropriate integer (e.g., 2, 4, 8, etc ). For example, “a” being equal to 4 implies that the output channel dimension of layer 210 is 16. Thus, for the example in which “a” is 4, 25% or 16 of the 64 original channels remain in the output of the layer 210 after pruning.

[0091] Note that the batch normalization layer 211 receives the feature maps with channel dimension of 64 / a, and transmits the feature maps with channel dimension of 64 / a (after batch normalization) to the ReLu 212, and then to the convolution layer 213 (see Fig. 2).

[0092] To maintain compatibility in the channel dimensions and dimensional integrity of the skip connection, the convolution layer 213 converts the 64 / a number of channels back to 64 channels. Accordingly, the output of the convolution layer 213 has 64 channels.

[0093] Note that the output of the convolution layer 213 in the original unpruned ML model and in the pruned ML model are the same (e.g., 64). However, the input to the convolution layer 213 in the original unpruned ML model is 64, whereas the input to the convolution layer 213 in the pruned ML model is 64 / a.

[0094] In an example (such as where the residual unit 204a is a bottleneck residual unit), the convolution layer 216 is a 1 x 1 convolution layer, and does not change a number of channels, and generates feature map outputs with 64 channels to the summer 218. Also, the output of the convolution layer 203a (which is also the input to the convolution layer 210) has 64 channels, which is also transmitted to the summer 218. Accordingly, both inputs to the summer 218 has the same channel dimensions of 64, resulting in compatibility in channel dimensions in the summer 218, thereby maintaining dimensional integrity of the skip connection 220 within the ML model 200.

[0095] Similarly, for example, for the residual unit 204b, the input to the convolution layer 230 (which is also an input to the summer 238) is 64, and the output of the convolution layer 236 (which is also another input to the summer 238) is also 64. Note that in intermediate layers of the residual unit 204b, the channel dimensions are pruned from 64 to 64 / b (where “b” is another prune ratio setting, and where a and b may be the same or different), and then back to 64. Accordingly, as illustrated in Fig. 4, both inputs to the summer 218 have the same channel dimensions (e.g., 64), resulting in compatibility in channel dimensions in the summer 218, thereby maintaining dimensional integrity of the skip connection 240 within the ML model 200.

[0096] Accordingly, with this pruning described above with respect to Fig. 4, same or different pruning ratios can be selected for each block of layers (e.g., residual units, or a block of multiple convolution layers followed by other types of layers like batch-normalization and / or ReLU layers). For example, in Fig. 4, a prune ration setting of “a” is achieved for the residual block 204a, and a prune ration setting of “b” is achieved for the residual block 204b. Thus, pruning of each block is independent from other blocks in Fig. 4.

[0097] Fig. 5 illustrates a table 500 depicting another example input and output channel dimensions of various layers of a ML model, such as any of the ML models described herein, where pruning ratios of different residual blocks or convolutional blocks are set in unison, to ensure dimensional integrity of the skip connections within the ML model. Similar to the table 400 of Fig. 4, in table 500 of Fig. 5, only those layers that affect dimensionality of the feature maps are illustrated. For example, as the batch normalization layers and ReLU layers do not affect dimensionality of the feature maps, the batch normalization layers 203b, 211, 214, 217, 231, 234, 237 and ReLU activation layers 212, 215, 219, 232, 235, 239 are not illustrated in Fig. 5.

[0098] In Figs. 2 and table 500, a residual unit has at least a first convolution layer, a second convolution layer, and a third convolution layer. For example, the residual unit 204a has convolution layers 210, 213, and 216. For a given residual unit, the first convolution layer can be pruned independently (e.g., active pruning, described below). For example, the first convolution layer 210 changes a channel dimension from 64 / a to 64 / b (note that the prune settings a, b, and / or c in table 500 may be different from those in table 400). However, the second (or the third) convolution layer changes the channel dimension back to the specific channel dimensions of the input to the first convolution layer (e.g., passive pruning of the second convolution layer, alsodescribed below). For example, the second convolution layer 213 changes the channel dimension back to 64 / a. Thus, input to the residual unit 204a has a channel dimension of 64 / a and output of the residual unit 204a also has a channel dimension of 64 / a. Thus, both the inputs to the summer 218 have the channel dimension of 64 / a, thereby maintaining dimensional integrity of the skip connection 220 within the ML model 200.

[0099] Similarly, for the residual block 204b, the input to the residual block 204b (e.g., the input to the convolution layer 230) has a channel dimension of 64 / a. Also, the output of the residual block 204b (e.g., the output of the convolution layer 236) has a channel dimension of 64 / a. Thus, both the inputs to the summer 238 have the channel dimension of 64 / a, thereby maintaining dimensional integrity of the skip connection 240 within the ML model 200.

[0100] Note that in table 500, the channel dimension of the output of the convolution layer 203a (which is 64 / a in Fig. 5) dictates the channel dimensions of inputs and outputs of various subsequent layers. Thus, there is a ripple effect here, where one residual unit dictates pruning ratios of subsequent residual units.

[0101] Note that the matching of the channel dimensions also holds for other types of skip connections, such as a skip connection between an encoder and a decoder of the U-Net ML model 300 of Fig. 3. For example, if matching of channel dimensions during the concatenation operations is desired, then the pruning has to take into account the channel dimensions at corresponding layers in both encoder 362 and decoder 364. For example, for the skip connection 320b, the pruning ratios of convolution layer 304g, 304h, 304i, 304q, and / or 304r may be adjusted that the outputs of the compute instances 304i and 304r have the same channel dimensions, thereby maintaining dimensional integrity of the skip connection 320b within the ML model 300.

[0102] Referring to the tables 400 and 500 of Figs. 4 and 5, pruning a convolution layer in a ML model triggers or necessities corresponding pruning of one or more other convolution layers in one or more residual units. For example, in Fig. 5, pruning the convolution layer 203a from 64 to 64 / a triggers corresponding pruning of various other convolution layers within the residual blocks 213, 233. Such forced pruning may result in pruning of relatively significant or important filters. As such, a non-uniform sparsity (discussed herein below) may be used, to address potential situations where uniform pruning does not effectively preserve model performance.Active versus passive pruning

[0103] A convolution layer may be pruned actively or passively. An “active” pruning refers to pruning output channel dimension of a layer of a ML model, so as to compress the ML model. For convolutional layers, the output channel dimension may be changed (such as reduced), e.g., by changing (such as reducing) a number of filters used in the convolution layer. An actively pruned layer impacts the pruning of a subsequent layer (such as an immediate subsequent layer), as described above with respect to Figs. 4 and 5. For example, referring to Fig. 4, the convolution layer 210 is actively pruned, to reduce the output channel dimension from 64 to 64 / a. This affects the pruning of a subsequent convolution layer 213.

[0104] In a passive pruning, an output channel dimension of a convolution layer “has” to be pruned, to accommodate the output dimension of an inter-connected previous convolution layer and to maintain dimensional integrity of skip connections within the ML model.. For example, the convolution layer 213 has to be pruned, such that the output channel dimension of this layer becomes 64 (see Fig. 4), e.g., for maintaining dimensional integrity of the skip connection within the ML model. Thus, active pruning of a convolution layer triggers a passive pruning of one or more downstream convolution layers.

[0105] The active pruning can be thought of as a choice, where a convolution layer is actively pruned for ML model compression purposes. In contrast, the passive pruning of a convolution layer is necessitated by pruning of a previous convolution layer, in order to maintain dimension compatibility at a summer or a concatenation layer.Unstructured, structured, and semi-structured pruning

[0106] Based on how the model parameters are pruned, pruning algorithms may be classified into different categories, such as unstructured pruning, structured pruning, and semi -structured pruning. In unstructured pruning, using one or more pruning heuristic, a determination is made as to a relative importance of individual parameters of the ML model, and relatively unimportant ones are pruned by replacing them with zero values. Unstructured pruning may be relatively easier to implement, and may not significantly degrade model performance. However, available hardware may not be suitable (such as optimized) for unstructured sparse matrices, and thus, in anexample, unstructured pruning may not improve the latency or inference speedup (e.g., unless specific hardware and corresponding software and / or firmware system designs are developed specifically for unstructured pruning).

[0107] In structured pruning, pruning is done following a fixed pattern. For example, an entire filter of a convolutional layer or an entire layer may be removed. In contrast to the unstructured pruning, since the entire convolution layer or an entire filter is removed, such pruning induces model speedup. Different pruning heuristics for structured pruning, such as LI pruning, L2 pruning, iterative magnitude pruning, non-uniform pruning, network slimmer, etc. have been described below in further detail.

[0108] Semi-structured pruning explores sparsity patterns between unstructured and structured pruning. For example, in contrast to structured pruning where an entire filter may be completely removed, semi-structured pruning explores sparsity patterns between unstructured and structured pruning, such as block sparsity or n:m sparsity. Semi-structured pruning selectively prunes groups of weights while preserving some structure, achieving a balance between efficiency and hardware compatibility.Type of pruners

[0109] Various types of pruners may be used, such as pruning based on LI norm, pruning based on L2 norm, iterative magnitude pruning, non-uniform pruning, network slimmer, etc., as described below.LI pruning

[0110] In LI pruning, the pruner prunes relatively less important filters from a trained model and optionally minimizing the accuracy drop via fine tuning (e.g., subsequent to pruning). Such a pruning method uses an LI norm of one or more filters (such as each filter) of a convolution layer as a heuristic, to measure a relative importance of each filter. For example, the motivation is that filters with relatively smaller magnitudes (where selection of filters are described below) tend to produce feature maps with weak activations as compared to the other filters in the layer. As such, after pruning the filters with the relatively smaller magnitude, the ML model predictions may not change significantly. For LI pruning, the pruning mechanism is as follows. For each filter in a convolution layer, a sum of its absolute kernel weight (e.g., LI norm weight) is calculated. Forexample, if the filter is a 3x3 kernel, then a sum of absolute values of the 9 entries of the kernel is calculated. A plurality of filters of the convolution layer is sorted in ascending order in a list. A specific number (such as M number of filters) of bottom-most filters from the list are selected, where the number of selected filters may be based on a target sparsity ratio. To ensure correct dimension for matrix multiplication, the kernels of the same indices in each filter of the next convolution layers are also correspondingly pruned, as described above. Optionally, the pruned and compressed model may be fine tuned to recover any drop in performance.L2 pruning

[0111] L2 pruning is at least in part similar to LI pruning. However, in L2 pruning, the L2 norm of the kernel weights is calculated. For example, if the filter is a 3x3 kernel, then the L2 norm of a vector comprising the 9 entries of the kernel is calculated. L2 pruning will be evident to those skilled in the art, based on the above description with respect to the LI pruning.Iterative magnitude pruning

[0112] Iterative Magnitude Pruning (IMP) involves repeatedly pruning low-magnitude weights, retraining the model, and iterating this process multiple times. An algorithm for IMP to achieve desired sparsity ratio of ‘m’ is as follows. The full or uncompressed ML model is initialized randomly (such as with random weights). The full model is trained for a specific number (e.g., k number) of epochs and / or iterations, and m / n number of weights are pruned. The model is trained further and m / n number of weights are further pruned. This process is iteratively repeated (e.g., for n number of times). After completion of the n number of iterations, the final pruned model is achieved. The final model has a desired sparsity level of m. Here, “n” refers to the number of iterations to achieve “m” sparsity ratio. For example, with m of 80% and n being 10, this means pruning 80 / 10 = 10% of filters (or other target model components) at each iteration for 10 iterations.Non-uniform pruning

[0113] In an example, not all the layers in a given ML model are equally important. For example, some of the layers can learn relatively more important information and pruning such layers may result in relatively higher performance drop (e.g., compared to pruning layers that are of relatively less important). In contrast to the uniform pruning where all the layers are prunedwith the same ratio, in an example, in non-uniform pruning, relatively more important layers are identified. The identification may be based on a goal of the ML model, role of individual layers in achieving the goal, and / or other heuristic or deterministic factors. The identified relatively more important layers are then pruned to a lower sparsity ratio. The actual pruning can be achieved using any of the techniques discussed herein (e.g., LI pruning, L2 pruning, IMP, etc.), in an example.Network slimmer

[0114] A network slimmer is a pruning technique in which redundant or less important parameters are pruned. In an example, a network slimmer focuses on channel pruning, where entire channels or filters in convolutional layers are removed rather than just individual weights. In an example, network slimmer uses LI -regularization on the scaling factors of BatchNorm layers during model training to induce sparsity of the scaling factors. That is, LI -norm of the scaling factor in the loss function pushes relatively less important scaling factors corresponding to the less important feature maps close to zero. Therefore, instead of calculating the LI and / or L2 norm of the filters, network slimmer selects filters with a smaller scaling factor after training, e.g., as a heuristic for pruning filters.Pruning approaches

[0115] Below are discussions on three example pruning approaches, such as pruning (i) using condition statements, (ii) separation of model components and pruning strategies, and (iii) pruning based on identification of connected layers,Pruning approach 1 : Using condition statements

[0116] In an example, an ML model pruning program may be implemented using condition statements (e.g., using “if . .. then” clauses). For example, one or more convolution layer may be associated with such condition statements. Merely as an example, referring to Fig. 5, a condition statement for convolution layer 213 may be as follows: If output channel dimensions of convolution layers 203a and 210 do not match after pruning, then prune filters of the convolution layer 213 such that output channel dimension of the convolution layer 213 matches with that of the output of the convolution layer 203a. In an example, this approach calls for separate pruning pipelines for each model architecture and thus has the relatively high level of dependency on the ML architecture to be pruned. This approach may be adapted to prune ML models, e.g., when theML model is relatively simple and thus, the condition statement list is relatively short and relatively easy to construct.Pruning approach 2: Separation of model components and pruning strategies

[0117] In an example, pruning functionalities may be separated from model components, thereby reducing any model architectural dependency for model pruning. In an example, different pruning settings (such as the pruning ratios described above with respect to Figs. 4 and 5) may be received from a model designer or a model user. For example, the same layout of the configuration files may be reused for different model architectures and different pruning strategies. Fig. 6 illustrates example pseudo-codes for specifying model pruning settings and model components. Thus, the pruning configuration files may be generated to separately include the pruning strategies and a list of model components to prune. For each model component, additional pruning strategies may be included, along with layer-specific dependencies. For example, for a given convolution layer (e.g., layerl .O.convl), a desired prune ratio is set to 0.5, and also specified is a type of pruning (e.g., active pruning, and not passive pruning) for this layer. In contrast, for another convolution layer (e.g., layerl. O.convl), the type of pruning is set to passive pruning. Note that active and passive pruning are described above. In Fig. 6, also mentioned in prune strategies are prune criteria (such as LI or L2 pruning), which are described elsewhere in this disclosure in further detail.Pruning approach 3: Pruning based on identification of layers of the model

[0118] In an example, in this pruning approach, the pruning can be performed automatically for any given ML model. This approach involves two steps: (i) initially, a connectivity searching program scans a ML model architecture and identifies one or more (such as all) model layers and connections between various layers, and (ii) subsequently, a configuration file dictates the pruning program as to which layers are to be pruned and corresponding the pruning strategies.

[0119] A difference between the above-described pruning approach 2 (separation of model components and pruning strategies) and pruning approach 3 (pruning based on identification of layers of the model) lies in how the layer connections are identified. For example, in the pruning approach 2, a user or a programmer manually identifies roles of individual layers, and provides information associated with layer connectivity and layer-wise pruning dependencies (e.g., in the form of configuration files, identifying whether active or passive pruning settings are to be applied to a layer, etc.). In contrast, in the pruning approach 3, the layer connectivity and lay er- wisepruning dependencies are automatically identified by a connectivity searching program. For example, the connectivity searching program applies an appropriate technique (such as graph theory) to identify connectivity between various layers. As an example, with pruning approach 3, a user has the option to not explicitly specify “passive” pruning layers in the configuration file, but only include which layers to be “actively” pruned, since one or more (such as all) the interconnected layers to be “passively” pruned are already identified automatically by the connectivity searching program.Methodologies

[0120] Fig. 7 illustrates an example flow chart of a computer-implemented method 700 for pruning a ML model including one or more skip connections. At 704, a ML model is trained using techniques for training ML models. At 708, one or more skip connections within the ML model are identified. For example, a user or programmer can specify interconnections between the various layers of the ML model, thereby implicitly identifying the skip connections, as described above. In another example, a connectivity searching program (such as a program based on graph theory) may be used to identify interconnections between the various layers of the ML model, thereby also identifying the skip connections, as also described above.

[0121] At 712, one or more filters of the ML model are pruned, while maintaining dimensional integrity of the skip connection within the ML model. For example, as described above, one or more layers (such as layer 210, see Figs. 4 and 5) may be actively pruned, for purposes of model compression. In order to maintain dimensional integrity of the skip connection 220 due to the pruning of the layer 210, the layer 213 has also to be passively pruned, as described above with respect to Figs. 4 and 5.

[0122] At 716, the pruned ML model may be fine-tuned, e.g., by retraining the pruned ML model. For example, pruned weights may be further adjusted during the retraining process. In an example, operations 712 and 716 may be iteratively repeated, e.g., until a desired pruning sparsity is achieved.Experimental resultsCell-level instance segmentation and classification modeling

[0123] In an example, the HoverNet model may be used for evaluating pruning methods for digital pathology. The HoverNet model is trained on PanNuke dataset, which is a relatively challenging nuclei instance segmentation datasets, comprising about 200,000 annotated Nuclei into 5 clinically relatively important classes across 19 tissue types. Following the same training paradigm as; the encoder is initialized with the pre-trained ImageNet weights and the encoder is frozen for the initial few epochs of the training. An Adam optimizer is used with a learning rate of le-4. As described above, HoverNet architecture has three decoder branches - NP branch (e.g., nuclear pixel branch 120, see Fig. 1), NC branch (e.g., nuclear classification branch 140), and HV branch (e.g., HoVer branch 130) - to generate a binary segmentation mask, classification, horizontal and vertical vertical gradients for edge, respectively.Pruning ResNets

[0124] In an example, an effectiveness of model pruning for the tile-wise classification of slide artifacts was validated, e.g., by applying a L2-norm based pruning of ResNetl8 (e.g., where L2 norm-based pruning has been described above). ResNetl8 is a commonly ResNet family model used for biomedical image analysis. The classification model classifies each image tile into two classes, such as either non-artifact or artifact. For example, the artifact tiles are at least partly occupied by pen marks (pen mark colors include red, green, blue, black, etc.). For example, Fig. 8 illustrates example whole slide images with pen-mark artifacts. In the experiments, the first convolution layer in the active pruning mode and the subsequent batch-normalization layer in a passive pruning mode is pruned for each residual block.

[0125] The training set comprises about 115.7 million hematoxylin and eosin (H&E) image tiles generally used in histopathological images, the validation set comprises about 30.4 million tiles, and the testing set comprises 19.5 million image tiles (e.g., collected from a set of independent slide images). In an example, one or more (such as all) such image tiles are of 224x224 pixels in size. ImageNet pretrained ResNetl8 model may be used as starting point, and the model may be fine-tuned with the training data, e.g., with a learning rate of 0.0000005, a maximum epoch number of 60, and with early stopping when validation loss does not improve for 3 or more epochs. One-shot uniform pruning is then applied with target sparsity ratios of 25%, 50%, 75%, 87.5%, and 93.75% for each experiment, respectively, which is followed by fine-tuning with a maximum epoch number of 25 and the same early stopping as the initial fine-tuning.Metrics used

[0126] One or more of the following metrics may be used.Dice Coefficient Metrics

[0127] Dice Coefficient is used to statistically measure a similarity between the ground truth segmentation mask and the predicted or generated segmentation mask. Mathematical, dice coefficient between two masks X and Y may be described in terms of set operations as:2* |XnY| / (|X|+|Y|)Detection Quality Metrics

[0128] Detection Quality (DQ) is a metric used for evaluation of segmentation methods. DQ is a Fl score that measures the quality of instance detection. Mathematically, it can be described as:Segmentation Quality (SO) Metrics

[0129] Segmentation Quality (SQ) measures how close each correctly detected instance is to their ground truth mask. Mathematically, it can be described as:Panoptic Quality (PQ) Metrics

[0130] Panoptic Quality (PQ) is a unified score for comparing both segmentation and classification between the predicted labels and ground truth. Mathematically, PQ is the product of DQ and SQ described above.Results and Observation

[0131] Cell Segmentation

[0132] Different pruning methods (e.g., heuristics pruning methods) to compress the original HoverNet architecture are used to reduce its latency and memory footprint. As HoverNet hasseparate branches for classification and segmentation, the HoverNet model is evaluated with and without the classification branch.

[0133] Uniformly prune all prunable layers with I -norm pruner

[0134] In this experiment, LI pruner is used to uniformly prune one or more (such as all) the layers of HoverNet with only the segmentation branch. Dice loss is used as the validation loss. Fig. 9 illustrates a graph 900 depicting validation loss, e.g., when a LI pruner is used to uniformly prune one or more layers of HoverNet with only a segmentation branch. As illustrated in Fig. 9, the network can be pruned up to 70% sparsity with the negligible drop in performance.

[0135] Memory footprint

[0136] Pruning filters reduces a number of parameters in the model and therefore reduces the memory footprint of the model. Fig. 10 illustrates a graph 1000 depicting a reduction of memory footprint as a result of pruning. As illustrated in Fig. 10, at about a 50% sparsity ratio, memory footprint is almost reduced by 60%. This can reduce storage requirements for saving trained models and run time memory.Cell instance segmentation and classification: Comparison of different pruning methods

[0137] To compare different pruning methods for the HoverNet model, the pruning methods for both segmentation and classification branches are evaluated. Fig. 11A illustrates a graph 1100 depicting an effect of pruning on segmentation validation loss for the HoverNet model. Fig. 11B illustrates a graph 1150 depicting an effect of pruning on classification validation loss for the HoverNet model. As illustrated in Figs. 11 A and 1 IB, about 50% of the pruned model can be removed with almost similar segmentation and classification performance.Latency

[0138] To compare the latency, the same GPU hardware with the same batch size are used. Fig. 12 illustrates a graph 1200 depicting an effect of pruning on latency for the HoverNet model. As shown in Fig. 12, about 50% pruned model is about two times faster than the original HoverNet model.Iterative Magnitude Pruning

[0139] To further improve the performance of the pruned, iterative pruning is performed for compressing the HoverNet model. Fig. 13 illustrates a graph 1300 depicting a comparison between an iterative magnitude pruning versus a one shot pruning for a segmentation branch. Fig. 14 illustrates a graph 1400 depicting a comparison between an iterative magnitude pruning versus an one shot pruning for a classification branch. As illustrated in Figs. 13 and 14, with iterative magnitude pruning, the model can be pruned to higher sparsity ratio, while keeping the performance segmentation and classification task similar to the original HoverNet model.Slide artifact classification

[0140] Here experimental results on pruning ResNetl8 adopted for classifying slide pen mark artifacts are demonstrated. With one-shot pruning of the residual blocks independently of each other, even when pruning up to 93.75% of the filters in the first convolution layer, the first batchnormalization layers and the input dimension of the second convolution filters in each residual block (this is referred to as sparsity level 93.75%), it is observed that model performance did not degrade, indicating the prunablity of such models in this use case. Fig. 15A illustrates a graph 1500 depicting classification model performance (e.g., Fl, see DQ metric described above) at various pruning levels. Fig. 15B illustrates a graph 1520 depicting classification model performance (e.g., sensitivity) at various pruning levels. Fig. 15C illustrates a graph 1550 depicting classification model performance (e.g., specificity) at various pruning levels.

[0141] Fig. 16 illustrates a graph 1600 depicting a classification model parameter (e.g., memory footprint) reduction at various pruning levels. Fig. 17 illustrates a graph 1700 depicting a reduction in latency of classification inference at various pruning levels. Model parameters were drastically reduced from about 11.7 million to about 1.4 million at the sparsity level of 93.75%, as illustrated in Fig. 16, which demonstrates a significant decrease of memory footprint (e.g., on disks and during inference). Inference latency was reduced by 30% (see Fig. 17) at the sparsity level of 93.75%. Despite of a smaller percentage of latency reduction compared to that of memory footprint reduction, when the time saving is considered for millions and even more number of image tiles routinely processed by classification models, the absolute time saved with the pruned model growslinearly with the tile number as well as with the more number of pruned models that deployed compared to their original counterparts.Example system

[0142] Fig. 18 is a block diagram illustrating an example overview of a system for processing digital pathology images, where ML models are used for such processing of digital pathology images, and where the ML models may be pruned using techniques described herein. The exemplary system 1800 may include one or more computer systems 1805 connected with an image generation system 1820 through a network 1815. The system 1800 may further include one or more databases 1810 for the processing and storing of data (e.g., histopathology images). Database 1810 may be integral to a memory system on the computer 1805 or in secondary storage such as a hard disk, floppy disk, optical disk, or other non-volatile mass storage devices. The computer 1805 and the databases 1810 may be further connected to one or more communications networks 1815. The computer 1805 may include a client terminal in communication with one or more servers, or personal digital / data assistants (PDA), laptop computers, mobile computers, internet appliances, one or two-way pagers, mobile phones, or other similar desktop, mobile or hand-held electronic devices.

[0143] The computer system 1805 of the exemplary system 1800 includes a processing system with one or more Central Processing Unit(s) (CPU), Graphic Processing Unit(s) (GPU), processors, and / or one or more memories. The computer system 1805 may also include a memory for storing a plurality of processing modules or logical instructions that are executed by the one or more processors coupled. The computer memory that stores data may also be maintained on a computer readable medium including magnetic disks, optical disks, organic memory, and any other Volatile (e.g., Random Access Memory (“RAM)) or non-volatile (e.g., Read-Only Memory (“ROM), flash memory, etc.) mass storage system readable by the CPU. The computer readable medium includes cooperating or interconnected computer readable medium, which exist exclusively on the processing system or can be distributed among multiple interconnected processing systems that may be local or remote to the processing system.

[0144] The communications network 1815 may include, internet, an intranet, a wired Local Area Network (LAN), a wireless LAN (WiLAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), Public Switched Telephone Network (PSTN) and other types ofcommunications networks. The communications network 1815 may include one or more gateways, routers, or bridges. The communications network 1815 may include one or more servers and one or more web-sites accessible by users to send and receive information usable by the one or more computers 1805. The one or more servers may also include one or more associated databases for storing electronic information. The communications network 1815 includes, but is not limited to, data networks using the Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Internet Protocol (IP) and other data protocols.

[0145] Besides processors and memory, the computer system 1805 may also include user input and output devices such as a keyboard, mouse, stylus, and a display / touchscreen. For instance, the computer system 1805 may provide a means for inputting image data from one or more scanned IHC slides to memory. Image data may include data related to color channels or frequency channels. A biological specimen, for example a tissue section, may need to be stained for biomarkers associated with chromogenic stains for brightfield imaging or fluorophores for fluorescence imaging. Staining assays can use chromogenic stains for brightfield imaging, or combinations of organic fluorophores, synthetic fluorophores, or quantum dots for fluorescence imaging. In the analysis of biological specimens, different stains may be specified to identify one or more types of biomarkers.

[0146] In an example, the computer system 1805 may be loaded with a ML model for analysis images received from the image generation system 1820. In an example, the ML model may be trained and / or pruned at another computer system, which may be at least in part similar to, or different from the computer system 1805. In an example, a computer system in which the ML model may be trained and / or pruned may also include a processing system with one or more CPUs, GPUs, processors, and / or one or more memories. Such a computer system may also include a memory for storing a plurality of processing modules or logical instructions that are executed by the one or more processors coupled. The computer memory that stores data may also be maintained on a computer readable medium including magnetic disks, optical disks, organic memory, and any other Volatile (e.g., RAM) or non-volatile (e.g., ROM, flash memory, etc.) mass storage system readable by the CPU. The computer readable medium includes cooperating or interconnected computer readable medium, which exist exclusively on the processing system or can be distributed among multiple interconnected processing systems that may be local or remote to the processing system.

[0147] The term ‘sample’ may be understood as material derived from a biological organism, comprising but not limited to hair, skin samples, tissue samples, cultured cells, cultured cell media, and biological fluids. The term ‘tissue’ refers to a mass of interconnected cells (e.g., lung tissue, neural tissue, or eye tissue) derived from a human or other animal and includes the connecting material and the liquid material in association with the cells. In the context of histopathology, the term “slide” refers to a glass microscope slide carrying a thin section of tissue that has been stained for microscopic examination. The term 'sample’ also includes media containing isolated cells. One skilled in the art may determine the quantity of samples required to obtain a reaction by standard laboratory techniques.

[0148] Fig. 19 shows an exemplary network of a digital pathology image generation system from Fig. 18. Images are generated by an image generation system 1820. A fixation / embedding system 1915 fixes and / or embeds a tissue sample (e.g., a liquid fixing agent, such as formaldehyde solution) and / or an embedding substance (e.g., a historical wax, such as paraffin wax and / or one or more resins, such as styrene or polyethylene). Each slice may be fixed by exposing the slice to a fixating agent for a predefined period of time (e.g., at least 3 hours) and by then dehydrating the slice (e.g., via exposure to an ethanol solution and / or a clearing intermediate agent). The embedding substance can infiltrate the slice when it is in liquid state (e.g., when heated).

[0149] A tissue slicer 1920 then slices the fixed and / or embedded tissue sample (e.g., a sample of a tumor) to obtain a series of sections, with each section having a thickness of, for example, 4- 5 microns. Such sectioning can be performed by first chilling the sample and then slicing the sample in a warm water bath. The tissue can be sliced using (for example) a vibratome or compresstome.

[0150] Because the tissue sections and the cells within them are virtually transparent, preparation of the slides typically includes staining (e.g., automatically staining) the tissue sections to render relevant structures more visible. In some instances, the staining is performed manually. In some instances, the staining is performed semi-automatically or automatically using a staining system 1925.

[0151] The staining can include exposing an individual section of the tissue to one or more different stains (e.g., consecutively, or concurrently) to reveal different characteristics of the tissue. For example, each section may be exposed to a predefined volume of a staining agent for apredefined period of time. The staining agent can include (for example) an RNA probe, protein probe (e.g., nuclear-protein probe or cytoplasm-protein probe), an immunohistochemistry stain, a probe for a secreted substance, etc. In some instances, the staining agent is one that stains for KAPPA mRNA or LAMBDA mRNA.

[0152] One exemplary type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining may be used to indicate general aspects of tissue morphology and / or cell microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to indicate lipid droplets, etc.). One example of a histochemical stain is hematoxylin and eosin (H&E). Other examples of histochemical stains include trichrome stains (e.g., Masson's Trichrome), Periodic Acid-Schiff (PAS), silver stains, and iron stains. The molecular weight of a histochemical staining reagent (e.g., dye) is typically about 500 kilodaltons (kD) or less, although some histochemical staining reagents (e.g., Alcian Blue, phosphomolybdic acid (PMA)) may have molecular weights of up to two or three thousand kD. One case of a high-molecular-weight histochemical staining reagent is alpha-amylase (about 55 kD), which may be used to indicate glycogen.

[0153] Another type of tissue staining is immunohistochemistry (IHC, also called "immunostaining"), which uses a primary antibody that binds specifically to a target antigen of interest (biomarker). IHC may be direct or indirect. In direct IHC, the primary antibody is directly conjugated to a label (e.g., a chromophore or fluorophore). In indirect IHC, the primary antibody is first bound to the target antigen, and then a secondary antibody that is conjugated to a label (e.g., a chromophore or fluorophore) is bound to the primary antibody. The molecular weights of IHC reagents are much higher than those of histochemical staining reagents, as the antibodies have molecular weights of about 150 kD or more.

[0154] The sections may then be individually mounted on corresponding slides, which an imaging system 1930 can then scan to generate raw multiplex digital-pathology images 1935a-n. Each section may be mounted on a slide, which is then scanned to create a digital image that may be subsequently examined by digital pathology image analysis and / or interpreted by a human pathologist (e.g., using image viewer software). The imaging may include capturing bright-field images of the slide section.

[0155] In some instances, a pathologist or other expert may review and manually annotate the digital images of the slides (e.g., tumor area, necrosis, etc.). In some instances, annotation of regions of interest are performed automatically using a computer-vision technique. Some of the digital -pathology images 1935a-n may be used by a color demultiplexing system 300.

[0156] A digital histopathology image (e.g., 1935a) typically includes an array, usually a rectangular matrix, of pixels. Each ‘pixel’ is one picture element and is a digital quantity that is a value that represents some property of the image at a location in the array corresponding to a particular location in the image. Typically, in monochrome tone black and white images the pixel values represent a gray scale value. Pixel values for a digital image typically conform to a specified range. For example, each array element may be one byte (i.e., eight bits) representing pixel values in the range of 0 to 255. In a gray scale image, a 255 may represent absolute white and zero total black (or visa-versa). Color images consist of three-color planes, generally corresponding to red, green, and blue (RGB). For a particular pixel, there is one value for each of these color planes, (i.e., a value representing the red component, a value representing the green component, and a value representing the blue component). By varying the intensity of these three components, all colors in the color spectrum typically may be created.

[0157] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform part or all of one or more methods and / or part or all of one or more processes disclosed herein.

[0158] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention as claimed has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosedmay be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.

[0159] The present description provides preferred exemplary embodiments only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the present description of the preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It is understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0160] Specific details are given in the present description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

CLAIMSWhat is claimed is:

1. A non-transitory computer-readable medium including instructions that when executed by one or more processors, cause the one or more processors to perform operations including: training a machine learning (ML) model; subsequent to training the ML model, identifying a skip connection within the ML model; and pruning one or more fdters of the ML model, while maintaining dimensional integrity of the skip connection within the ML model.

2. The non-transitory computer-readable medium of claim 1, wherein pruning the one or more filters of the ML model while maintaining dimensional integrity of the skip connection comprises: configuring a first layer of the ML model to have a first output including a first number of channels, wherein a second layer and a third layer are in series with the first layer of the ML model; pruning the second layer of the ML model, such that second layer has a second output including a second number of channels, wherein the second number of channels is less than the first number of channels; pruning the third layer of the ML model, such that third layer has a third output including the first number of channels; and summing or concatenating (i) the first output having the first number of channels and (ii) the third output having the first number of channels.

3. The non-transitory computer-readable medium of claim 2, wherein the operations further include: prior to summing or concatenating the first output and the third output, processing each of the first output and the third output, wherein processing the first output comprises prior to summing or concatenating, performing, without altering a dimensionality of a number of channels of the first output, one ormore of (i) an activation operation, (ii) a batch normalization operation, and / or (iii) a convolution operation on the first output; and wherein processing the third output comprises prior to summing or concatenating, performing, without altering a dimensionality of a number of channels of the third output, one or more of (i) another activation operation, (ii) another batch normalization operation, and / or (iii) another convolution operation on the third output.

4. The non-transitory computer-readable medium of claim 2, wherein: the first output by the first layer has N number of channels, where N is a positive integer that is the first number; the second output by the second layer has N / a number of channels, where N / a is the second number, and wherein “a” is a positive integer pruning ratio implemented by the second layer; and the third output by the third layer has the N number of channels.

5. The non-transitory computer-readable medium of claim 4, wherein prior to pruning the ML model, the second output by the second layer has the N number of channels.

6. The non-transitory computer-readable medium of claim 2, wherein the operations further include: prior to summing or concatenating the first output and the third output, processing the third output by a fourth layer, wherein the fourth layer does not change a dimensionality of the first output, such that the output of the fourth layer includes the first number of channels, wherein the first layer, the second layer, the third layer, and the fourth layer implement corresponding convolution operations.

7. The non-transitory computer-readable medium of claim 2, wherein the operations further include: subsequent to summing or concatenating the first output and the third output, providing a summed or concatenated output to a fourth layer of the ML model.

8. The non-transitory computer-readable medium of claim 2, wherein: the skip connecting is from the first layer to a layer performing the summing or concatenating; and maintaining the dimensional integrity of the skip connection within the ML model comprising ensuring that the first output and the third output being summed or concatenated have the same first number of channels.

9. The non-transitory computer-readable medium of claim 2, wherein pruning the one or more filters of the ML model comprises: configuring a fourth layer of the ML model to receive a combined output, wherein the combined output is the sum or concatenation of the first output and the third output, and wherein the combined output has a third number of channels; pruning the fourth layer such that the fourth layer has a fourth output including a fourth number of channels that is less than the third number of channels, wherein a fifth layer is in series with the fourth layer of the ML model; pruning the fifth layer of the ML model, such that fifth layer has a fifth output including the third number of channels; and summing or concatenating (i) the combined output having the third number of channels and (ii) the fifth output having the third number of channels.

10. The non-transitory computer-readable medium of claim 2, wherein the operations further include: identifying the second layer of the ML model as a first candidate for pruning, so as to reduce a size of the ML model; identifying the third layer of the ML model as a second candidate for pruning, so as to maintain the dimensional integrity of the skip connection within the ML model; calculating a first pruning ratio for the second layer; calculating a second pruning ratio for the third layer, the second pruning ratio is calculated based on the first pruning ratio, such that the third layer reverses the channel reduction achieved by the second layer; andpruning one or more filters of the ML model comprises pruning the second layer and the third layer in accordance with the first pruning ration and the second pruning ration, respectively.

11. The non-transitory computer-readable medium of claim 2, wherein: summing or concatenating the first output and the third output comprises summing, in a summer, the first output and the third output; the second layer, the third layer, and the summer are within a residual unit of the ML model; and the first output from the first layer forms an input to the residual unit.

12. The non-transitory computer-readable medium of claim 2, wherein: summing or concatenating the first output and the third output comprises concatenating, in a concatenation layer, the first output and the third output; the ML model is a U-Net model; the first layer, the second layer, and the third layer are within an encoder of the U-Net model; and the concatenation layer is within a decoder of the U-Net model.

13. The non-transitory computer-readable medium of claim 1, wherein the operations further include: subsequent to pruning the one or more filters of the ML model, fine tuning the ML model by at least in part retraining the ML model.

14. The non-transitory computer-readable medium of claim 1, wherein the ML model is usable to process digital pathology images.

15. The non-transitory computer-readable medium of claim 1, wherein the ML model has one of a HoVer-Net architecture, a ResNet architecture, or a U-Net architecture.

16. A method compri sing : training a machine learning (ML) model;subsequent to training the ML model, identifying a skip connection within the ML model; and pruning one or more filters of the ML model, while maintaining dimensional integrity of the skip connection within the ML model.

17. The method of claim 16, wherein pruning the one or more filters of the ML model while maintaining dimensional integrity of the skip connection comprises: configuring a first layer of the ML model to have a first output including a first number of channels, wherein a second layer and a third layer are in series with the first layer of the ML model; pruning the second layer of the ML model, such that second layer has a second output including a second number of channels, wherein the second number of channels is less than the first number of channels; pruning the third layer of the ML model, such that third layer has a third output including the first number of channels; and summing or concatenating (i) the first output having the first number of channels and (ii) the third output having the first number of channels.

18. The method of claim 16, further comprising: using the pruned ML model to process digital pathology images.

19. A system comprising: one or more data processors; and a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform a set of actions including: training a machine learning (ML) model; subsequent to training the ML model, identifying a skip connection within the ML model; and pruning one or more filters of the ML model, while maintaining dimensional integrity of the skip connection within the ML model.

20. The system of claim 19, wherein the set of actions further comprises: configuring a first layer of the ML model to have a first output including a first number of channels, wherein a second layer and a third layer are in series with the first layer of the ML model; pruning the second layer of the ML model, such that second layer has a second output including a second number of channels, wherein the second number of channels is less than the first number of channels; pruning the third layer of the ML model, such that third layer has a third output including the first number of channels; and summing or concatenating (i) the first output having the first number of channels and (ii) the third output having the first number of channels.

Images