Neural Networks for Identifying Regions of Interest in Luminescence Images

Abstract

1. A computer-implemented method comprising: receiving a pair of images of an object, the pair including an image of the object and an luminescent image of the object; using a first machine learning model to generate, from the image of the object, coordinates of a bounding box around the object in the image; extracting a pair of cropped images, the pair including a cropped image of the object and a cropped luminescent image of the object, based on the coordinates of the bounding box around the object; and using a second machine learning model to generate boundary shape parameters that identify a region of interest, the second machine learning model being a neural network including one or more convolutional layers configured to generate a first feature map that encodes features of the cropped image and a second feature map that encodes features of the cropped luminescent image; and one or more fully connected layers configured to convert the first feature map and the second feature map into boundary shape parameters that identify the region of interest.

Description

[Technical field]

[0001] The present specification relates to the use of machine learning models to identify regions of interest in luminescent images, such as bioluminescent or fluorescent images. [Background technology]

[0002] Bioluminescence imaging (BLI) and fluorescence imaging (FLI) are techniques for in vivo imaging that allow non-invasive study of ongoing biological processes, such as tumor evolution, drug efficacy, cell tracking, target engagement, etc. BLI and FLI have broad applications and can be used, for example, in the study of cancer treatment in disseminated models such as acute myeloid lymphoma with typical lesions in bone (e.g., long bones, vertebrae, skull), immuno-oncology, tracking of T cells and NK cells, and understanding mRNA delivery. These techniques can also be used in cell imaging and other in vitro imaging techniques. Summary of the Invention [Problem to be solved by the invention]

[0003] Currently, regions of interest (ROIs) are manually drawn on images using dedicated software to quantify signal intensity and evaluate the biological activity of drugs on transfected and / or labeled cells. This is a very time-consuming process for large-scale studies with potential inter-operator variability accuracy in ROI identification, especially when multiple people perform the analysis of images. Therefore, there is a need to provide an accurate method of identifying regions of interest in luminescent images that is less time-consuming and provides more consistent identification of ROIs. [Means for solving the problem]

[0004] According to a first aspect of the present specification, there is provided a computer-implemented method for identifying one or more regions of interest in an image, the method comprising: receiving a pair of images of the object, the pair including an image of the object and a luminescence image of the object; generating, from an image of the object, coordinates of a bounding box around the object in the image using a first machine learning model; extracting a pair of cropped images from an image of the object and an illuminating image of the object based on coordinates of a bounding box around the object, the pair of images including the cropped image of the object and the cropped illuminating image of the object; and generating boundary shape parameters from the pair of cropped images using a second machine learning model that identifies the region of interest; The second machine learning model is one or more convolutional layers configured to generate a first feature map encoding features of the cropped image and a second feature map encoding features of the cropped luminescent image; and one or more fully connected layers configured to convert the first feature map and the second feature map into boundary shape parameters that identify the region of interest. The present invention relates to a method for generating a neural network comprising:

[0005] The method may further include, after applying the first neural network, using a thresholding algorithm to generate a mask that identifies a contour of the object within the bounding box.

[0006] The second machine learning model may further include one or more residual connections. The parameters of the boundary shape may include parameters of an ellipse or a circle. The parameters of the ellipse may be: (i) x-axis location of the center point; (ii) y-axis location of the center point; (iii) length of the semi-minor axis; (iv) length of the semi-major axis; and (v) rotation angle of the ellipse from vertical. The subject may be an animal, such as a mouse.

[0007] The method may further include selecting a region of interest category from the list of anatomical regions, wherein the second machine learning model corresponds to the selected region of interest category.

[0008] The method may include checking whether the image and the luminescent image are of a predetermined size before generating the first feature map and the second feature map, and if not, resizing the image and the luminescent image to the predetermined size.

[0009] The method may further include displaying the image of the object and / or the luminescence image of the object with the indicated region of interest.

[0010] The image may be a photographic image of the object.The method may further include concatenating the first feature map and the second feature map prior to transforming the first feature map and the second feature map.

[0011] According to a second aspect of the present specification, there is provided a computer-implemented method for training a model for identifying one or more regions of interest in an image, the method comprising: For each of a plurality of training samples, each training sample includes: a training pair including an image of the subject and a luminescence image of the subject; and ground truth parameters of a boundary shape that identify a region of interest of the subject: generating, from each training pair, candidate boundary shape parameters that identify the region of interest using a neural network; and Comparing the candidate parameters of the boundary shape with corresponding ground truth parameters of the boundary shapes of the training samples; and updating parameters of the neural network based on the comparison; The neural network: one or more convolutional layers configured to generate a first feature map that encodes image features and a second feature map that encodes luminescence image features; and one or more fully connected layers configured to convert the first feature map and the second feature map into candidate parameters of a boundary shape that identifies the region of interest. A method is described, including:

[0012] Comparing the candidate parameters of the boundary shapes with corresponding ground truth parameters of the boundary shapes of the training samples may include evaluating a loss function; and updating the parameters of the neural network based on the comparison may include applying an optimization routine to the loss function.

[0013] According to a third aspect of the present specification, a computer program product is described comprising computer readable code which, when executed by a computing system, causes the computing system to perform a method according to the first or second aspect.

[0014] According to a third aspect of the present specification, a system is described that includes one or more processors and a memory, the memory storing computer readable instructions that, when executed by the one or more processors, cause the system to perform a method according to the first or second aspect.

[0015] Embodiments will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0016] [Figure 1] 1 shows a schematic diagram of an exemplary method for identifying a region of interest in a luminescence image. [Diagram 2] FIG. 1 shows a schematic diagram of a convolutional neural network for generating a feature map from an input image. [Figure 3A] Shown is a photographic image containing five mice with annotated regions of interest. [Figure 3B] Bioluminescence images of the same five mice as in FIG. 3B are shown. [Figure 4] 4 illustrates the result of an additional contouring step that may be performed on the image. [Diagram 5] 1 shows a flowchart of an exemplary method for identifying a region of interest in a luminescent image. [Figure 6]1 shows a flowchart of an exemplary method for training a model for identifying a region of interest in a luminescent image. [Figure 7] 1 shows a schematic example of a system / apparatus for carrying out any of the methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] FIG. 1 shows a schematic diagram of a method 100 for identifying a region of interest in a luminescent image. The term luminescent image is used herein to mean an image that includes at least one object exhibiting some form of luminescence, such as, but not limited to, bioluminescence or fluorescence. A first input image 102 is input to a first machine learning model 104. The first input image may include pixel data arranged in an array and may include one or more color channels (e.g., RBG channels or grayscale color channels). The first input image 102 may be, for example, a grayscale digital photograph.

[0018] The first input image 102 includes one or more objects to be analyzed. In the example of FIG. 1, the first input image 102 is of several mice injected with a bioluminescent or fluorescent substance, with each object being a single mouse. However, the techniques described herein are applicable to many different scenarios and a wide range of organisms, such as plants, animals, fungi, bacteria, and viruses. In some examples, images of individual mice can be provided as input images.

[0019] The first machine learning model 104 processes the first input image 102 to determine bounding boxes for one or more objects in the first input image 102. The first machine learning model 104 may be any suitable type of machine learning bounding box model. The first machine learning model 104 may be a fine-tuned Faster-RCNN in some examples. The initial weights of the model may be obtained by training on the COCO dataset. The weights may then be updated by training on a more specific dataset. The first input image 102 may be a grayscale photograph with pixel values ​​scaled between 0 and 1 prior to model computation. The first machine learning model 104 may output a result image 106 in which each object in the image is identified with a rectangular bounding box. Alternatively, the first machine learning model 104 may output coordinates of one or more bounding boxes in the first input image 102, each bounding box corresponding to one object in the first input image. The coordinates of each bounding box generated by the first machine learning model 104 are used to extract a pair of cropped images (108a, 108b). The pair of cropped images (108a, 108b) includes a cropped image of the object 108a and a cropped luminescent image of the object 108b. The cropped image 108a of the object is cropped from the first input image 102.

[0020] A cropped luminescence image of the object 108b is obtained by applying the coordinates of the bounding box generated by the first machine learning model 104 to a second input image (not shown) of the same object captured by a high-sensitivity CCD camera capable of detecting bioluminescence or fluorescence signals.

[0021] The pair of cropped images (108a, 108b) are then input to a machine learning model 110 that outputs parameters of the boundary shape. The machine learning model 110 is a neural network with one or more convolutional layers. In some embodiments, such as the illustrated embodiment, separate heads are used to input the cropped image of the object 108a and the cropped luminescent image of the object 108b. The cropped image of the object 108a may be processed using a first sub-network, and the cropped luminescent image of the object 108b may be processed by a second sub-network.

[0022] In some implementations, the cropped images 108a, 108b are resized / rescaled to a predefined size before being input to the machine learning model 110. Typically, the bounding box of an object may vary in size, for example depending on the size and pose of the object. As a result, the size of the cropped images may change. However, the machine learning model 110, in some embodiments, can only accept inputs of a predefined fixed size (e.g., n×m pixels). Thus, the cropped images 108a, 108b are resized to this fixed input size. Interpolation techniques can be used to perform the rescaling and fill in missing pixel data.

[0023] The machine learning model 110 can be specific to a particular anatomical feature of the subject in the image. For example, multiple different machine learning models 110 can be utilized that are trained to identify regions of interest in the head, vertebrae, or hind limbs of the subject, or specifically the subject's internal organs (e.g., liver, lungs, spleen, etc.). Part of the method can include selecting a particular machine learning model to use that corresponds to the anatomical feature of the subject. Alternatively, the machine learning model can be generally applicable and configured to identify all regions of interest in the luminescent image, regardless of where they are located on the subject.

[0024] Each sub-network of the machine learning model 110 may include a ResNet backbone, such as a ResNet18 backbone (i.e., 18 convolutional blocks with 3×3 filters with residual connections between them) as shown in FIG. 2. The machine learning model 110 generates a first feature map that encodes features of the cropped image and a second feature map that encodes features of the cropped luminescence image. The first and second feature maps may be, for example, 512-d feature maps each, although it will be understood that many other feature map dimensions are possible.

[0025] The machine learning model 110 further comprises one or more fully connected layers configured to convert the first feature map and the second feature map into boundary shape parameters that identify the region of interest. In this way, information from both the photographic image of the object and the luminescent image of the object is taken into account in identifying the region of interest. Before being converted into boundary shape parameters, the first and second feature maps may be combined, for example by concatenation or stacking. If the first and second feature maps are 512-d feature maps, the concatenated feature map may be a 1024-d feature map.

[0026] The boundary shape is defined by parameters output by one or more fully connected layers of the machine learning model 110. If there are several different machine learning models corresponding to different anatomical features of the subject, a different boundary shape can be output for each model. In some examples, the boundary shape is an ellipse. An ellipse can be described by the following five parameters: (i) x-axis location of the center point; (ii) y-axis location of the center point; (iii) the length of the semi-minor axis; (iv) the length of the semi-major axis; and (v) the rotation angle of the ellipse from the vertical. However, it will be understood that there are other ways of defining an ellipse that can be used.

[0027] The boundary shape can then be displayed on the cropped image of object 108a, on the cropped luminescent image of object 108b, and / or on a composite image showing both the overlaid photographic and luminescent paired images, as shown in output image 112 of FIG. 1.

[0028] 2 shows a schematic diagram of a convolutional neural network 200 for generating a feature map from an input image. As described above, the cropped image of the object 108a and the cropped luminescence image of the object 108b may each be input to a sub-network including a convolutional neural network, and then one or more fully connected layers of the neural network convert the resulting feature map into parameters of a boundary shape that identifies the region of interest. The sub-network may be, for example, the convolutional neural network 200 shown in FIG. 2.

[0029] The convolutional neural network 200 includes a series of convolutional layers 202, each configured to apply a respective plurality of convolutional filters to a respective input. In the illustrated example, each convolutional filter is a 3×3 convolutional filter, although it will be appreciated that convolutional filters of other sizes may alternatively or additionally be applied.

[0030] The convolutional neural network 200 further includes one or more skip connections 204A-H (also called "residual connections"). The skip connections 204A-H take the output of a convolutional layer and use it as part of the input to a subsequent non-successive layer of the network, i.e., skipping one or more convolutional layers. In some embodiments, the input of the convolutional layer at the end of a skip connection includes a combination of the output of the immediately preceding layer and the output linked by the skip connection. For example, the output of the immediately preceding layer may be added to the output connected by the skip connection.

[0031] The convolutional neural network 200 may further include one or more average pooling layers 206, which generate lower dimensional outputs from their inputs by averaging over patches of those inputs.

[0032] FIG. 3a shows a photographic image 300 containing five mice, and FIG. 3b shows a luminescent image 302 of the same five mice. The results of the first and second machine learning models are shown in the images. In particular, a bounding box is generated for each mouse, and two different regions of interest (along the vertebrae and the head) are identified with ellipses for each mouse. The bounding boxes are omitted in FIG. 3b. The second machine learning model takes input pair images of a single subject, but can reconstruct the original arrangement of multiple subjects after parameters of the boundary shape identifying the regions of interest are output for each subject. This can be advantageous when the subjects are all processed in the same way and a visual side-by-side comparison of the subjects is potentially beneficial.

[0033] FIG. 4 shows the results of additional contouring steps that may be performed to improve the accuracy with which regions of interest can be identified. After the first machine learning model generates the coordinates of a bounding box around an object in the image, additional processing steps may be performed to obtain the contour of the object in the image. A thresholding algorithm is applied to generate a mask. This mask is refined by a series of morphological operations that may include one or more of removing small objects, morphological closing, and filling small holes. FIG. 4 also shows a comparison of the manual annotations (green) with the regions of interest identified by the methods described herein, as well as additional contours (red).

[0034] The additional false contour information is not passed to the second machine learning model, but is displayed and used in downstream analysis by the scientist in the same way as the generated ellipses. The false contour information can be thought of as an additional ROI.

[0035] 5 shows a flowchart of an exemplary method for identifying one or more regions of interest in an image. The method can be performed by a computing system such as the system described in connection with FIG.

[0036] In step 500, a pair of images of an object is received. The pair of images includes an image of the object and a luminescence image of the same object. These two images may have been captured simultaneously or successively at any time before the method is performed. The object in the image and the luminescence image may be a living organism.

[0037] In step 502, a first machine learning model is used to generate bounding box coordinates around an object in an image. This step is performed using an image of the object. The bounding box coordinates can then be applied to both the image of the object and the luminescence image of the object to generate a pair of images of the same object. This step is primarily to limit the image region input to the second machine learning model to the object of interest. However, this step may also be used to separate an input image containing multiple objects into multiple images each containing one object. As shown in FIG. 3, an original image can contain multiple objects. Such an image can be input to a first machine learning model, which generates a bounding box around each object in the image.

[0038] Optionally, an additional step may be performed between steps 502 and 504, using a thresholding algorithm to generate a mask that identifies the contours of the object within the bounding box. Examples of thresholding algorithms that may be used are described in "Minimum Cross Entropy Thresholding" (Li CH and Lee CK (1993) Pattern Recognition, 26(4):617-625) and "An Iterative Algorithm for Minimum Cross Entropy Thresholding" (Li CH and Tam PKS (1998) Pattern Recognition Letters, 18(8):771-776), the contents of which are incorporated herein by reference.

[0039] In step 504, a pair of cropped images is extracted from the image of the object and the luminescent image of the object. The extraction is based on the coordinates of a bounding box around the object. Thus, the pair of images includes a cropped image of the object and a cropped luminescent image of the object.

[0040] In step 506, a second machine learning model is used to generate boundary shape parameters that identify the region of interest from the pair of cropped images. The second machine learning model is a neural network that includes: (i) one or more convolutional layers configured to generate a first feature map that encodes features of the cropped image and a second feature map that encodes features of the cropped luminescent image; and (ii) one or more fully connected layers configured to convert the first feature map and the second feature map into boundary shape parameters that identify the region of interest.

[0041] 6 shows a flowchart of an exemplary method for training a model for identifying regions of interest in a luminescent image, the method being capable of being performed by a computing system such as the system described in connection with FIG.

[0042] In step 600, one or more training samples are received, each training sample including a training pair including an image of an object and an emission image of the object, and ground truth parameters of boundary shapes that identify regions of interest of the object. The raw training images may include several objects per image, with each object in the image having a bounding box and one or more regions of interest identified. As a precursor to step 600, multiple objects in an image may be separated such that the model is trained on a single object of boundary shapes and associated ground truth parameters at a time.

[0043] At step 602, a neural network is used to generate candidate parameters of boundary shapes that identify the region of interest from each training pair. The neural network is described above in relation to Figures 1 and 2 and includes one or more convolutional layers configured to generate a first feature map encoding image features and a second feature map encoding luminescence image features, and one or more fully connected layers configured to convert the first feature map and the second feature map into candidate parameters of boundary shapes that identify the region of interest.

[0044] In step 604, the candidate parameters of the boundary shape are compared to corresponding ground truth parameters of the boundary shapes of the training samples. Comparing the candidate parameters of the boundary shape to the corresponding ground truth parameters of the boundary shapes of the training samples may include evaluating a loss function, such as an L2 or L1 loss, between the candidate parameters and the corresponding ground truth parameters.

[0045] Steps 602 and 604 are repeated across one or more training samples until candidate boundary shape parameters for each training sample have been determined.

[0046] In step 606, parameters of the neural network are updated based on the comparison. Updating the parameters of the neural network based on the comparison may include applying an optimization routine to the loss function. Such an optimization routine may include, but is not limited to, a stochastic gradient descent routine. Steps 600 through 606 are iterated over the training data set until a threshold condition is met. The threshold condition may be a threshold number of training epochs and / or a threshold performance on a validation data set.

[0047] 7 shows a schematic example of a system / apparatus 700 for performing any of the methods described herein. The illustrated system / apparatus is an example of a computing device. Those skilled in the art will appreciate that other types of computing devices / systems, such as, for example, distributed computing systems, may alternatively be used to perform the methods described herein.

[0048] The device (or system) 700 includes one or more processors 702. The one or more processors control the operation of other components of the system / device 700. The one or more processors 702 may include, for example, a general purpose processor. The one or more processors 702 may be single-core or multi-core devices. The one or more processors 702 may include a central processing unit (CPU) or a graphical processing unit (GPU). Alternatively, the one or more processors 702 may include specialized processing hardware, such as a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

[0049] The system / apparatus includes a working or volatile memory 704. One or more processors may access the volatile memory 704 to process data and may control data storage in the memory. The volatile memory 704 may include any type of RAM, such as static RAM (SRAM), dynamic RAM (DRAM), or it may include flash memory, such as an SD-card.

[0050] The system / apparatus includes a non-volatile memory 706. The non-volatile memory 706 stores a set of operating instructions 708 in the form of computer readable instructions for controlling the operation of the processor 702. The non-volatile memory 706 can be any type of memory, such as a read only memory (ROM), flash memory, or magnetic drive memory.

[0051] The one or more processors 702 are configured to execute operational instructions 708 to cause the system / device to perform any of the methods described herein. The operational instructions 708 may include code for the hardware components of the system / device 700 (i.e., drivers), as well as code for the basic operation of the system / device 700. Typically, the one or more processors 702 execute one or more instructions of the operational instructions 708, which are stored permanently or semi-permanently in the non-volatile memory 706, and the volatile memory 704 is used to temporarily store data generated during execution of the operational instructions 708.

[0052] Implementations of the methods described herein may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof, which may include a computer program product (e.g., software stored on a magnetic disk, optical disk, memory, programmable logic device, etc.) containing computer readable instructions that, when executed by a computer, cause the computer to perform one or more of the methods described herein, such as those described with respect to FIG.

[0053] System features described herein may also be provided as method features, and vice versa. As used herein, means-plus-function features may be expressed in terms of their corresponding structure. In particular, method aspects may also apply to system aspects, and vice versa.

[0054] Furthermore, any, some, and / or all of the features in one embodiment may apply to any, some, and / or all of the features in any other embodiment, in any appropriate combination. It is also to be understood that specific combinations of the various features described and defined in any embodiment of the invention may be implemented and / or provided and / or used separately.

[0055] While several embodiments have been shown and described, those skilled in the art will understand that changes can be made to those embodiments without departing from the principles of the present disclosure, the scope of which is defined in the claims or equivalents thereof.

[0056] As used herein, the terms "drug" or "medicament" are used interchangeably to refer to a pharmaceutical formulation containing one or more active pharmaceutical ingredients or pharma- ceutically acceptable salts or solvates thereof and, optionally, a pharma- ceutically acceptable carrier. An active pharmaceutical ingredient ("API"), in its broadest sense, is a chemical structure that exerts a biological effect on humans or animals. In pharmacology, drugs or agents are used to treat, cure, prevent, or diagnose disease or otherwise enhance physical or mental comfort. Drugs or agents may be used for a limited period of time or on a regular basis for chronic conditions.

[0057] As described below, a drug or medicament may include at least one API or a combination thereof in various types of formulations for the treatment of one or more diseases. Examples of APIs may include small molecules with a molecular weight of 500 Da or less; polypeptides, peptides and proteins (e.g., hormones, growth factors, antibodies, antibody fragments and enzymes); carbohydrates and polysaccharides; and nucleic acids, double-stranded or single-stranded DNA (including naked DNA and cDNA), RNA, antisense nucleic acids such as antisense DNA and RNA, small interfering RNA (siRNA), ribozymes, genes and oligonucleotides. Nucleic acids may be incorporated into molecular delivery systems such as vectors, plasmids or liposomes. Mixtures of one or more drugs are also contemplated.

[0058] The drug or agent may be placed in a primary package or "drug container" adapted for use with the drug delivery device. The drug container may be, for example, a cartridge, syringe, reservoir, or other sturdy or flexible vessel configured to provide a suitable chamber for storage (e.g., short-term or long-term storage) of one or more drugs. For example, in some cases, the chamber may be designed to store the drug for at least one day (e.g., from one day to at least 30 days). In some examples, the chamber may be designed to store the drug for about one month to about two years. Storage may be at room temperature (e.g., about 20°C) or at refrigerated temperatures (e.g., from about -4°C to about 4°C). In some cases, the drug container may be or include a dual-chamber cartridge configured to separately store two or more components of a pharmaceutical formulation to be administered (e.g., an API and a diluent or two different drugs), one in each chamber. In such cases, the two chambers of the dual-chamber cartridge may be configured to allow mixing between the two or more components before and / or during administration into the human or animal body. For example, the two chambers may be configured such that they are in fluid communication with one another (e.g., by a conduit between the two chambers) and may allow a user to mix the two components if desired prior to administration. Alternatively or additionally, the two chambers may be configured to allow mixing upon administration of the components into the human or animal body.

[0059] The drugs or agents contained in the drug delivery device as described herein may be used for the treatment and / or prevention of many different types of medical disorders. Examples of disorders include, for example, diabetes mellitus or complications associated with diabetes mellitus, such as diabetic retinopathy, thromboembolism, such as deep vein thromboembolism or pulmonary thromboembolism. Further examples of disorders include acute coronary syndrome (ACS), angina pectoris, myocardial infarction, cancer, macular degeneration, inflammation, hay fever, atherosclerosis and / or rheumatoid arthritis. Examples of APIs and drugs are those listed in handbooks such as the Rote Liste 2014, for example, but not limited to, main group 12 (antidiabetic drugs) or 86 (oncology drugs) and the Merck Index, 15th edition.

[0060] Examples of APIs for the treatment and / or prevention of type 1 or type 2 diabetes mellitus or complications associated with type 1 or type 2 diabetes mellitus include insulin, e.g. human insulin or a human insulin analog or derivative, glucagon-like peptide (GLP-1), a GLP-1 analog or a GLP-1 receptor agonist or an analog or derivative thereof, a dipeptidyl peptidase-4 (DPP4) inhibitor or a pharma-ceutically acceptable salt or solvate thereof or any mixture thereof. As used herein, the terms "analog" and "derivative" refer to a polypeptide having a molecular structure that is formally derivable from the structure of a naturally occurring peptide, e.g. the structure of human insulin, by deletion and / or replacement of at least one amino acid residue present in the naturally occurring peptide and / or addition of at least one amino acid residue. The added and / or replaced amino acid residues can be either codable amino acid residues or other naturally occurring residues or purely synthetic amino acid residues. Insulin analogs are also referred to as "insulin receptor ligands". In particular, the term "derivative" refers to a polypeptide having a molecular structure formally derivable from that of a naturally occurring peptide, such as the structure of human insulin, in which one or more organic substituents (e.g., fatty acids) are attached to one or more of the amino acids. Optionally, one or more amino acids present in the naturally occurring peptide may be deleted and / or substituted by other amino acids, including non-codable amino acids, or amino acids, including non-codable amino acids, may be added to the naturally occurring peptide.

[0061] Examples of insulin analogues are Gly(A21), Arg(B31), Arg(B32) human insulin (insulin glargine); Lys(B3), Glu(B29) human insulin (insulin glulisine); Lys(B28), Pro(B29) human insulin (insulin lispro); Asp(B28) human insulin (insulin aspart); human insulin in which the proline in position B28 can be replaced by Asp, Lys, Leu, Val or Ala and in position B29 Lys can be replaced by Pro; Ala(B26) human insulin; Des(B28-B30) human insulin; Des(B27) human insulin and Des(B30) human insulin.

[0062] Examples of insulin derivatives are e.g. B29-N-myristoyl-des(B30) human insulin, Lys(B29)(N-tetradecanoyl)-des(B30) human insulin (insulin detemir, Levemir®); B29-N-palmitoyl-des(B30) human insulin; B29-N-myristoyl human insulin; B29-N-palmitoyl human insulin; B28-N-myristoyl-LysB28ProB29 human insulin; B28-N-palmitoyl-LysB28ProB29 human insulin; B30-N-myristoyl-ThrB29LysB30 human insulin. B29-N-palmitoyl-ThrB29LysB30 human insulin; B29-N-(N-palmitoyl-gamma-glutamyl)-des(B30) human insulin, B29-N-omega-carboxypentadecanoyl-gamma-L-glutamyl-des(B30) human insulin (insulin degludec, Tresiba®); B29-N-(N-lithocholyl-gamma-glutamyl)-des(B30) human insulin; B29-N-(ω-carboxyheptadecanoyl)-des(B30) human insulin and B29-N-(ω-carboxyheptadecanoyl) human insulin.

[0063] Examples of GLP-1, GLP-1 analogues and GLP-1 receptor agonists are e.g. lixisenatide (Lyxumia®), exenatide (exendin-4, Byetta®, Bydureon®, a 39 amino acid peptide produced by the salivary glands of the Gila monster), liraglutide (Victoza®), semaglutide, taspoglutide, albiglutide (Syncria®), dulaglutide (Trulicity®), rExendin-4, CJC-1134-PC, PB-1023, TTP-054, langrenatide / HM-11260C (efpegrenatide), HM-15211, CM-3, GLP-1 Erigen, ORMD-0901, NN-9423, NN-9709, NN-9924, NN-9926, NN-9927, Nodexene, Viador-GLP-1, CVX-096, ZYOG-1, ZYD-1, GSK-2374697, DA-3091, MAR-701, MAR709, ZP-2929, Z P-3022, ZP-DI-70, TT-401 (pegapamoditide), BHM-034, MOD-6030, CAM-2036, DA-15864, ARI-2651, ARI-2255, tirzepatide (LY3298176), bamadutide (SAR425899), exenatide-XTEN and glucagon-Xten.

[0064] An example of an oligonucleotide is, for example, mipomersen sodium (Kynamro®), a cholesterol-lowering antisense therapeutic for the treatment of familial hypercholesterolemia, or RG012 for the treatment of Alport Syndrome.

[0065] Examples of DPP4 inhibitors include linagliptin, vildagliptin, sitagliptin, denagliptin, saxagliptin, and berberine.

[0066] Examples of hormones include pituitary or hypothalamic hormones or regulatory active peptides such as gonadotropins (follitropin, lutropin, chorion gonadotropin, menotropin), somatropin (somatropin), desmopressin, terlipressin, gonadorelin, triptorelin, leuprorelin, buserelin, nafarelin and goserelin, and their antagonists.

[0067] Examples of polysaccharides include glycosaminoglycans, hyaluronic acid, heparin, low molecular weight heparin or very low molecular weight heparin or derivatives thereof, or sulfated polysaccharides, such as polysulfated forms of the above polysaccharides and / or pharmaceutically acceptable salts thereof.An example of a pharmaceutically acceptable salt of polysulfated low molecular weight heparin is enoxaparin sodium.An example of a hyaluronic acid derivative is Hylan GF 20 (Synvisc®), sodium hyaluronate.

[0068] The term "antibody" as used herein refers to an immunoglobulin molecule or an antigen-binding portion thereof. Examples of antigen-binding portions of an immunoglobulin molecule include F(ab) and F(ab')2 fragments that retain the ability to bind to an antigen. An antibody can be a polyclonal antibody, a monoclonal antibody, a recombinant antibody, a chimeric antibody, a deimmunized or humanized antibody, a fully human antibody, a non-human (e.g., murine) antibody, or a single chain antibody. In some embodiments, an antibody has effector functions and can fix complement. In some embodiments, an antibody has reduced or no binding ability to Fc receptors. For example, an antibody can be an isotype or subtype, an antibody fragment, or a mutant that does not support binding to Fc receptors, e.g., with a mutation or deletion of the Fc receptor binding region. The term antibody also includes antigen-binding molecules based on tetravalent bispecific tandem immunoglobulins (TBTIs) and / or dual variable region antibody-like binding proteins with crossover binding region orientation (CODV).

[0069] The term "fragment" or "antibody fragment" refers to a polypeptide (e.g., antibody heavy and / or light chain polypeptide) derived from an antibody polypeptide molecule that does not include a full-length antibody polypeptide but comprises at least a portion of a full-length antibody polypeptide that is still capable of binding to an antigen. An antibody fragment can include truncated portions of a full-length antibody polypeptide, but the term is not limited to such truncated fragments. Antibody fragments useful in the present invention include, for example, Fab fragments, F(ab')2 fragments, scFv (single-chain Fv) fragments, linear antibodies, monospecific or multispecific antibody fragments, such as bispecific, trispecific, tetraspecific and multispecific antibodies (e.g., diabodies, triabodies, tetrabodies), monovalent or multivalent antibody fragments, such as bivalent, trivalent, tetravalent and multivalent antibodies, minibodies, chelating recombinant antibodies, tribodies or bibodies, intrabodies, nanobodies, small modular immunopharmaceuticals (SMIPs), binding domain immunoglobulin fusion proteins, camelized antibodies and VHH-containing antibodies. Additional examples of antigen-binding antibody fragments are known in the art.

[0070] The term "complementarity determining region" or "CDR" refers to short polypeptide sequences in the variable regions of both heavy and light chain polypeptides that are primarily responsible for mediating specific antigen recognition. The term "framework region" refers to amino acid sequences in the variable regions of both heavy and light chain polypeptides that are not CDR sequences, but are primarily responsible for maintaining the proper arrangement of the CDR sequences to allow antigen binding. Although the framework region itself typically does not directly participate in antigen binding, as is known in the art, certain residues in the framework region of a particular antibody may be directly involved in antigen binding or may affect the ability of one or more amino acids in the CDR to interact with the antigen. Examples of antibodies are anti-PCSK-9 mAbs (e.g., alirocumab), anti-IL-6 mAbs (e.g., sarilumab), and anti-IL-4 mAbs (e.g., dupilumab).

[0071] Pharmaceutically acceptable salts of any of the APIs described herein are contemplated for use in the drug or pharmaceutical preparation in the drug delivery device. Pharmaceutically acceptable salts include, for example, acid addition salts and base salts.

[0072] It will be understood by those skilled in the art that modifications (additions and / or deletions) may be made to the various components of the APIs, formulations, devices, methods, systems and embodiments described herein without departing from the full scope and spirit of the invention, and that the invention encompasses such modifications and any and all equivalents thereof.

[0073] Exemplary drug delivery devices may include needle-based injection systems as described in Table 1 of Section 5.2 of ISO11608-1:2014(E). As described in ISO11608-1:2014(E), needle-based injection systems may be broadly categorized into multi-dose container systems and single-dose (partial or full discharge) container systems. The containers may be replaceable or integrated non-replaceable containers.

[0074] As further described in ISO11608-1:2014(E), a multi-dose container system may include a needle-based injection device with a replaceable container. In such a system, each container holds multiple doses and may be fixed or variable (pre-set by the user) in size. Other multi-dose container systems may include a needle-based injection device with an integrated non-replaceable container. In such a system, each container holds multiple doses and may be fixed or variable (pre-set by the user) in size.

[0075] As further described in ISO11608-1:2014(E), the single-dose container system may include a needle-based injection device with a replaceable container. In one example of such a system, each container holds a single dose and the entire deliverable volume is expelled (full expelled). In another example, each container holds a single dose and a portion of the deliverable volume is expelled (partial expelled). As also described in ISO11608-1:2014(E), the single-dose container system may include a needle-based injection device with an integrated non-replaceable container. In one example of such a system, each container holds a single dose and the entire deliverable volume is expelled (full expelled). In a further example, each container holds a single dose and a portion of the deliverable volume is expelled (partial expelled).

Claims

1. 1. A computer-implemented method for identifying one or more regions of interest in an image, the method comprising: receiving a pair of images of the object including an image of the object and a luminescence image of the object; generating, from the image of the object, coordinates of a bounding box around the object in the image using a first machine learning model; extracting a pair of cropped images from the image of the object and the illuminating image of the object based on the coordinates of the bounding box around the object, the pair of images including a cropped image of the object and a cropped illuminating image of the object; and generating boundary shape parameters from the pair of cropped images that identify the region of interest using a second machine learning model; The second machine learning model comprises: one or more convolutional layers configured to generate a first feature map encoding features of the cropped image and a second feature map encoding features of the cropped luminescent image; and one or more fully connected layers configured to convert the first feature map and the second feature map into the parameters of the boundary shape that identify the region of interest. The method is a neural network comprising:

2. The method of claim 1 , further comprising, after applying the first neural network, using a thresholding algorithm to generate a mask that identifies a contour of the object within the bounding box.

3. The method of claim 1 or 2, wherein the second machine learning model further comprises one or more residual connections.

4. The method of any one of claims 1 to 3, wherein the parameters of the boundary shape comprise parameters of an ellipse or a circle.

5. The parameters of the ellipse are: (i) the x-axis location of the center point; (ii) the y-axis location of the center point; (iii) the length of the semi-minor axis; (iv) the length of the semi-major axis; and 5. The method of claim 4, wherein (v) is the angle of rotation of the ellipse from vertical.

6. The method of any one of claims 1 to 5, wherein the subject is an animal.

7. 7. The method of claim 1, further comprising selecting a region of interest category from a list of anatomical regions, wherein the second machine learning model corresponds to the selected region of interest category.

8. 8. The method of claim 1, further comprising: before generating the first feature map and the second feature map, checking whether the image and the illuminating image are of a predetermined size, and if not, resizing the image and the illuminating image to the predetermined size.

9. The method according to any one of claims 1 to 8, further comprising displaying the image of the object in which the region of interest is indicated and / or the luminescence image of the object.

10. The method of any one of claims 1 to 9, wherein the image is a photographic image of the object.

11. 11. The method of claim 1, further comprising concatenating the first feature map and the second feature map before transforming the first feature map and the second feature map.

12. 1. A computer-implemented method for training a model for identifying one or more regions of interest in an image, comprising: For each of a plurality of training samples, each training sample includes: a training pair including an image of an object and a luminescence image of the object; and ground truth parameters of a boundary shape that identifies a region of interest of the object: generating, from each training pair, candidate parameters for a boundary shape that identifies the region of interest using a neural network; and comparing the candidate parameters of the boundary shape with corresponding ground truth parameters of the boundary shapes of the training samples; and updating parameters of the neural network based on the comparison; The neural network: one or more convolutional layers configured to generate a first feature map encoding features of the image and a second feature map encoding features of the luminescence image; and one or more fully connected layers configured to convert the first feature map and the second feature map into candidate parameters of a boundary shape that identifies the region of interest. A method comprising:

13. 13. The method of claim 12, Comparing the candidate parameters of the boundary shape to the corresponding ground truth parameters of the boundary shapes of the training samples includes evaluating a loss function; and The method, wherein updating parameters of the neural network based on the comparison comprises applying an optimization routine to the loss function.

14. A computer program product comprising computer readable code which, when executed by a computing system, causes said computing system to perform the method of any one of claims 1 to 13.

15. A system comprising one or more processors and a memory, the memory storing computer readable instructions that, when executed by the one or more processors, cause the system to perform a method according to any one of claims 1 to 13.

Images