Object Detection in Image Processing Applications

Abstract

Disclosed are target object detection in image processing applications, specifically devices, systems, and techniques for training and applying a first machine learning model to identify multiple regions of interest within an input image, and for training and applying multiple second machine learning models to identify one or more objects within each region of interest identified by the first machine learning model.

Description

Technical Field

[0001] At least one embodiment relates to processing resources for performing and facilitating artificial intelligence. For example, at least one embodiment relates to operations encountered in training and using machine learning models for efficient image processing and object detection according to the various novel techniques described herein. Background Art

[0002] Machine learning is often applied to image processing, such as the recognition of objects depicted in an image. Object recognition is used in medical imaging, scientific research, autonomous driving systems, robotic automation, security applications, law enforcement practices, and many other scenarios. Machine learning involves training a computing system (using training images and other training data) to recognize patterns in images that can facilitate object recognition. Training can be supervised or unsupervised. Machine learning models can use various computing algorithms, such as decision tree algorithms (or other rule-based algorithms), artificial neural networks, etc. During the inference phase, new images are input into the trained machine learning model, and the patterns and features identified during training can be used to identify various target objects of interest (e.g., vehicles in road images). BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1A is a block diagram of an example computer system that uses machine learning to segment images into regions of interest and perform efficient object recognition, according to at least some embodiments;

[0004] Figure 1B is an example computing device that can implement machine learning for segmenting images into regions of interest and performing efficient object recognition in accordance with at least one embodiment;

[0005] Figure 2 illustrates an example data flow during training of a machine learning model that can be used to segment images into regions of interest and perform efficient object recognition, in accordance with at least one embodiment;

[0006] Figure 3 illustrates example operations performed during identification of regions of interest using a machine learning model in accordance with at least one embodiment;

[0007] Figure 4 illustrates an example data flow during an inference phase of operation of a machine learning model that can be used to segment an image into regions of interest and perform efficient object recognition in accordance with at least one embodiment;

[0008] Figure 5is a flow chart of an example method for training a machine learning model for segmenting an image into regions of interest and performing efficient object recognition in accordance with at least one embodiment;

[0009] Figure 6 is a flow chart of an example method for segmenting an image into regions of interest and performing efficient object recognition using a trained machine learning model in accordance with at least one embodiment;

[0010] Figure 7A Inference and / or training logic according to at least one embodiment is shown;

[0011] Figure 7B Inference and / or training logic according to at least one embodiment is shown;

[0012] Figure 8 illustrates the training and deployment of a neural network according to at least one embodiment;

[0013] Figure 9 is an example data flow diagram of a high-level computing pipeline according to at least one embodiment;

[0014] Figure 10 is a system diagram of an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment. DETAILED DESCRIPTION

[0015] Automated object recognition is used in many technologies and industries, including but not limited to medical imaging, autonomous driving / parking systems, autonomous robot / machine operation, facial recognition, manufacturing quality control, alphanumeric character recognition, robotics and many other applications. In one example, object recognition can be used in medical diagnosis involving computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single photon emission computed tomography (SPECT), ultrasound imaging, etc. Modern imaging technology often produces images of large parts of the patient's body, which may include multiple organs. Computer-based object recognition technology can be used to identify the pathology (or its absence) of various organs and tissues depicted by the image. Object recognition technology can involve training a machine learning model (MLM) based on multiple training images of healthy and abnormal organs and tissues. The trained MLM can use large images (in some cases depicting the patient's entire body or most of the body) and identify the presence of tumors in the patient's internal organs, the patient's lung infection, or any other pathological conditions.

[0016] Training an existing MLM using images comprising a large medical volume can be a time-consuming process that can take several days (or longer). However, in many cases, the medical expert making the diagnosis is interested in the state of a specific organ (e.g., the heart or liver) that represents a relatively small portion (e.g., 5%, 10%, 20%, etc.) of the total image. Therefore, during the training phase and the inference phase (when new images are input into the MLM), the MLM may be processing a large amount of data that is irrelevant to the diagnostic purpose. On the other hand, simply dividing a large image into smaller sub-images may not always be possible because different patients may have variations in the size and location of their different organs (or cars and other objects or road users may have unpredictable locations along the road). Therefore, in some cases, a purely size-based segmentation may exclude portions of the targeted organ (ROI), while in other cases, portions of adjacent organs may be captured that may appear as artifacts or trigger the generation of false-positive diagnostic results.

[0017] Aspects and embodiments of the present disclosure address these and other technical challenges by disclosing methods and systems for segmenting large images into smaller regions of interest (ROIs) and performing object recognition within the segmented ROIs. In at least one embodiment, during a first phase, ROIs (e.g., a patient's organs, vehicles / pedestrians on or near a road, robots / objects in a factory or warehouse) within a large image are determined by using geometric relationships (identified during training) between the ROI's location and various reference features that are common to all (or most) objects (e.g., images of patients or traffic conditions). In one non-limiting example, one or more features of a bone structure visible within the image can serve as such reference features—the location of the lowest / highest bone point, portions of a bone structure that are relatively isolated from other bones, reference lines connecting certain bones / points within bones, etc. The geometric relationships can reflect the likely locations of different ROIs relative to the reference features. For example, the lungs can be located within a portion of an area defined between the top of the pelvic bone and the top of the thorax (in the vertical / lateral direction) and within an area between the left and right edges of the thorax (in the horizontal / sagittal direction). During the inference phase, a large input image can be segmented into one or more ROIs using MLMs trained to determine the locations of ROIs. A second set of trained MLMs can then be used to identify objects of interest (e.g., lesions, sub-parts / lobes of an organ of interest, etc.) within each ROI. The second set of MLMs can be trained on ROI-specific images of smaller size that depict the corresponding ROI and annotate the objects within the ROI. For example, training a second heart-specific MLM can be accomplished using one or more training images of the heart annotated with the locations of various parts of the heart (such as ventricles, atria, valves, myocardium, etc.). In at least one embodiment, the training images for the second set of MLMs can be ROI-specific images that do not include other ROIs. In at least one embodiment, the training images for the second set of MLMs can be larger images that include two (or more) levels of annotation (e.g., higher-level ROI annotations and lower-level object annotations within each ROI).

[0018] Although various concepts are illustrated throughout this disclosure using medical imaging examples, substantially the same or similar concepts can be used for object recognition in other contexts, such as object recognition in driving or industrial environments, in security applications, in scientific and investigative research, and many other applications.

[0019] System Architecture

[0020] Figure 1A is a block diagram of an example computer system 100 that uses machine learning for segmenting images into regions of interest and performing efficient object recognition, according to at least some embodiments. Figure 1A As depicted, computing system 100 may include computing device 102, image repository 150, and training server 160 connected to network 140. Network 140 may be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wireless network, a personal area network (PAN), or a combination thereof.

[0021] Computing device 102 can be a desktop computer, laptop computer, smartphone, tablet computer, server, or any other suitable computing device capable of executing the techniques described herein. Computing device 102 can be configured to receive image 101. Image 101 can be generated by one or more devices connected to computing device 102. For example, the device capable of generating image 101 can be a medical imaging device (e.g., a scanner such as CT, MRI, PET, SPECT), an autonomous vehicle sensing device (e.g., a LiDAR, radar, long-range and mid-range cameras), a video recorder, a photographic device, a scanner, a video, etc. Image 101 can be a medical image, an image of an automotive environment, an image of a person or crowd, an image of a natural or artificial (e.g., outdoor or indoor) landscape, an image of text or any other set of alphanumeric characters, or any other type of image. Image 101 can be in any digital (e.g., pixel-based or vector-based) format, including but not limited to JPEG, GIF, PNG, BMP, TIFF, CIB, DIMAP, NITF, etc. Image 101 can be stored (e.g., together with other images) in image repository 150. In addition, according to some embodiments disclosed herein, image repository 150 may store training images 151 for training one or more MLMs for ROI segmentation and object recognition. Image repository 150 may be accessed directly by computing device 102 or (as shown) via network 140.

[0022] Image repository 150 may be a persistent storage device capable of storing images and metadata for the stored images. Image repository 150 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage-based disks, tapes, or hard drives, NAS, SAN, and the like. Although depicted as separate from computing device 102, in at least one embodiment, image repository 150 may be part of computing device 102. In at least some embodiments, image repository 150 may be a network-attached file server, while in other embodiments, image repository 150 may be some other type of persistent storage device, such as an object-oriented database, a relational database, and the like, which may be hosted by a server machine or one or more different machines coupled to computing device 102 via network 140.

[0023] The computing device 102 may include a memory 104 communicatively coupled to one or more processing devices, such as one or more graphics processing units (GPUs) 110 and one or more central processing units (CPUs) 130. The memory 104 may store one or more MLMs, such as a ROI segmentation model (SM) 120 and one or more object identification models (OIMs) 124. The computing device 102 may host an inference engine 122 for running the SM 120 and the one or more OIMs 124. The inference engine 122 may be executed by the GPU 110 and / or the CPU 130. The inference engine 122 may use the image 101 (or training image 151) as input to the SM 120, which is trained to identify one or more ROIs (e.g., the liver, heart, lungs, etc. within a medical CT image). Inference engine 122 may then use each identified ROI as input to a corresponding one of OIMs 124, which are trained to identify one or more objects of interest (or the absence thereof) within the corresponding ROI, such as anatomical features, lesions, tumors, blood vessels, etc. In at least one embodiment, one or more of SM 120 and / or OIM 124 may be a model comprising a neural network.

[0024] SM 120 and / or one or more OIMs 124 can be trained by a training server 160. In at least one embodiment, training server 160 can be part of computing device 102. In other embodiments, training server 160 can be communicatively coupled to computing device 102 directly or via network 140. Training server 160 can be (and / or include) a rack-mounted server, a router computer, a personal computer, a laptop computer, a tablet computer, a desktop computer, a media center, or any combination thereof. Training server 160 can include a training engine 162. In at least one embodiment, training engine 162 can generate one or more machine learning models (e.g., SM 120 and OIM 124). The generated MLMs can be trained by training engine 162 using training data that can include one or more training inputs 165 and one or more corresponding target outputs 167. In at least one embodiment, different MLMs can be trained for different types of images. For example, gallbladder-specific OIMs can be trained separately from kidney-specific OIMs using gallbladder-related images and kidney-related images, respectively.

[0025] For training of SM 120, one or more training inputs 165 may include one or more training images 151 and various metadata for the training images 151. The training images 151 may be large-scale images, such as images of an entire medical volume, panoramic images of an automotive environment, etc. ROI annotation engine 163 may annotate the large-scale images with indications (e.g., bounding boxes or other bounding shapes, such as spheres, ellipsoids, cylindrical shapes, etc.) of corresponding ROIs (organs in medical images, vehicles in automotive environments, etc.) and generate one or more target outputs 167, which may be the one or more training inputs 165 with ROI annotations. In addition, training engine 162 may generate mapping data 166 (e.g., metadata) that associates the one or more training inputs 165 with one or more correct target outputs 167. During training of SM 120, training engine 162 may identify patterns in the one or more training inputs 165 based on one or more desired target outputs 167, and may train SM 120 to perform ROI segmentation using the identified patterns. The predictive utility of the identified patterns can then be verified using additional training input / target output associations and then used in future processing of new images during the inference phase by the SM 120. For example, upon receiving a new image of a patient's abdomen, the trained SM 120 can identify various ROIs, such as the left and right kidneys, pancreas, duodenum, spleen, etc.

[0026] Similarly, to train one or more OIMs 124, an object annotation engine 164 can annotate an image of the ROI (one or more ROI-level training inputs 165) with indications (e.g., pointers, bounding boxes, etc.) of different target objects (tumors, aneurysms, dilatations, and other pathologies) within the ROI and generate one or more corresponding ROI-level target outputs 167 (e.g., by adding the ROI annotations to the one or more training inputs 165). In at least one embodiment, the one or more training inputs 165 can be generated from the large-scale image by acquiring a subset of the large-scale image corresponding to the ROI. The training engine 162 can also generate mapping data 166 that associates the one or more ROI-level training inputs 165 with one or more correct ROI-level target outputs 167. During training of the one or more OIMs 124, patterns in the one or more ROI-level training inputs 165 can be identified based on one or more desired ROI-level target outputs 167 and used to perform object recognition within the corresponding ROI using the identified ROI-level patterns. The identified patterns can then be validated using additional training (or testing) input / target output associations. Based on the identified / validated patterns, the trained OIM 124 can then process new images during the inference phase. For example, upon receiving a new image 101, performing ROI segmentation with the SM 120, and identifying the gallbladder ROI, the trained gallbladder-specific OIM 124 can be deployed to identify different objects within the gallbladder ROI, such as stones, inflammation, obstruction, etc.

[0027] In at least one embodiment, each or some of the SM 120 and OIM 124 can be implemented as a deep learning neural network with multiple levels of linear or nonlinear operations. For example, each or some of the SM 120 and OIM 124 can be a convolutional neural network, a recurrent neural network (RNN), a fully connected neural network, or the like. In at least one embodiment, each or some of the SM 120 and OIM 124 can include multiple neurons, each of which can receive its input from other neurons or from an external source and can generate an output by applying an activation function to the sum of (trainable) weighted inputs and bias values. In at least one embodiment, each or some of the SM 120 and OIM 124 can include multiple neurons arranged in layers, each of which includes an input layer, one or more hidden layers, and an output layer. Neurons from adjacent layers can be connected via weighted edges. Initially, some starting (e.g., random) values can be assigned to the edge weights. For each training input 165, the training engine 162 can cause each or some of the SM 120 and OIM 124 to generate one or more outputs. The training engine 137 may then compare one or more observed outputs to one or more desired target outputs 167. The resulting error or mismatch (e.g., the difference between one or more desired target outputs 167 and one or more actual outputs of the neural network) may be back-propagated through the corresponding neural network, and the weights in the neural network may be adjusted to bring the actual output closer to the target output. This adjustment may be repeated until the output error for a given training input 165 satisfies a predetermined condition (e.g., falls below a predetermined value). Subsequently, a different training input 165 may be selected, a new output generated, and a new series of adjustments implemented until the corresponding neural network is trained to an acceptable level of accuracy.

[0028] Figure 1B1 is an example computing device 102 that can implement machine learning for segmenting images into regions of interest and performing efficient object recognition, according to at least one embodiment. In at least one embodiment, an inference engine 122 can be executed by one or more GPUs 110, and can use an SM 120 and one or more OIMs 124 to perform the segmentation into regions of interest and efficient object recognition. In at least one embodiment, GPU 110 includes multiple cores 111, each core capable of executing multiple threads 112. Each core can run multiple threads 112 concurrently (e.g., in parallel). In at least one embodiment, threads 112 can access registers 113. Registers 113 can be thread-specific registers whose access is limited to the corresponding thread. In addition, shared registers 114 can be accessible to all threads of the core. In at least one embodiment, each core 111 can include a scheduler 115 for distributing computing tasks and processes among the different threads 112 of the core 111. A dispatch unit 116 can use the correct private registers 113 and shared registers 114 to implement the scheduled tasks on the appropriate thread. Computing device 102 may include one or more input / output components 134 for facilitating the exchange of information with one or more users or developers.

[0029] In at least one embodiment, GPU 110 may have a (high-speed) cache 118, access to which may be shared by multiple cores 111. Furthermore, computing device 102 may include GPU memory 119, where GPU 110 may store intermediate and / or final results (outputs) of various computations performed by GPU 110. After completing a particular task, GPU 110 (or CPU 130) may move the output to (main) memory 104. In at least one embodiment, CPU 130 may execute processes involving serial computational tasks (assigned by inference engine 122), while GPU 110 may execute tasks amenable to parallel processing (such as multiplying a neural node's input by a weight and adding the bias). In at least one embodiment, inference engine 122 may determine which processes will be executed on GPU 110 and which processes will be executed on CPU 130. In other embodiments, CPU 130 may determine which processes will be executed on GPU 110 and which processes will be executed on CPU 130.

[0030] Figure 2 An example data flow 200 is shown during training of a machine learning model that can be used to segment an image into regions of interest and perform efficient object recognition, according to at least one embodiment. In at least one embodiment, the data flow 200 can be composed of Figure 1A and Figure 1B The inference engine 122 of the computing device 102 is implemented as Figure 2As depicted, image 201 (e.g., a raw medical image) may undergo a number of operations. In at least one embodiment, image 201 may be a three-dimensional image represented as intensity data I(x, y, z), which provides intensity values I for a plurality of voxels identified by coordinates x, y, z. Alternatively, any other identification scheme may be used to identify the voxels of the image (e.g., spherical coordinates, cylindrical coordinates, etc.). The intensity value I may represent the degree of interaction between the imaged material (e.g., human organs and tissues, objects in an automotive environment, etc.) and the imaging medium (e.g., X-rays from a CT scanner, a magnetic field from an MRI device, a laser beam from a LiDAR device, etc.). For example, the intensity value I may represent the absorption of X-rays by biological tissue. The intensity may be measured in any suitable unit, for example, within a finite range extending from 0 to 1 (0 to 100, or within any other finite range), where I=1 corresponds to complete absorption of the X-rays and I=0 corresponds to free passage of the X-rays (or vice versa). Alternatively, any other units may be used, for example, unbounded units such as Hounsfield units that relate the X-ray absorption to that of water (and air). In at least one embodiment, the image may be a two-dimensional image, wherein the intensity data I(x,y) specifies intensity values for a plurality of pixels identified by two-dimensional coordinates x,y. In at least one embodiment, the intensity data may include a plurality of intensity values {I(x,y) for different pixels (or voxels) of the image 201. j (x,y)}(or {I j (x, y, z)}), for example, multiple intensity values corresponding to RGB (red, green, blue) colors or CMYK (cyan, magenta, yellow, black) colors, etc. In at least one embodiment, multiple intensity values can be obtained for different time instances. For example, a patient can be imaged multiple times during the evolution of a tumor (e.g., months or years). Similarly, multiple time series data can be acquired during a single imaging session. In at least one embodiment, a pixel (or voxel) of image 201 can have multiple intensity values obtained using different imaging techniques. For example, intensity value I1(x, y, z) can represent the results of lidar imaging of an autonomous vehicle environment, while intensity value I2(x, y, z) can represent the results of camera imaging of the same environment.

[0031] In at least one embodiment, image 201 may undergo pre-processing 204, which may include cropping, image sharpening, blurring, noise or other artifact removal, compression, resampling, normalization, upsampling, or other operations, or any combination thereof. Image 201 may be input into ROI annotation engine 163 (which may often occur after pre-processing 204, but in some instances may also occur before pre-processing 204) for annotating ROIs within image 201. In at least one embodiment, ROI annotation engine 163 may be a computational module that implements ROI annotations applied by a human developer. For example, a human developer may use reference points, bounding boxes, numbers, coordinates of bounding planes, bounding lines, and other geometric or numerical markers to identify (e.g., on a screen accessible to ROI annotation engine 163) different ROIs within image 201 to generate ROI-annotated image 210. In at least one embodiment, ROI annotation engine 163 may generate ROI-annotated image 210 with reduced input from a human developer. For example, the ROI annotation engine 163 can perform a rough identification of ROI candidates (e.g., based on the known anatomical locations of various organs within the patient's body) and provide the identified ROI candidates to the developer for confirmation or final adjustment. In at least one embodiment, the ROI annotation engine 163 can generate the ROI-annotated image 210 without any input from the developer. For example, the annotation of the image 201 with an indication of the ROI can be generated by different models, which can use mathematical formulas, decision trees, support vector machines, or some other algorithms.

[0032] In at least one embodiment, the ROI-annotated image 210 may correspond to the image 201 (e.g., no voxels or pixels may be modified or added within the image 201), with the additional annotation metadata provided by the ROI annotation engine 163 and associated with the image 201. Thus, during training of the ROI segmentation model 120, the unannotated image 201 may be used as the training input 165, while the annotation metadata may be used as mapping data 166 to associate the training input 165 with the correct target output 167. In at least one embodiment, the ROI-annotated image 210 (including the image 201 and the annotation metadata) may be input into the ROI segmentation model (SM) 120. The input to the SM 120 may be a digital representation of an intensity map I(x, y, z) of the image 201. The SM 120 may process the input intensity map and output a prediction identifying the ROI within the image 201. The predicted ROI 222 may then be compared to the target ROI (e.g., identified in the annotation metadata of the ROI-annotated image 210) using a loss function 230. SM120 can output multiple predicted ROIs 222 for each target ROI. The predicted ROIs 222 can be specified via a digital representation of the location of the corresponding predicted ROI 222, such as the coordinates of the eight vertices of a parallelepiped (bounding box) that surrounds the corresponding ROI. In at least one embodiment, a different digital representation can be used, such as the center of a bounding sphere with a specified radius (four parameters), the center of an ellipsoid with three specified semi-axes (six parameters), or any other geometric representation. In at least one embodiment, different digital representations can be used for different ROIs (organs). For example, an ellipsoidal bounding box / shape can be specified for the kidney ROI, while a parallelepiped bounding box / shape can be specified for the liver ROI.

[0033] Loss function 230 can compare the representation of the predicted ROI with the annotation metadata. Loss function 230 can be a binary cross-entropy loss function, a mean squared error loss function, a weighted mean squared error loss function, a mean absolute error loss function, a Huber loss function, a hinge loss function, a multi-class cross-entropy loss function, a Kullback-Liebler loss function, or the like. In at least one embodiment, loss function 230 can compare the position of the bounding box of the predicted ROI 222 with the position of the bounding box identified by the annotation metadata and determine a value (a loss value or cost value) that describes how accurately SM 120 identified the position of the predicted ROI 222. In at least one embodiment, more than one loss function 230 can be deployed. Some of the deployed loss functions 230 can be ROI-specific and can weight errors differently for different target ROIs. For example, a loss function 230 for a larger organ (e.g., the lungs) may be more tolerant of errors in the prediction of the corresponding bounding box, while a loss function 230 for a smaller organ (e.g., the gallbladder) may be less tolerant of such errors. In at least one embodiment, the loss function 230 may output a single loss value for the entire bounding box. In at least one embodiment, the loss function 230 may output multiple loss values, such as separate loss values for various parameters of the bounding box, such as a separate loss value for each vertex of the bounding box.

[0034] In at least one embodiment, the training engine 162 can adjust the parameters of the SM 120 to minimize one or more loss values determined by the one or more loss functions 230. For example, in embodiments where the SM 120 includes one or more neural networks, the one or more loss values can be back-propagated through the various layers of the neural network, and the parameters (biases, weights, etc.) of the neural network can be adjusted (by the training engine 162) until the observed difference (loss value) between the location of the one or more predicted ROIs 222 and the target location specified in the annotation metadata is minimized. Subsequently, the SM 120 can be trained with additional images 201 until the SM 120 can successfully identify ROIs in new images with a target accuracy, for example, until ROIs within a test image set are misidentified in fewer than a certain percentage of cases.

[0035] The ROIs segmented from the image 201 can be used to train the ROI-specific OIM 124 to recognize various objects of interest within each ROI. Figure 2As further depicted in , ROI data 251 may include predicted ROI 222 and may additionally include various metadata, such as metadata related to the type of machine or device that generated initial image 201, information about the patient (e.g., age, gender, history of past surgical procedures, etc.). In at least one embodiment, ROI image 251 may include one or more identifications of one or more bounding boxes output by SM 120 in addition to an identification of the (large) image 201 (e.g., the location of image 201 in memory 104 or image repository 150). In at least one embodiment, ROI image 251 may be an actual sub-image of image 201, e.g., a segmented and saved (to memory) region of image 201. The identification scheme for voxels (or pixels) of ROI image 251 may be the same or similar to the scheme used to identify voxels (or pixels) of image 201.

[0036] In at least one embodiment, the ROI image 251 can undergo additional pre-processing 254, which can include any of the pre-processing techniques and methods referenced with respect to pre-processing 204 but customized for the type of ROI being processed. The ROI image 251 can be input into the ROI annotation engine 163 (before or after pre-processing 254). In at least one embodiment, the ROI annotation engine 163 can perform object annotation similar to how the ROI annotation engine 163 generates the ROI-annotated image 210. For example, the ROI annotation engine 163 (with or without developer input) can use reference points, bounding boxes, numbers, coordinates of bounding planes, lines, and other geometric or digital markers to identify different objects within the ROI image 251 to generate the object-annotated ROI image 260.

[0037] In at least one embodiment, object-annotated ROI image 260 may correspond to ROI image 251 (e.g., no voxels or pixels have been added to or modified within ROI image 251) with additional object annotation metadata provided by ROI annotation engine 163 and associated with ROI image 251. Thus, during training of ROI-specific OIM 124, unannotated ROI image 251 may be used as training input, while the object annotation metadata may be used to identify the correct target output. In at least one embodiment, object-annotated ROI image 260 (including image 251 and object annotation metadata) may be input into ROI-specific OIM 124. In at least one embodiment, the number of OIMs 124 used during the inference phase may be at least the number of different types of ROIs within image 201. The input to OIM 124 may be a digital representation of the intensity map I(x, y, z) of ROI image 251. The OIM 124 can process the input intensity map and output predictions of identified objects 280 within the ROI image 251. The identified objects 280 can be compared to the object annotation metadata of the object-annotated ROI image 251 using a ROI-specific loss function 290. Multiple identified objects 280 can be output by the OIM 124. Identified objects 280 can be specified via digital representations of the positions of corresponding bounding boxes or using any other identifier (e.g., the coordinates of the eight vertices of a parallelepiped bounding box that encloses the corresponding identified object 280). In at least one embodiment, a bounding sphere, ellipsoid, or any other geometric shape can be used. In at least one embodiment, different digital representations can be used for different identified objects 280 within a single ROI. For example, an ellipsoidal bounding box / shape can be specified for gallbladder obstruction, while a parallelepipedal bounding box / shape can be used to indicate gallstones.

[0038] ROI-specific loss function 290 can compare the representation of the identified object with the object annotation metadata generated by the ROI annotation engine. In some embodiments, ROI-specific loss function 290 can be any of the loss functions listed above with reference to loss function 230. In at least one embodiment, ROI-specific loss function 290 can be the same as loss function 230. In at least one embodiment, ROI-specific loss function 290 can be different from loss function 230 and can also be different for different ROI-specific OIMs 124. ROI-specific loss function 290 can compare the position of one or more bounding boxes of one or more identified objects 280 with the position of the bounding boxes identified by the object annotation metadata to determine a value (a loss value or cost value) that describes how accurately the OIM 124 has determined the position of the identified object 280.

[0039] In at least one embodiment, the training engine 162 can adjust the parameters of the OIM 124 to minimize one or more loss values determined by one or more ROI-specific loss functions 290. For example, in embodiments where the OIM 124 includes one or more neural networks, the one or more loss values can be back-propagated through the layers of the neural network, and the parameters (biases, weights, etc.) of the neural network can be adjusted (by the training engine 162) until the observed difference (loss value) between the position of the identified object 280 and the target object position specified in the object annotation metadata is minimized. Subsequently, the OIM 124 can be trained using additional ROI images 251 until the OIM 124 can successfully identify objects in new ROI images 251 with a target accuracy, for example, until objects within a set of test ROI images 251 are misidentified less than a certain percentage of the time.

[0040] Figure 3 Example operations 300 performed during the identification of regions of interest using a machine learning model in accordance with at least one embodiment are shown. Example operations for a training phase 302 and an inference phase 304 are shown. Operations 300 are illustrated using the non-limiting example of processing a three-dimensional medical image 301, but a similar sequence of operations can be used to process any other type of image. Other images can be processed in the context of automotive, marine, aeronautical, military, security, robotics, industrial, geological, or any other application. In at least one embodiment, image 301 can be a training image (and part of training input 165). Training input 165 can further include ROI annotation metadata (e.g., as provided by ROI annotation engine 163) indicating the location of various ROIs within image 301. Training input 165 can further include other metadata describing the type of image 301 (e.g., CT or MRI), the settings of the device that generated image 301 (e.g., exposure, signal strength, programming sequence, etc.), patient information (e.g., medical history, biographical information, etc.), and other types of information. During the inference phase 304, the image 301 may be a new image (not previously encountered during the training phase 302) and may be part of the inference input 302. The inference input 302 may further include some types of metadata that are also included in one or more of the training inputs 165 (e.g., medical imaging device and diagnostic technique identification), but may not include other types of metadata (e.g., ROI annotation metadata).

[0041] During training phase 302, the ROI-annotated image 301 can be processed by a machine learning model (e.g., ROI segmentation model 120) to identify the ROI characteristics 306 of the respective ROIs that the SM 120 is being trained to identify, as well as to identify reference features 308 present in the image 301. The ROI characteristics 306 can be determined based on an analysis of one or more intensity data I(x,y,z) of the image 301. For example, the ROI characteristics 306 can include the size of the corresponding (annotated) ROI, e.g., the total number of voxels (or pixels) of the ROI determined to have an intensity value above (or below) a certain intensity threshold I(x,y,z)>I0 (or I(x,y,z)<I0). The ROI characteristics 306 can also include the location of the corresponding ROI, e.g., the coordinates of the bounding box of the region of voxels (or pixels) enclosing the ROI, where the intensity value of the voxels (or pixels) of the ROI is above (or below) the intensity threshold I(x,y,z)>I0 (or I(x,y,z)<I0). The ROI characteristics 306 can also include a description of the corresponding ROI location relative to other ROIs, e.g., a description of adjacent organs located above, below, behind, etc. the corresponding ROI / organ. The ROI characteristics 306 can also include the shape of the corresponding ROI, such as a representation (e.g., vector representation) of the boundary of the corresponding ROI (e.g., geometric surface I(x,y,z)=I0) via a plurality of predefined shapes (sphere, ellipsoid, cylinder, plane, etc.). The ROI characteristics 306 can additionally include the topology of the corresponding ROI, such as the number of lobes, interconnected sub-parts of the ROI, the type of connections between sub-parts, etc. The ROI characteristics 306 can further include the average intensity value, maximum intensity, minimum intensity, degree of uniformity or non-uniformity of intensity (e.g., represented by the variance of the intensity within the ROI) across the bounding box or across the region where I(x,y,z)>I0 (or I(x,y,z)<I0). Various other representative parameters can be included in the ROI characteristics 306.

[0042] During the training phase 302, the ROI-annotated images 301 may also be processed by the SM 120 to identify reference features 308 of the images 301. While the ROI characteristics 306 may relate to properties of different individual ROIs, the reference features 308 may relate to properties of the image 301 at a larger scale or to features common to images of the same type as the image 301. For example, medical images of a patient's torso and / or abdomen may include images of bones (skull, spine, rib cage, pelvic bones, etc.), which may be clearly visible and present in most or every image of the same type of diagnostic procedure and, therefore, may be used as reference features 308 associated with identifying the locations of different ROIs. Organs such as the lungs may have a fairly reproducible (from patient to patient and from image to image) arrangement of bones (including ribs, clavicle, sternum, etc.), even when the exact position of the patient is not exactly the same for different images 301. Similarly, the arrangement of vehicles on a road may have a reproducible relationship to the surface of the road, regardless of weather, time of day and week, traffic intensity, etc. For example, an image captured by an autonomous vehicle's camera in the direction of motion of the autonomous vehicle may capture a vehicle traveling in the same direction if such a vehicle is between the center line and the right edge line; or a vehicle traveling in the opposite direction if such a vehicle is between the center line and the left edge line. Similarly, an image may capture a parked vehicle if such a vehicle is outside the edge line (or on top of the edge line), and so on.

[0043] During the training phase 302, the SM 120 can be trained to identify one or more ROI features 306 that are most representative (and / or reproducible) among various attributes and visual characteristics of various ROIs. Similarly, the SM 120 can be trained to identify the most representative reference features 308 among various features that may be present in the training image, and to identify one or more geometric relationships 310 between each ROI feature 306 and the reference features 308. The identified geometric relationships 310 can be used to determine the location of the ROI within the image 301. For example, shown in image 301(a) is a region 323 of the visible portion of the patient's skeleton 321, where a two-dimensional cross-sectional image perpendicular to the X axis (along the YZ cross-sectional plane) only reveals the vertebrae and no other bones. Conversely, above the region 323 (e.g., toward increasing coordinate X), the YZ cross-sectional image will also capture images of one or more ribs, the sternum, etc., while below the region 323, the cross-sectional image will capture one or more portions of the pelvic bone. Thus, region 323 (or the center of region 323 ) may be used as one of reference features 308 .

[0044] Based on learned (during training) ROI characteristics 306 for each ROI (e.g., an organ), the location of the ROI can be determined based on one or more geometric relationships 310 between region 323 (whose size and position are used as reference features 308) and each ROI within image 301 (e.g., the location of each organ of the patient). For example, the diaphragm can be located a distance l1 above the center of region 323, the lower border of the lungs can be located a distance l2 above the center of region 323, the top of the bladder can be located a distance l3 below the center of region 323, and so on. In at least one embodiment, some or all of the distances l1, l2, and / or l3 can be measured in absolute units of length (e.g., centimeters, inches, etc.). In at least one embodiment, some or all of the distances l1, l2, and / or l3 can be measured in relative units (e.g., in units of the length of region 323 (e.g., along direction X)), which can be smaller or larger depending on the size of the patient. Other reference features 308 can include various skeletal features, such as the vertebrae, the sternum, the widest portion of the pelvic bones, etc.

[0045] In at least one embodiment, one or more intensity maps I(x,y,z) may be used to identify one or more reference features 308 in image 305. For example, SM 120 may select one or more three-dimensional sampling regions 326 and extract (sample) a plurality of two-dimensional intensity maps Ii corresponding to respective values of coordinate x using discretized values x=i×Δx. ( y,z), where i assumes integer values within an interval corresponding to a particular sampling region 326. Alternatively (or additionally), the SM 120 may sample a plurality of two-dimensional cross-sectional images I corresponding to respective values of coordinates x and y using discretized values from the three-dimensional image I(x,y,z). j (x,y) and I k (y, z), the discretized values are, for example, y=j×Δy and z=k×Δz, where j and k are assumed to correspond to integer values within the corresponding intervals of the specific sampling area 326. The SM 120 may be based on the plurality of two-dimensional intensity maps I i (y,z),I j (x,y) and I k (y, z) to determine the reference features 308 (e.g., location, size, associated strength, etc.). In addition, in some embodiments, each or some of these two-dimensional intensity maps can be further subdivided (sampled) into multiple one-dimensional intensity maps I ij (z),I jk (x), and I ik (y). In some implementations, each intensity map I ij (z),I jk (x),I ik (y)(and / or Ii (y,z),I j (x,y), and I k (y, z)) may be input into a neural network (or other machine learning model) of the SM 120 , and the SM 120 may determine the position, geometry, intensity, etc. of each reference feature 308 .

[0046] In at least one embodiment, the selection of sampling regions 326 within image 305 can be performed based on a predetermined heuristic. For example, sampling regions 326 can be large enough so that a fixed arrangement of sampling regions 326 relative to an input image (e.g., image 305) is likely to capture all or most reference features 308, even when input images are taken of patients of different sizes, ages, genders, or patients positioned in slightly different positions. In at least one embodiment, the selection of sampling regions 326 within image 305 can be performed randomly. In at least one embodiment, the combined sampling regions 326 can occupy the entire volume of image 305 or can even overlap with adjacent sampling regions 326.

[0047] In one non-limiting example of a medical image, the SM 120 may analyze a plurality of two-dimensional intensity maps I i (y, z), and in each image, pixels corresponding to bone structures are identified. For example, bone structures can be identified with intensity values greater than a certain minimum value I bone (or, depending on the imaging technique used, less than some maximum value I bone More specifically, the SM 120 may determine the bone region z in the i-th section perpendicular to the x-axis. i Equation I for the boundary of (y) i (y,z)=I bone Identify each intensity map I i The SM 120 can also identify the bone structure visible within (y,z). max (i) = max(z i (y): maximum height (along the z-axis) of y). Plot 307 shows (not to scale) this maximum height z for each distance x=i×Δx max (i) (For simplicity, plot 307 depicts the successive maximum heights z max (x) curve, although in a specific embodiment, a discretized version z max (i)). SM 120 may then calculate the max (i) identifies the lung ROI. For example, SM 120 may identify the function z max (i) corresponds to the two main features of the pelvic bone 330 and the rib cage 332. SM 120 can further identify the curve zmax (i) Multiple points (reference points or control points). For example, point A may correspond to z max (i) and can be identified by region 323, in which no bones other than the vertebrae of the spine are located. SM 120 can further identify point B, which is the apex of the thoracic cavity. Points A and B can be some of reference features 308. SM 120 can also determine the location of lung ROI 334 based on the geometric relationship of lung ROI 334 to points A and B. In at least one embodiment, SM 120 can identify point C, at which the height of thoracic cavity 332 is reduced from its maximum height at point B by a factor a (e.g., a=2, 2.5, ..., or any other value determined during training of SM 120). SM 120 can further identify point D, at which the height of thoracic cavity 332 is reduced by a factor b (e.g., b=2.5, 3, ..., or any other value determined during training). In another embodiment, the location of the bottom of lung ROI 334 can be predicted to be at a certain distance from point A (determined during training state 302). The x-coordinates of points C and D may represent the location (within the x-axis) of lung ROI 334. The location of lung ROI 334 (or any other ROI) along the other two axes (or one axis if image 305 is two-dimensional) may be determined in a similar manner.

[0048] During the inference phase 304, using a new input image as part of the reference input 302, the trained SM 120 may perform similar analysis and similar calculations based on one or more reference features 308 identified within the new image (e.g., part or all of points A, B, C, and / or D, etc.). Using the identified reference features 308 and one or more geometric relationships 310, the trained SM 120 may identify one or more ROIs 322. The reference features and geometric relationships described with respect to operation 300 are intended to be examples. There are a virtually unlimited number of reference features and geometric relationships of such features to ROIs that may be identified in different contexts in which aspects and implementations of the present disclosure may be used.

[0049] Although Figure 3120 , operation 300 is illustrated in the context of the segmentation model 120 identifying an ROI, but similar operations may be performed with respect to different OIMs 124 identifying individual objects within an ROI 122 during the training and inference phases. For example, during the training phase of one or more OIMs 124, object characteristics of individual target objects within the ROI may be identified. For example, such object characteristics may include the number of lobes of an organ (e.g., three lobes of the right lung, two lobes of the left lung, etc.), the typical size of the object, its shape, its relative orientation with respect to other objects, etc. For each ROI, one or more reference features may be identified by the corresponding OIM 124, such as the myocardium separating the left ventricle from the right ventricle, and one or more geometric relationships may be determined between the identified object characteristics and the reference features of the objects. Based on such geometric relationships, the corresponding OIM 124 identifies individual target objects within each (or some) ROI using, for example, three-dimensional and two-dimensional intensity maps, similar to how individual ROIs may be determined, for example, as described above with respect to Figure 3 As described. In at least one embodiment, additional MLMs can be trained to identify sub-objects within an object (and similarly, sub-sub-objects within sub-objects, etc.) by performing further segmentation of the object into target sub-objects using geometric relationships between corresponding reference features and sub-object properties. Multiple levels of the object hierarchy (e.g., sub-sub-objects, etc.) can be defined and processed in a similar manner.

[0050] Figure 4 An example data flow 400 is shown during the inference phase of operation of a machine learning model that may be used to segment images into regions of interest and perform efficient object recognition, in accordance with at least one embodiment. Figure 4 Depicted in and used with Figures 1 to Figure 3 The various modules and objects represented by numbers differing by the first digit used in the figures may refer to the same modules and objects as depicted in the corresponding figures; for example, the ROI segmentation model 420 may be the same as Figures 2 to 3 The same (or similar) model as SM120. Figure 4 As shown, the new image 401 can be input into the ROI segmentation model 420, which can identify one or more ROIs 422 (e.g., 422-1, 422-2, 422-3, etc.). Each of the identified ROIs 422 can be input into a corresponding object recognition model in the object recognition model 424 to identify one or more target objects 480 (e.g., 480-1, 480-2, 480-3, etc.). The identified target objects can be combined into merged ROI / object data 490 and provided to a user (e.g., a medical professional).

[0051] Figure 5 and Figure 6 Flowcharts of example methods 500 and 600, respectively, for facilitating object recognition in images according to some embodiments of the present disclosure. In at least one embodiment, methods 500 and 600 may be performed by a processing unit of training server 160 or computing device 102. Methods 500 and 600 may be performed by one or more processing units (e.g., a CPU and / or GPU), which may include one or more memory devices (or communicate with one or more memory devices). In at least one embodiment, methods 500 and 600 may be performed by multiple processing threads (e.g., CPU threads and / or GPU threads), each thread executing one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, the processing threads implementing method 500 (and similarly, method 600) may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 500 (and similarly, method 600) may execute asynchronously with respect to each other. The various operations of methods 500 and 600 may be executed in a manner similar to Figure 5 and 6 Some operations of these methods may be performed concurrently with other operations. In at least one embodiment, the operations may not always be performed. Figure 5 and Figure 6 One or more operations shown in .

[0052] Figure 5 The flowchart of an example method 500 for training a machine learning model for image segmentation into regions of interest and efficient object recognition, according to at least one embodiment, is provided. Method 500 can be performed to train a machine learning model for medical imaging, autonomous driving systems, industrial scenarios, and many other contexts. A processing unit executing method 500 may generate first training data at block 510 that may include a first training input and a first target output. The first training input may include an intensity map of a first training image. The first target output may include first annotation data identifying a plurality of ROIs in the first training image. The first training image may be a medical image, an image of a driving environment, an image of a shipping facility, an image of a group of people, or any other image suitable for object recognition. The first training input may be one of a first set of training inputs. Similarly, the first target output may be one of the first set of target outputs. At block 520, method 500 may continue by the processing unit providing the first training data to train a segmentation machine learning model. The segmentation machine learning model may use the intensity map to identify a plurality of reference features within the first training image. The segmentation machine learning model may further use the annotation data to identify one or more geometric relationships between each reference feature and one or more corresponding ROIs of the plurality of ROIs.

[0053] In at least one embodiment, during training, the segmentation MLM can select multiple reference features within the training image. The selection can be made using human (e.g., developer) input. In at least one embodiment, the selection can be made without any human input. The selected reference features can be features that are likely to be present in many images of a similar type. For example, the paved surface of a road, the intersection of two or more roads, lane markings, etc. can be selected as reference features. In some medical images, the selected multiple reference features can be associated with the appearance of the patient's bones within the target image.

[0054] In at least one embodiment, the processing unit executing method 500 may identify one or more geometric relationships between each selected reference feature and one or more ROI locations associated with a corresponding ROI type from a plurality of ROI types. For example, in a medical image, the identified geometric relationships may be between certain reference ribs and the location of the lungs and between (the same or other) reference ribs and the location of the liver. The segmentation MLM may identify multiple ROIs in the training image. The identification of the ROIs may be based, at least in part, on one or more identified geometric relationships between each of the selected plurality of reference features. For example, the segmentation MLM may indicate the location of the lungs based on their geometric relationship to the reference ribs and may similarly indicate the location of the liver.

[0055] In at least one embodiment, the processing unit performing method 500 may determine mismatches between the identified multiple ROIs and multiple target ROIs for the training image. For example, the target positions of the lungs and / or liver (e.g., specified in the annotation metadata of the training image) may not be completely accurate, e.g., may have an error (mismatch) exceeding a maximum tolerance error. In response to determining that the maximum tolerance error is exceeded, the processing unit performing method 500 may update the one or more identified geometric relationships based on the identified mismatches. For example, the segmentation MLM may update the geometric relationship between the reference rib and the position of the lungs and / or liver (e.g., in a direction that decreases or increases the distance). In at least one embodiment, the geometric relationship can be updated in any (multiple) directions, and the updated geometric relationship can be back-propagated through the MLM for the same training image. Based on the results of the back-propagation (one or more) steps, the derivative (or gradient) of the loss function can be calculated, and the most favorable updated direction can be selected. This process can be repeated until the mismatch with the target ROI position is minimized.

[0056] In at least one embodiment, the segmentation MLM can also update the selection of multiple reference features. For example, as an alternative (or in addition) to updating the geometric relationship between the reference ribs and the ROI position, the segmentation MLM can be switched to using the sternum as the new (updated) reference feature. Similarly, other bones or organs of the patient can be used as reference features by the segmentation MLM. This process can also be repeated until the mismatch between the reference features and the target ROI position is minimized. In at least one embodiment, the updating of the geometric relationship and the updating of the reference features can be performed separately (e.g., sequentially). In at least one embodiment, the updating of the geometric relationship and the updating of the reference features can be performed concurrently (e.g., in parallel).

[0057] At block 530, the processing unit executing method 500 may generate second (third, etc.) training data including second (third, etc.) training inputs and second (third, etc.) target outputs. The second (third, etc.) training inputs may include second (third, etc.) training images (ROI images) representing a first (second, etc.) ROI among a plurality of ROIs. The second (third, etc.) training data may also include second (third, etc.) target outputs, which include second (third, etc.) annotation data identifying one or more objects within the first ROI, the first ROI being a first ROI type among a plurality of ROI types. The second (third, etc.) training input may be one of a second (third, etc.) group of training inputs. Similarly, the second (third, etc.) target output may be one of a second (third, etc.) group of target outputs.

[0058] At block 540, the processing unit executing method 500 may provide second (third, etc.) training data to train the first (second, etc.) object recognition MLM. After training based on the second (third, etc.) set of training inputs and the second (third, etc.) set of target outputs, the first (second, etc.) object recognition MLM may be able to recognize objects in images of the first (second, etc.) ROI type.

[0059] Figure 6 is a flow chart of an example method 600 for segmenting an image into regions of interest and performing efficient object recognition using a trained machine learning model in accordance with at least one embodiment. A processing unit executing method 600 may identify an input image to be provided to a first MLM (e.g., SM 120) at block 610. The input image may be a new image that has not been previously used for training of the machine learning model. In at least one embodiment, the input image may be a multidimensional image of a first dimension (e.g., a three-dimensional image). In at least one embodiment, the first MLM may be trained to detect a plurality of reference features based at least in part on processing a plurality of segmented images associated with the multidimensional image. Each of the plurality of segmented images may be a second dimension (e.g., a two-dimensional image) representing a portion of the input image.

[0060] At block 620, method 600 may continue with the processor applying the first MLM to the input image to identify a plurality of ROIs within the input image. In at least one embodiment, the first MLM may be trained using method 500 (or a similar method). Figure 6 As depicted in the expanded portion of , applying the first MLM may include (at box 622) detecting a plurality of reference features within an input image. Applying the first MLM may further include (at box 624) using one or more geometric relationships between each reference feature and one or more locations associated with one of a plurality of ROI types. In at least one embodiment, the first MLM is trained based on a plurality of training images having reference features of a type common to the type of reference features within the input image. In at least one embodiment, the first MLM may include a neural network having at least one hidden layer. In at least one embodiment, applying the first MLM to the input image may involve performing one or more computations associated with the first MLM on one or more graphics processing units. In at least one embodiment, the input image is a medical image of a patient, and the plurality of reference features are associated with the appearance of the patient's skeleton.

[0061] The output of the first MLM may be a plurality of (box 630) ROIs identified within the input image. In at least one embodiment, the first ROI of the plurality of ROIs may include a representation of a first organ of the patient or at least a portion of the first organ. The first ROI may include a sufficiently accurate representation (e.g., a depiction) of the first organ and may not include a representation of any portion of the patient's other (e.g., second, third, etc.) organs. The patient's other organs may be represented by other ROIs identified by the first MLM. For example, the second ROI of the plurality of ROIs may include a representation of a second organ or at least a portion of the second organ. In at least one embodiment, each of the identified ROIs may be specified by the location of the corresponding ROI within the input image. In at least one embodiment, each ROI may be identified by an actual image of the corresponding ROI or some other representation.

[0062] At block 640, method 600 may continue with the processing unit providing each of the identified plurality of ROIs as input to a corresponding second MLM in a plurality of second MLMs (e.g., OIM 124) to obtain an output from the corresponding second MLM. Each of the plurality of second MLMs may correspond to one of a plurality of ROI types. The output from each of the second MLMs (block 650) may include an indication of one or more objects within the ROI of the corresponding ROI type. In at least one embodiment, providing each of the plurality of ROIs to the corresponding second MLM in the plurality of second MLMs may include providing at least one of a location of the corresponding ROI within the input image or a representation of the corresponding ROI (e.g., a portion of the input image that includes the ROI).

[0063] Reasoning and training logic

[0064] Figure 7A Inference and / or training logic 715 is shown for performing inference and / or training operations associated with one or more embodiments.

[0065] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 701 for storing forward and / or output weights and / or input / output data, and / or other parameters for configuring neurons or layers of a neural network being trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 701 for storing graph code or other software to control timing and / or sequence, wherein weights and / or other parameter information are loaded to configure logic, including integer and / or floating point units (collectively referred to as arithmetic logic units (ALUs) or simply circuits). In at least one embodiment, code (such as graph code) loads weights or other parameter information into a processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, the code and / or data storage 701 stores input / output data during training and / or inference using aspects of one or more embodiments and / or weight parameters during forward propagation of weight parameters for each layer of a neural network trained or used in conjunction with one or more embodiments. In at least one embodiment, any portion of code and / or data storage 701 may be included within other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.

[0066] In at least one embodiment, any portion of code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 701 may be cache memory, dynamic random access memory ("DRAM"), static random access memory ("SRAM"), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether code and / or data storage 701 is internal or external to a processor, for example, or composed of DRAM, SRAM, flash memory, or some other type of storage, may depend on the available storage space on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in inference and / or training of the neural network, or some combination of these factors.

[0067] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 705 for storing backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, during training and / or inference using aspects of one or more embodiments, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of the neural network trained or used in conjunction with one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 705 for storing graph code or other software to control the timing and / or sequence in which weight and / or other parameter information is loaded to configure logic, which includes integer and / or floating point units (collectively referred to as arithmetic logic units (ALUs)).

[0068] In at least one embodiment, code (such as graph code) causes weights or other parameter information to be loaded into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, any portion of code and / or data storage 705 can be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and / or data storage 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 705 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether code and / or data storage 705 is internal or external to the processor, for example, whether it is composed of DRAM, SRAM, flash memory, or some other type of storage, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the data batch size used in inference and / or training of the neural network, or some combination of these factors.

[0069] In at least one embodiment, code and / or data store 701 and code and / or data store 705 may be separate storage structures. In at least one embodiment, code and / or data store 701 and code and / or data store 705 may be the same storage structure. In at least one embodiment, code and / or data store 701 and code and / or data store 705 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data store 701 and code and / or data store 705 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0070] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 710 (including integer and / or floating point units) for performing logical and / or mathematical operations based at least in part on or directed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values from a layer or neuron within a neural network) stored in activation storage 720, which are functions of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, activations are performed in response to executing instructions or other code, linear algebra and / or matrix-based mathematics performed by ALU 710 to generate activations stored in activation storage 720, wherein weight values stored in code and / or data storage 705 and / or in code and / or data storage 701 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data storage 705 or code and / or data storage 701 or other on-chip or off-chip storage.

[0071] In at least one embodiment, one or more ALUs 710 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment, one or more ALUs 710 may be external to the processor or other hardware logic devices or circuits that use them (e.g., coprocessors). In at least one embodiment, one or more ALUs 710 may be included within the execution units of a processor or otherwise included in an ALU bank accessible by the execution units of the processor, which may be within the same processor or distributed between different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may share a processor or other hardware logic device or circuit, while in another embodiment, they may be in different processors or other hardware logic devices or circuits or some combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 720 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Additionally, inference and / or training code may be stored with other code accessible to a processor or other hardware logic or circuitry and may be retrieved and / or processed using the processor's fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0072] In at least one embodiment, activation storage 720 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 720 can be completely or partially internal or external to one or more processors or other logic circuits. In at least one embodiment, whether activation storage 720 is internal or external to the processor, for example, or comprises DRAM, SRAM, flash memory, or other storage types, can be selected based on the available on-chip or off-chip storage, the latency requirements for performing training and / or inference functions, the batch size of data used in inferring and / or training neural networks, or some combination of these factors.

[0073] In at least one embodiment, Figure 7A The inference and / or training logic 715 shown in FIG may be used in conjunction with an application specific integrated circuit (“ASIC”), such as the one from Google. Processor from Graphcore TM Inference Processing Unit (IPU) from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7A The illustrated inference and / or training logic 715 may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware such as a field programmable gate array (“FPGA”).

[0074] Figure 7B Inference and / or training logic 715 is shown in accordance with at least one embodiment. In at least one embodiment, inference and / or training logic 715 may include, but is not limited to, hardware logic where computing resources are dedicated or otherwise uniquely used in conjunction with weight values or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown in FIG can be used in conjunction with an application specific integrated circuit (ASIC), such as the one from Google. Processor from Graphcore TM Inference Processing Unit (IPU) from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7BThe inference and / or training logic 715 shown in can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware such as a field programmable gate array (FPGA). In at least one embodiment, the inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 7B In at least one embodiment shown in FIG, code and / or data storage 701 and code and / or data storage 705 are each associated with dedicated computing resources, such as computing hardware 702 and computing hardware 706, respectively. In at least one embodiment, computing hardware 702 and computing hardware 706 each include one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) solely on the information stored in code and / or data storage 701 and code and / or data storage 705, respectively, with the results of the functions being stored in activation storage 720.

[0075] In at least one embodiment, each of the code and / or data stores 701 and 705 and the corresponding computing hardware 702 and 706 corresponds to a different layer of a neural network, such that activations from one "storage / compute pair 701 / 702" of the code and / or data store 701 and computing hardware 702 are provided as input to the next "storage / compute pair 705 / 706" of the code and / or data store 705 and computing hardware 706, reflecting the conceptual organization of the neural network. In at least one embodiment, each storage / compute pair 701 / 702 and 705 / 706 can correspond to more than one neural network layer. In at least one embodiment, additional storage / compute pairs (not shown) can be included in the inference and / or training logic 715 after or in parallel with the storage / compute pairs 701 / 702 and 705 / 706.

[0076] Neural network training and deployment

[0077] Figure 8The training and deployment of a deep neural network according to at least one embodiment is shown. In at least one embodiment, an untrained neural network 806 is trained using a training dataset 802. In at least one embodiment, the training framework 804 is the PyTorch framework, while in other embodiments, the training framework 804 is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training frameworks. In at least one embodiment, the training framework 804 trains the untrained neural network 806 and enables it to be trained using the processing resources described herein to generate a trained neural network 808. In at least one embodiment, the weights can be randomly selected or pre-trained using a deep belief network. In at least one embodiment, the training can be performed in a supervised, partially supervised, or unsupervised manner.

[0078] In at least one embodiment, untrained neural network 806 is trained using supervised learning, where training dataset 802 includes inputs paired with expected outputs for the inputs, or where training dataset 802 includes inputs with known outputs and neural network 806 has manually graded outputs. In at least one embodiment, untrained neural network 806 is trained in a supervised manner, processing inputs from training dataset 802 and comparing the resulting outputs to a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through untrained neural network 806. In at least one embodiment, training framework 804 adjusts the weights that control untrained neural network 806. In at least one embodiment, training framework 804 includes tools for monitoring the degree to which untrained neural network 806 converges toward a model (e.g., trained neural network 808) that is suitable for generating correct answers (e.g., results 814) based on input data (e.g., new dataset 812). In at least one embodiment, training framework 804 repeatedly trains untrained neural network 806 while adjusting weights to improve the output of untrained neural network 806 using a loss function and an adjustment algorithm (e.g., stochastic gradient descent). In at least one embodiment, the training framework 804 trains the untrained neural network 806 until the untrained neural network 806 reaches a desired accuracy. In at least one embodiment, the trained neural network 808 can then be deployed to implement any number of machine learning operations.

[0079] In at least one embodiment, untrained neural network 806 is trained using unsupervised learning, wherein untrained neural network 806 attempts to train itself using unlabeled data. In at least one embodiment, unsupervised learning training dataset 802 will include input data without any associated output data or "ground truth" data. In at least one embodiment, untrained neural network 806 can learn groupings within training dataset 802 and can determine how individual inputs relate to untrained dataset 802. In at least one embodiment, unsupervised training can be used to generate a self-organizing map in trained neural network 808 that can perform operations useful for reducing the dimensionality of new dataset 812. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows identification of data points in new dataset 812 that deviate from the normal pattern of new dataset 812.

[0080] In at least one embodiment, semi-supervised learning can be used, which is a technique in which a mixture of labeled and unlabeled data is included in the training dataset 802. In at least one embodiment, the training framework 804 can be used to perform incremental learning, for example, through a transfer learning technique. In at least one embodiment, incremental learning enables the trained neural network 808 to adapt to the new dataset 812 without forgetting the knowledge that was instilled into the trained neural network 808 during the initial training.

[0081] Reference Figure 9 , Figure 9 is an example data flow diagram of a process 900 for generating and deploying a processing and inference pipeline in accordance with at least one embodiment. In at least one embodiment, the process 900 can be deployed to perform game name recognition analysis and inference on user feedback data at one or more facilities 902, such as a data center.

[0082] In at least one embodiment, process 900 can be performed within training system 904 and / or deployment system 906. In at least one embodiment, training system 904 can be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use with deployment system 906. In at least one embodiment, deployment system 906 can be configured to offload processing and computing resources in a distributed computing environment to reduce infrastructure requirements at facility 902. In at least one embodiment, deployment system 906 can provide a pipeline platform for selecting, customizing, and implementing virtual instruments for use with computing devices at facility 902. In at least one embodiment, a virtual instrument can include a software-defined application for performing one or more processing operations with respect to feedback data. In at least one embodiment, one or more applications in the pipeline can use or call services (e.g., reasoning, visualization, computation, AI, etc.) of deployment system 906 during application execution.

[0083] In at least one embodiment, some applications used in the high-level processing and reasoning pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at the facility 902 using feedback data 908 (e.g., feedback data) stored at the facility 902, or may be trained using feedback data 908 from another or more facilities, or a combination thereof. In at least one embodiment, the training system 904 may be used to provide applications, services, and / or other resources to generate a working, deployable machine learning model for the deployment system 906.

[0084] In at least one embodiment, the model registry 924 can be backed by an object store that can support version control and object metadata. In at least one embodiment, the model registry 924 can be accessed from within the cloud platform through, for example, a cloud store (e.g., Figure 10 The object store is accessed through an application programming interface (API) compatible with the cloud 1026. In at least one embodiment, machine learning models within the model registry 924 can be uploaded, listed, modified, or deleted by developers or partners of the system interacting with the API. In at least one embodiment, the API can provide access to methods that allow users with appropriate credentials to associate a model with an application so that the model can be executed as part of the execution of a containerized instantiation of the application.

[0085] In at least one embodiment, the training pipeline 1004 ( Figure 10) may include scenarios where a facility 902 is training its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, feedback data 908 may be received from various channels (e.g., forums, web forms, etc.). In at least one embodiment, once feedback data 908 is received, AI-assisted annotation 910 may be used to help generate annotations corresponding to the feedback data 908 to serve as ground truth data for the machine learning model. In at least one embodiment, AI-assisted annotation 910 may include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that can be trained to generate annotations corresponding to certain types of feedback data 908 (e.g., from certain devices) and / or certain types of anomalies in the feedback data 908. In at least one embodiment, AI-assisted annotation 910 can then be used directly or adjusted or fine-tuned using annotation tools to generate ground truth data. In at least one embodiment, in some examples, labeled data 912 may be used as ground truth data for training the machine learning model. In at least one embodiment, AI-assisted annotation 910, labeled data 912, or a combination thereof may be used as ground truth data for training the machine learning model. In at least one embodiment, the trained machine learning model may be referred to as an output model 916 and may be used by the deployment system 906 as described herein.

[0086] In at least one embodiment, the training pipeline 1004 ( Figure 10) may include situations where facility 902 requires a machine learning model for performing one or more processing tasks for one or more applications in deployment system 906, but facility 902 may not currently have such a machine learning model (or may not have a model optimized, efficient, or effective for this purpose). In at least one embodiment, an existing machine learning model can be selected from model registry 924. In at least one embodiment, model registry 924 may include machine learning models that have been trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, the machine learning models in model registry 924 may be trained on imaging data from a different facility (e.g., a remotely located facility) than facility 902. In at least one embodiment, the machine learning model may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when trained on imaging data from a specific location, the training may be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts its transfer off-site (e.g., to comply with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model is trained or partially trained at one location, the machine learning model can be added to the model registry 924. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be used in the model registry 924. In at least one embodiment, the machine learning model can then be selected from the model registry 924 (and referred to as the output model 916) and can be deployed in the deployment system 906 to perform one or more processing tasks for one or more applications of the deployment system.

[0087] In at least one embodiment, the training pipeline 1004 ( Figure 10) can be used in scenarios including a facility 902 that requires a machine learning model for performing one or more processing tasks for one or more applications in a deployment system 906, but the facility 902 may not currently possess such a machine learning model (or may not possess an optimized, efficient, or effective model). In at least one embodiment, a machine learning model selected from the model registry 924 may not be fine-tuned or optimized for the feedback data 908 generated at the facility 902 due to population differences, genetic variation, robustness, diversity of training data anomalies, and / or other issues with the training data used to train the machine learning model. In at least one embodiment, AI-assisted annotation 910 can be used to help generate annotations corresponding to the feedback data 908 for use as ground truth data for training or updating the machine learning model. In at least one embodiment, labeled data 912 can be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 914. In at least one embodiment, model training 914 (e.g., AI-assisted annotation 910, labeled data 912, or a combination thereof) can be used as ground truth data for retraining or updating the machine learning model.

[0088] In at least one embodiment, deployment system 906 may include software 918, services 920, hardware 922, and / or other components, features, and functionality. In at least one embodiment, deployment system 906 may include a software "stack" such that software 918 may be built on top of services 920 and may use services 920 to perform some or all processing tasks, and services 920 and software 918 may be built on top of hardware 922 and use hardware 922 to perform processing, storage, and / or other computing tasks of deployment system 906.

[0089] In at least one embodiment, the software 918 may include any number of different containers, each of which may execute an instantiation of an application. In at least one embodiment, each application may execute one or more processing tasks (e.g., reasoning, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in a high-level processing and reasoning pipeline. In at least one embodiment, for each type of computing device, there may be any number of containers that may execute data processing tasks on the feedback data 908 (or other data types, such as those described herein). In at least one embodiment, in addition to receiving and configuring imaging data for use by each container and / or for use by the facility 902 after processing through the pipeline, a high-level processing and reasoning pipeline may be defined based on the selection of different containers desired or required to process the feedback data 908 (e.g., to convert the output back into a usable data type for storage and display at the facility 902). In at least one embodiment, the combination of containers within the software 918 (e.g., that comprise the pipeline) may be referred to as a virtual instrument (as described in more detail herein), and the virtual instrument may utilize services 920 and hardware 922 to execute some or all of the processing tasks of the application instantiated in the container.

[0090] In at least one embodiment, data can be pre-processed as part of a data processing pipeline to prepare the data for processing by one or more applications. In at least one embodiment, post-processing can be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare output data for the next application and / or prepare the output data for transmission and / or use by a user (e.g., as a response to an inference request). In at least one embodiment, the inference task can be performed by one or more machine learning models, such as trained or deployed neural networks, which can include the output model 916 of the training system 904.

[0091] In at least one embodiment, tasks of a data processing pipeline can be encapsulated in containers, with each container representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing a machine learning model. In at least one embodiment, containers or applications can be published to a private (e.g., limited access) area of a container registry (described in more detail herein), and trained or deployed models can be stored in the model registry 924 and associated with one or more applications. In at least one embodiment, an image of an application (e.g., a container image) can be made available in a container registry, and once a user selects an image from the container registry for deployment in a pipeline, the image can be used to generate an instantiated container for the application for use by the user's system.

[0092] In at least one embodiment, a developer can develop, publish, and store applications (e.g., as containers) for performing processing and / or reasoning on provided data. In at least one embodiment, development, publishing, and / or storage can be performed using a software development kit (SDK) associated with the system (e.g., to ensure that the developed applications and / or containers conform to or are compatible with the system). In at least one embodiment, the developed applications can be tested locally (e.g., at a first facility, on data from the first facility) using the SDK, which serves as a system (e.g., Figure 10 The system 1000 in FIG. 1000 may support at least some of the services 920. In at least one embodiment, once validated by the system 1000 (e.g., for accuracy, etc.), the application is made available in the container registry for selection and / or implementation by a user (e.g., a hospital, clinic, laboratory, healthcare provider, etc.) to perform one or more processing tasks on data at the user's facility (e.g., a second facility).

[0093] In at least one embodiment, the developer can then share the application or container over a network for use by a system (e.g., Figure 10 924). In at least one embodiment, completed and validated applications or containers can be stored in a container registry, and associated machine learning models can be stored in a model registry 924. In at least one embodiment, a requesting entity (providing an inference or image processing request) can browse the container registry and / or model registry 924 to obtain applications, containers, datasets, machine learning models, etc., select the desired combination of elements to include in a data processing pipeline, and submit a processing request. In at least one embodiment, the request can include the input data necessary to execute the request and / or can include a selection of the application and / or machine learning model to be executed when processing the request. In at least one embodiment, the request can then be passed to one or more components of the deployment system 906 (e.g., a cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 906 can include referencing elements (e.g., applications, containers, models, etc.) selected from the container registry and / or model registry 924. In at least one embodiment, once results are generated by the pipeline, the results can be returned to the user for reference (e.g., for viewing in a viewing application suite executed locally, on a local workstation, or on a terminal).

[0094] In at least one embodiment, to assist in processing or executing applications or containers in the pipeline, services 920 may be utilized. In at least one embodiment, services 920 may include computing services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, services 920 may provide functionality that is common to one or more applications in software 918, thereby abstracting functionality into services that can be called or utilized by applications. In at least one embodiment, the functionality provided by services 920 may run dynamically and more efficiently, while also enabling applications to process data in parallel (e.g., using Figure 10 In at least one embodiment, rather than requiring each application that shares the same functionality provided by service 920 to have a corresponding instance of service 920, service 920 can be shared between and among various applications. In at least one embodiment, as non-limiting examples, the service may include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included that can provide machine learning model training and / or retraining capabilities.

[0095] In at least one embodiment, where the service 920 includes an AI service (e.g., an inference service), as part of the execution of an application, one or more machine learning models associated with an application for anomaly detection (e.g., tumor, growth abnormality, scarring, etc.) can be executed by calling (e.g., as an API call) the inference service (e.g., an inference server) to execute one or more machine learning models or processing thereof. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, the application can call the inference service to execute the machine learning model for performing one or more processing operations associated with the segmentation task. In at least one embodiment, the software 918 implementing the high-level processing and inference pipeline can be pipelined in that each application can call the same inference service to perform one or more inference tasks.

[0096] In at least one embodiment, the hardware 922 may include a GPU, a CPU, a graphics card, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX supercomputer system), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 922 may be used to provide efficient, purpose-built support for the software 918 and services 920 in the deployment system 906. In at least one embodiment, the use of GPU processing may be implemented to perform local processing (e.g., at the facility 902) within the AI / deep learning system, in the cloud system, and / or in other processing components of the deployment system 906 to improve the efficiency, accuracy, and effectiveness of game name recognition.

[0097] In at least one embodiment, as a non-limiting example, with respect to deep learning, machine learning, and / or high-performance computing, software 918 and / or services 920 may be optimized for GPU processing. In at least one embodiment, at least some of the computing environments of deployment system 906 and / or training system 904 may be executed in a data center, one or more supercomputers, or high-performance computer systems with GPU-optimized software (e.g., the hardware and software combination of an NVIDIA DGX system). In at least one embodiment, as described herein, hardware 922 may include any number of GPUs that can be deployed to perform data processing in parallel. In at least one embodiment, the cloud platform may also include GPU-optimized execution for deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, a cloud platform (e.g., NVIDIA's NGC) may be implemented using an AI / deep learning supercomputer and / or GPU-optimized software (e.g., as provided on NVIDIA's DGX system) as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform may integrate an application container clustering system or orchestration system (e.g., Kubernetes) across multiple GPUs to achieve seamless scaling and load balancing.

[0098] Figure 10 is a system diagram of an example system 1000 for generating and deploying a deployment pipeline according to at least one embodiment. In at least one embodiment, the system 1000 can be used to implement Figure 9 The system 1000 may include a training system 904 and a deployment system 906. In at least one embodiment, the training system 904 and the deployment system 906 may be implemented using software 918, services 920, and / or hardware 922, as described herein.

[0099] In at least one embodiment, system 1000 (e.g., training system 904 and / or deployment system 906) can be implemented in a cloud computing environment (e.g., using cloud 1026). In at least one embodiment, system 1000 can be implemented locally (with respect to a facility) or as a combination of cloud computing resources and local computing resources. In at least one embodiment, access to APIs in cloud 1026 can be restricted to authorized users by establishing security measures or protocols. In at least one embodiment, the security protocol can include a network token, which can be signed by an authentication (e.g., AuthN, AuthZ, Gluecon, etc.) service and can carry appropriate authorization. In at least one embodiment, the API of the virtual instrument (described herein) or other instances of system 1000 can be restricted to a set of public IPs that have been audited or authorized for interaction.

[0100] In at least one embodiment, the various components of system 1000 can communicate with each other using any of a variety of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communications between facilities and components of system 1000 (e.g., for sending inference requests, for receiving results of inference requests, etc.) can be transmitted via one or more data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0101] In at least one embodiment, similar to the present disclosure regarding Figure 9 As described, the training system 904 can execute a training pipeline 1004. In at least one embodiment, where the deployment system 906 will use one or more machine learning models in a deployment pipeline 1010, the training pipeline 1004 can be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1006 (e.g., without retraining or updating). In at least one embodiment, as a result of the training pipeline 1004, an output model 916 can be generated. In at least one embodiment, the training pipeline 1004 can include any number of processing steps, AI-assisted annotation 910, labeling or annotation of feedback data 908 (to generate labeled data 912), selecting a model from a model registry, model training 914, training, retraining, or updating a model, and / or other processing steps. In at least one embodiment, different training pipelines 1004 can be used for different machine learning models used by the deployment system 906. In at least one embodiment, similar to the description regarding Figure 9 The training pipeline 1004 of the first example described can be used for a first machine learning model, similar to the one described with respect to Figure 9The second example training pipeline 1004 described can be used for a second machine learning model, similar to the one described with respect to Figure 9 The third example training pipeline 1004 is described as being usable for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 904 may be used, depending on the requirements of each respective machine learning model. In at least one embodiment, one or more machine learning models may already be trained and ready for deployment, so the training system 904 may not perform any processing on the machine learning models, and the one or more machine learning models may be implemented by the deployment system 906.

[0102] In at least one embodiment, the output model 916 and / or the pre-trained model 1006 may include any type of machine learning model according to one or more embodiments. In at least one embodiment and without limitation, the machine learning model used by the system 1000 may include linear regression, logistic regression, decision tree, support vector machine (SVM), naive Bayes, k-nearest neighbor (Knn), k-means clustering, random forest, dimensionality reduction algorithm, gradient boosting algorithm, neural network (e.g., autoencoder, convolution, recursion, perceptron, long / short term memory (LSTM), dual-LSTM, Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machine, etc.), and / or other types of machine learning models.

[0103] In at least one embodiment, the training pipeline 1004 may include AI-assisted annotation. In at least one embodiment, the labeled clinical data 912 may be generated by any number of techniques (e.g., traditional annotation). In at least one embodiment, in some examples, labels or other annotations may be generated in a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of application suitable for generating ground-truth annotations or labels, and / or may be hand-drawn. In at least one embodiment, the ground-truth data may be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data), machine-generated (e.g., using feature analysis and learning to extract features from the data and then generate labels), manually annotated (e.g., a labeler or annotator expert defines the location of the labels), and / or a combination thereof. In at least one embodiment, for each instance of feedback data 908 (or other data type used by the machine learning model), there may be corresponding ground-truth data generated by the training system 904. In at least one embodiment, AI-assisted annotation may be performed as part of the deployment pipeline 1010; in addition to or in place of the AI-assisted annotation included in the training pipeline 1004. In at least one embodiment, system 1000 may include a multi-layer platform that may include a software layer (eg, software 918 ) of a diagnostic application (or other application type) that may perform one or more medical imaging and diagnostic functions.

[0104] In at least one embodiment, the software layer can be implemented as a secure, encrypted, and / or authenticated API through which an application or container can be invoked (e.g., called) from an external environment (e.g., facility 902). In at least one embodiment, the application can then call or execute one or more services 920 to perform computational, AI, or visualization tasks associated with the respective application, and the software 918 and / or services 920 can utilize hardware 922 to perform the processing tasks in an effective and efficient manner.

[0105] In at least one embodiment, the deployment system 906 can execute a deployment pipeline 1010. In at least one embodiment, the deployment pipeline 1010 can include any number of applications that can be sequential, non-sequential, or otherwise applied to feedback data (and / or other data types) - including AI-assisted annotations, as described above. In at least one embodiment, the deployment pipeline 1010 for an individual device, as described herein, can be referred to as a virtual instrument for the device. In at least one embodiment, there can be more than one deployment pipeline 1010 for a single device, depending on the information desired from the data generated by the device.

[0106] In at least one embodiment, applications that can be used to deploy pipeline 1010 can include any application that can be used to perform processing tasks on feedback data or other data from the device. In at least one embodiment, because various applications can share common image operations, in some embodiments, a data enhancement library (e.g., as one of the services 920) can be used to accelerate these operations. In at least one embodiment, in order to avoid the bottlenecks of traditional processing methods that rely on CPU processing, parallel computing platform 1030 can be used for GPU acceleration of these processing tasks.

[0107] In at least one embodiment, the deployment system 906 may include a user interface 1014 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications to be included in the deployment pipeline 1010, to arrange applications, to modify or change applications or their parameters or configuration, to use and interact with the deployment pipeline 1010 during setup and / or deployment, and / or to otherwise interact with the deployment system 906. In at least one embodiment, although not shown with respect to the training system 904, the user interface 1014 (or a different user interface) can be used to select models for use in the deployment system 906, to select models for training or retraining in the training system 904, and / or to otherwise interact with the training system 904.

[0108] In at least one embodiment, in addition to the application orchestration system 1028, a pipeline manager 1012 can be used to manage the interactions between applications or containers of the deployment pipeline 1010 and services 920 and / or hardware 922. In at least one embodiment, the pipeline manager 1012 can be configured to facilitate interactions from application to application, from application to service 920, and / or from application or service to hardware 922. In at least one embodiment, although shown as included in the software 918, this is not intended to be limiting, and in some examples, the pipeline manager 1012 can be included in the service 920. In at least one embodiment, the application orchestration system 1028 (e.g., Kubernetes, Docker, etc.) can include a container orchestration system that can group applications into containers as a logical unit for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline 1010 (e.g., rebuilding an application, partitioning an application, etc.) with individual containers, each application can execute in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0109] In at least one embodiment, each application and / or container (or its image) can be developed, modified, and deployed separately (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer), which can allow for focus and attention on the tasks of a single application and / or container without being hindered by the tasks of another application or container. In at least one embodiment, the pipeline manager 1012 and the application coordination system 1028 can facilitate communication and collaboration between different containers or applications. In at least one embodiment, the application coordination system 1028 and / or the pipeline manager 1012 can facilitate communication and resource sharing between and among each application or container, as long as the expected inputs and / or outputs of each container or application are known to the system (e.g., based on the configuration of the application or container). In at least one embodiment, because one or more applications or containers in the deployment pipeline 1010 can share the same services and resources, the application coordination system 1028 can coordinate, load balance, and determine the sharing of services or resources between and among the various applications or containers. In at least one embodiment, the scheduler can be used to track resource requirements of applications or containers, current or planned usage of those resources, and resource availability. Thus, in at least one embodiment, the scheduler can allocate resources to different applications and distribute resources between and among applications, taking into account the needs and availability of the system. In some examples, the scheduler (and / or other components of the application coordination system 1028) can determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as quality of service (QoS), the urgency of data output (e.g., to determine whether to perform real-time processing or delayed processing), etc.

[0110] In at least one embodiment, the services 920 utilized and shared by applications or containers in the deployment system 906 may include compute services 1016, AI services 1018, visualization services 1020, and / or other service types. In at least one embodiment, an application may call (e.g., execute) one or more services 920 to perform processing operations for the application. In at least one embodiment, an application may utilize compute services 1016 to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more compute services 1016 may be utilized to perform parallel processing (e.g., using a parallel computing platform 1030) to process data substantially simultaneously by one or more applications and / or one or more tasks of a single application. In at least one embodiment, a parallel computing platform 1030 (e.g., NVIDIA's CUDA) may implement general-purpose computing on a GPU (GPGPU) (e.g., GPU 1022). In at least one embodiment, a software layer of the parallel computing platform 1030 may provide access to a virtual instruction set and parallel computing elements of the GPU to execute computational kernels. In at least one embodiment, the parallel computing platform 1030 may include memory, and in some embodiments, the memory may be shared between and among multiple containers, and / or between and among different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within a container to use the same data from a shared memory segment of the parallel computing platform 1030 (e.g., where multiple different stages of an application or multiple applications are processing the same information). In at least one embodiment, rather than copying and moving data to different locations in memory (e.g., read / write operations), the same data in the same memory location may be used by any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, since data is used to generate new data as a result of processing, this information about the new location of the data may be stored and shared between the various applications. In at least one embodiment, the location of the data, as well as the location of updated or modified data, may be part of the definition of how the payload in the container is understood.

[0111] In at least one embodiment, AI service 1018 can be utilized to perform inference services for executing machine learning models associated with an application (e.g., tasked with executing one or more processing tasks of the application). In at least one embodiment, AI service 1018 can utilize AI system 1024 to execute machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, an application in deployment pipeline 1010 can use one or more output models 916 from training system 904 and / or other models of the application to perform inference on imaging data (e.g., DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.). In at least one embodiment, two or more instances of inference using application coordination system 1028 (e.g., a scheduler) can be available. In at least one embodiment, the first category can include high-priority / low-latency paths that can achieve higher service level agreements, such as for performing inference on urgent requests in emergency situations or for radiologists during diagnostic procedures. In at least one embodiment, the second category may include a standard priority path that may be used for requests that may not be urgent or where analysis can be performed at a later time. In at least one embodiment, the application coordination system 1028 may allocate resources (e.g., services 920 and / or hardware 922) for different reasoning tasks of the AI service 1018 based on the priority path.

[0112] In at least one embodiment, shared memory can be installed into the AI service 1018 in the system 1000. In at least one embodiment, the shared memory can operate as a cache (or other storage device type) and can be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a set of API instances of the deployment system 906 can receive the request and select one or more instances (e.g., for best fit, load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request can be entered into a database, and if it is not already in the cache, the machine learning model can be located from the model registry 924. A validation step can ensure that the appropriate machine learning model is loaded into the cache (e.g., shared storage), and / or a copy of the model can be saved to the cache. In at least one embodiment, if the application is not already running or there are not enough instances of the application, a scheduler (e.g., a scheduler of the pipeline manager 1012) can be used to start the application referenced in the request. In at least one embodiment, if an inference server has not yet been started to execute the model, an inference server can be started. In at least one embodiment, any number of inference servers can be started for each model. In at least one embodiment, in a pull model where the inference servers are clustered, the model can be cached whenever load balancing is beneficial. In at least one embodiment, the inference servers can be statically loaded into the corresponding distributed servers.

[0113] In at least one embodiment, inference can be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server can be associated with a model (and optionally with multiple versions of a model). In at least one embodiment, if an instance of an inference server does not exist when a request to perform inference on a model is received, a new instance can be loaded. In at least one embodiment, a model can be passed to the inference server when it is started, so that the same container can be used to serve different models as long as the inference server is running as a different instance.

[0114] In at least one embodiment, during application execution, an inference request for a given application may be received, a container (e.g., an instance hosting an inference server) may be loaded (if not already loaded), and a launcher may be called. In at least one embodiment, pre-processing logic in the container may load, decode, and / or perform any additional pre-processing on the incoming data (e.g., using the CPU and / or GPU). In at least one embodiment, once the data is ready for inference, the container may perform inference on the data as needed. In at least one embodiment, this may include a single inference call for a single image (e.g., a hand X-ray), or may request inference on hundreds of images (e.g., a chest CT scan). In at least one embodiment, the application may summarize the results before completion, which may include, but is not limited to, a single confidence score, pixel-level segmentation, voxel-level segmentation, generated visualizations, or generated text summarizing the results. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have real-time (TAT less than 1 minute) priority, while other models may have lower priority (e.g., TAT less than 10 minutes). In at least one embodiment, model execution time may be measured from the requesting mechanism or entity and may include collaborative network traversal time as well as execution time of the inference service.

[0115] In at least one embodiment, the transmission of requests between the service 920 and the inference application can be hidden behind a software development kit (SDK) and can provide robust transport via queues. In at least one embodiment, requests are placed in a queue for individual application / tenant ID combinations via an API, and the SDK pulls the request from the queue and delivers it to the application. In at least one embodiment, the name of the queue can be provided in the context of the queue from which the SDK will pick up work. In at least one embodiment, asynchronous communication via queues can be useful because it allows any instance of the application to pick up work when it is available. In at least one embodiment, results can be transmitted back through the queue to ensure no data is lost. In at least one embodiment, queues can also provide the ability to partition work, as the highest priority work can enter a queue connected to the majority of instances of the application, while the lowest priority work can enter a queue connected to a single instance, which processes the tasks in the order they are received. In at least one embodiment, the application can run on a GPU-accelerated instance generated in the cloud 1026, and the inference service can perform inference on the GPU.

[0116] In at least one embodiment, visualization services 1020 can be utilized to generate visualizations for viewing application and / or deployment pipeline 1010 outputs. In at least one embodiment, visualization services 1020 can utilize GPU 1022 to generate visualizations. In at least one embodiment, visualization services 1020 can implement rendering effects such as ray tracing to generate higher quality visualizations. In at least one embodiment, visualizations can include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slices, virtual reality displays, augmented reality displays, and the like. In at least one embodiment, a virtualized environment can be used to generate virtual interactive displays or environments (e.g., virtual environments) for system users (e.g., doctors, nurses, radiologists, etc.) to interact with. In at least one embodiment, visualization services 1020 can include internal visualizers, movies, and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0117] In at least one embodiment, hardware 922 may include GPUs 1022, AI system 1024, cloud 1026, and / or any other hardware used to execute training system 904 and / or deployment system 906. In at least one embodiment, GPUs 1022 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that can be used to perform processing tasks for any feature or functionality of compute services 1016, AI services 1018, visualization services 1020, other services, and / or software 918. For example, for AI services 1018, GPUs 1022 may be used to perform pre-processing on imaging data (or other data types used by machine learning models), post-processing on the output of machine learning models, and / or perform inference (e.g., to execute machine learning models). In at least one embodiment, cloud 1026, AI system 1024, and / or other components of system 1000 may use GPUs 1022. In at least one embodiment, cloud 1026 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1024 can use a GPU, and cloud 1026 (or at least part of the task being deep learning or inference) can be executed using one or more AI systems 1024. Likewise, while hardware 922 is shown as discrete components, this is not intended to be limiting, and any component of hardware 922 can be combined with or utilized by any other component of hardware 922.

[0118] In at least one embodiment, AI system 1024 may include a purpose-built computing system (e.g., a supercomputer or HPC) configured for reasoning, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, in addition to CPU, RAM, storage, and / or other components, features, or functions, AI system 1024 (e.g., NVIDIA's DGX) may also include software (e.g., a software stack) that can use multiple GPUs 1022 to execute GPU-optimized software. In at least one embodiment, one or more AI systems 1024 may be implemented in a cloud 1026 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1000.

[0119] In at least one embodiment, cloud 1026 may include a GPU-accelerated infrastructure (e.g., NVIDIA's NGC) that may provide a GPU-optimized platform for executing processing tasks of system 1000. In at least one embodiment, cloud 1026 may include an AI system 1024 for executing one or more AI-based tasks of system 1000 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1026 may be integrated with an application orchestration system 1028 that utilizes multiple GPUs to enable seamless scaling and load balancing between and among applications and services 920. In at least one embodiment, cloud 1026 may be responsible for executing at least some services 920 of system 1000, including compute services 1016, AI services 1018, and / or visualization services 1020, as described herein. In at least one embodiment, the cloud 1026 can perform large and small batch inference (e.g., executing NVIDIA's TENSOR RT), provide accelerated parallel computing APIs and platforms 1030 (e.g., NVIDIA's CUDA), execute application coordination system 1028 (e.g., Kubernetes), provide graphics rendering APIs and platforms (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality cinematic effects), and / or can provide other functionality for the system 1000.

[0120] In at least one embodiment, to protect patient confidentiality (e.g., in situations where patient data or records are used off-site), the cloud 1026 may include a registry - such as a deep learning container registry. In at least one embodiment, the registry may store containers for instantiating applications that may perform pre-processing, post-processing, or other processing tasks on the patient data. In at least one embodiment, the cloud 1026 may receive data that includes patient data as well as sensor data in containers, perform the requested processing only on the sensor data in those containers, and then forward the resulting output and / or visualization to the appropriate parties and / or devices (e.g., local medical devices for visualization or diagnosis) without extracting, storing, or otherwise accessing the patient data. In at least one embodiment, the confidentiality of the patient data is preserved in accordance with HIPAA and / or other data regulations.

[0121] At least one embodiment of the present disclosure may be described in terms of the following:

[0122] In clause 1, the method includes: applying an input image to a first machine learning model MLM to identify multiple regions of interest (ROIs) within the input image, wherein the first MLM is trained to identify the multiple ROIs based at least in part on multiple reference features detected within the input image and one or more geometric relationships between each reference feature and one or more locations associated with one of a plurality of ROI types; and providing each of the identified multiple ROIs as input to a corresponding second MLM among a plurality of second MLMs to obtain an output of the corresponding second MLM, wherein the corresponding second MLM corresponds to one of the plurality of ROI types, and the output of the corresponding second MLM indicates one or more objects within the ROI of the corresponding ROI type.

[0123] In clause 2, a method according to clause 1, wherein the input image is a multidimensional image of a first dimension, and the first MLM is trained to detect the plurality of reference features based at least in part on processing a plurality of cross-sectional images associated with the multidimensional image, each of the plurality of cross-sectional images having a second dimension and representing a cross-section of the input image, and wherein the second dimension is lower than the first dimension.

[0124] In clause 3, the method of clause 1, wherein the first MLM is trained based on a plurality of training images having reference features of a type common to the type of the reference features within the input image.

[0125] In Clause 4, the method of Clause 1, wherein the first MLM comprises a neural network having at least one hidden layer.

[0126] In clause 5, the method of clause 1, wherein the input image is a medical image of a patient, and the plurality of reference features are associated with one or more bones of the patient depicted by the input image.

[0127] In Clause 6, the method of Clause 5, wherein a first ROI of the plurality of ROIs includes a representation of at least a portion of a first organ of the patient and does not include a representation of any portion of a second organ of the patient.

[0128] In clause 7, the method of clause 6, wherein a second ROI in the plurality of ROIs comprises a representation of at least a portion of the second organ.

[0129] In clause 8, the method of clause 1, wherein providing each of the plurality of ROIs to the corresponding second MLM of the plurality of second MLMs comprises providing at least one of a location of the corresponding ROI within the input image or a representation of the corresponding ROI.

[0130] In Clause 9, the method of Clause 1, wherein applying the first MLM to the input image comprises performing one or more calculations associated with the first MLM on one or more graphics processing units.

[0131] In clause 10, the method comprises: generating first training data comprising a first training input and a first target output, wherein the first training input comprises an intensity map of a first training image and the first target output comprises first annotation data identifying a plurality of regions of interest (ROIs) in the first training image; providing the first training data for training a segmentation machine learning model (MLM) on a first set of training inputs comprising the first training input and a first set of target outputs comprising the first target output, wherein the segmentation MLM uses (i) the intensity map to identify a plurality of reference features within the first training image and (ii) the annotation data to identify each reference feature and one or more corresponding ROIs of the plurality of ROIs one or more geometric relationships between; generating second training data including a second training input and a second target output, the second training input including a second training image representing a first ROI among the multiple ROIs, and the second target output including second annotation data identifying one or more objects within the first ROI, the first ROI having a first ROI type among a plurality of ROI types; and providing the second training data to train a first object recognition MLM among a plurality of object recognition MLMs on a second set of training inputs including the second training input and a second set of target outputs including the second target output, wherein the first object recognition MLM is trained to recognize objects in images of the first ROI type.

[0132] In clause 11, the method according to clause 10 further includes: generating third training data including a third training input and a third target output, the third training input including a third training image representing a second ROI among the multiple ROIs, and the third target output including third annotation data identifying one or more objects within the second ROI, the second ROI having a second ROI type among the multiple ROI types; and providing the third training data for training a second object recognition MLM among the multiple object recognition MLMs on a third group of training inputs including the third training input and a third group of target outputs including the third target output, wherein the second object recognition MLM is trained to recognize objects in images of the second ROI type.

[0133] In clause 12, the method of clause 10, wherein the first training image is a medical image of a patient.

[0134] In clause 13, the system includes: a memory device; and one or more processing devices communicatively coupled to the memory device, configured to: apply an input image to a first machine learning model MLM to identify a plurality of regions of interest (ROIs) within the input image, wherein the MLM is trained to identify the plurality of ROIs based at least in part on a plurality of reference features detected within the input image and one or more geometric relationships between each reference feature and one or more locations associated with one of a plurality of ROI types; and provide each of the identified plurality of ROIs as input to a corresponding second MLM among a plurality of second MLMs to obtain an output of the corresponding second MLM, wherein the corresponding second MLM corresponds to one of the plurality of ROI types, and the output of the corresponding second MLM indicates one or more objects within the ROI of the corresponding ROI type.

[0135] In clause 14, a system according to clause 13, wherein the input image is a multidimensional image of a first dimension, and the first MLM is trained to detect the plurality of reference features based at least in part on processing a plurality of cross-sectional images associated with the multidimensional image, each of the plurality of cross-sectional images having a second dimension and representing a cross-section of the input image, and wherein the second dimension is lower than the first dimension.

[0136] In clause 15, the system of clause 14, wherein the first MLM is trained based on a plurality of training images having reference features of a type common to the type of the reference features within the input image.

[0137] In clause 16, the system of clause 13, wherein the input image is a medical image of a patient, and the plurality of reference features are associated with one or more bones of the patient depicted by the input image.

[0138] In clause 17, the system of clause 16, wherein a first ROI of the plurality of ROIs includes a representation of at least a portion of a first organ of the patient and does not include a representation of any portion of a second organ of the patient.

[0139] In clause 18, the system of clause 17, wherein the one or more processing devices comprises a graphics processing unit.

[0140] In clause 19, a non-transitory computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processing device, cause the processing device to: apply an input image to a first machine learning model MLM to identify a plurality of regions of interest (ROIs) within the input image, wherein the first MLM is trained to identify the plurality of ROIs based at least in part on a plurality of reference features detected within the input image and one or more geometric relationships between each reference feature and one or more locations associated with one of a plurality of ROI types; and provide each of the identified plurality of ROIs as input to a corresponding second MLM among a plurality of second MLMs to obtain an output of the corresponding second MLM, wherein the corresponding second MLM corresponds to one of the plurality of ROI types, and the output of the corresponding second MLM indicates one or more objects within the ROI of the corresponding ROI type.

[0141] In clause 20, a non-transitory computer-readable medium according to clause 19, wherein the input image is a multidimensional image of a first dimension, and the first MLM is trained to detect the plurality of reference features based at least in part on processing a plurality of cross-sectional images associated with the multidimensional image, each of the plurality of cross-sectional images having a second dimension and representing a cross-section of the input image, and wherein the second dimension is lower than the first dimension.

[0142] In clause 21, the non-transitory computer-readable medium of clause 19, wherein the first MLM is trained based on a plurality of training images having reference features of a type common to a type of the reference features within the input image.

[0143] In clause 22, the non-transitory computer-readable medium of clause 19, wherein the input image is a medical image of a patient, and the plurality of reference features are associated with one or more bones of the patient depicted by the input image.

[0144] In clause 23, the non-transitory computer-readable medium of clause 22, wherein a first ROI of the plurality of ROIs includes a representation of at least a portion of a first organ of the patient and does not include a representation of any portion of a second organ of the patient.

[0145] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. However, it should be understood that there is no intention to limit the disclosure to one or more specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0146] Unless otherwise noted or clearly contradicted by the context, the use of the terms "a" and "an" and "the" and similar references in the context of describing the disclosed embodiments (particularly in the context of the appended claims) should be interpreted as covering the singular and plural, rather than as definitions of terms. Unless otherwise noted, the terms "include," "have," "include," and "contain" should be interpreted as open-ended terms (meaning "including but not limited to"), unless otherwise noted. The term "connected" (when unmodified, refers to a physical connection) should be interpreted as partially or completely contained within, attached to, or connected together, even if there is some intervention. Unless otherwise noted herein, references to numerical ranges herein are intended only to be used as a shorthand method of referring to each individual value falling within the range, and each individual value is incorporated into the specification as if it were separately recited herein. In at least one embodiment, unless otherwise noted or contradicted by the context, the use of the term "set" (e.g., "item set") or "subset" should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but rather a subset and a corresponding set may be equivalent.

[0147] Unless expressly indicated otherwise or clearly contradicted by context, conjunctions such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C" are understood in context to generally refer to an item, clause, or the like, which may be A or B or C, or any non-empty subset of the set of A, B, and C. For example, in the illustrative example of a set having three members, the conjunctions "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctions are not generally intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless expressly indicated otherwise or contradicted by context, the term "plurality" refers to a plurality (e.g., "a plurality of items" refers to a plurality of items). In at least one embodiment, the number of items in the plurality of items is at least two, but may be more if expressly indicated or indicated by context. Further, unless stated otherwise or clear from context, the phrase "based on" means "based at least in part on" rather than "based solely on."

[0148] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that are collectively executed on one or more processors by hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium in the form of, for example, a computer program that includes a plurality of instructions that can be executed by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagated transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuits (e.g., buffers, caches, and queues). In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) having executable instructions stored thereon, which, when executed by one or more processors of a computer system (i.e., as a result of being executed), causes the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media in the plurality of non-transitory computer-readable storage media lacks all of the code, but rather the plurality of non-transitory computer-readable storage media collectively store all of the code. In at least one embodiment, the executable instructions are executed so that different instructions are executed by different processors, for example, a non-transitory computer-readable storage medium stores instructions, and a main central processing unit ("CPU") executes some instructions, while a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and different processors execute different subsets of instructions.

[0149] Thus, in at least one embodiment, a computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and / or software that enables the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment is a distributed computer system comprising multiple devices operating in different ways such that the distributed computer system performs the operations described herein and such that no single device performs all of the operations.

[0150] The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended merely to better illuminate embodiments of the present disclosure and does not limit the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0151] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0152] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms may not be intended as synonyms for each other. Rather, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0153] Unless expressly stated otherwise, it is understood that throughout this specification, terms such as “process,” “calculate,” “compute,” “determine,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that processes and / or converts data represented as physical quantities (e.g., electronic) in the registers and / or memories of the computing system into other data similarly represented as physical quantities in the memories, registers, or other such information storage, transmission, or display devices of the computing system.

[0154] In a similar manner, the term "processor" may refer to any device or portion of memory that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As non-limiting examples, a "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process may refer to multiple processes to execute instructions continuously or intermittently, sequentially, or in parallel. In at least one embodiment, the terms "system" and "method" may be used interchangeably herein, as long as a system may embody one or more methods, and a method may be considered a system.

[0155] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in a variety of ways, such as by receiving data as parameters of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be accomplished by transmitting data as input or output parameters of a function call, an application programming interface, or an interprocess communication mechanism.

[0156] Although the description herein sets forth example embodiments of the described technology, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities are defined above for descriptive purposes, the various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0157] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A method for detecting a target object, comprising: applying an input image to a first machine learning model (MLM) to identify a plurality of regions of interest (ROIs) within the input image, wherein the first MLM is trained to identify the plurality of ROIs based at least in part on a plurality of reference features detected within the input image and one or more geometric relationships between each reference feature and one or more locations associated with one of a plurality of ROI types; as well as Each of the identified multiple ROIs is provided as input to a corresponding second MLM among a plurality of second MLMs to obtain an output of the corresponding second MLM, wherein the corresponding second MLM corresponds to one of the multiple ROI types, and the output of the corresponding second MLM indicates one or more objects within the ROI of the corresponding ROI type.

2. The method of claim 1 , wherein the input image is a multidimensional image of a first dimension, the first MLM is trained to detect the plurality of reference features based at least in part on processing a plurality of cross-sectional images associated with the multidimensional image, each of the plurality of cross-sectional images having a second dimension and representing a cross-section of the input image, and wherein the second dimension is lower than the first dimension. 3 . The method of claim 1 , wherein the first MLM is trained based on a plurality of training images having reference features of a type common to the type of the reference features within the input image.

4. The method of claim 1, wherein the first MLM comprises a neural network having at least one hidden layer.

5. The method of claim 1, wherein the input image is a medical image of a patient, and the plurality of reference features are associated with one or more bones of the patient depicted by the input image. 6 . The method of claim 5 , wherein a first ROI of the plurality of ROIs includes a representation of at least a portion of a first organ of the patient and does not include a representation of any portion of a second organ of the patient. The method of claim 6 , wherein a second ROI of the plurality of ROIs comprises a representation of at least a portion of the second organ.

8. The method of claim 1 , wherein providing each of the plurality of ROIs to the corresponding second MLM of the plurality of second MLMs comprises: At least one of a location of the corresponding ROI within the input image or a representation of the corresponding ROI is provided.

9. The method of claim 1 , wherein applying the first MLM to the input image comprises: One or more calculations associated with the first MLM are performed on one or more graphics processing units.

10. A method for training a machine learning model, comprising: generating first training data comprising a first training input and a first target output, wherein the first training input comprises an intensity map of a first training image, and the first target output comprises first annotation data identifying a plurality of regions of interest (ROIs) in the first training image; providing the first training data for training a segmentation machine learning model (MLM) on a first set of training inputs including the first training inputs and a first set of target outputs including the first target outputs, wherein the segmentation MLM uses (i) the intensity map to identify a plurality of reference features within the first training image and (ii) the annotation data to identify one or more geometric relationships between each reference feature and one or more corresponding ROIs of the plurality of ROIs; generating second training data comprising a second training input and a second target output, the second training input comprising a second training image representing a first ROI of the plurality of ROIs, and the second target output comprising second annotation data identifying one or more objects within the first ROI, the first ROI being of a first ROI type of the plurality of ROI types; as well as The second training data is provided to train a first object recognition MLM of a plurality of object recognition MLMs on a second set of training inputs including the second training inputs and a second set of target outputs including the second target outputs, wherein the first object recognition MLM is trained to recognize objects in images of the first ROI type.

11. The method according to claim 10, further comprising: generating third training data comprising a third training input and a third target output, the third training input comprising a third training image representing a second ROI of the plurality of ROIs, and the third target output comprising third annotation data identifying one or more objects within the second ROI, the second ROI being of a second ROI type of the plurality of ROI types; as well as The third training data is provided for training a second object recognition MLM of the plurality of object recognition MLMs on a third set of training inputs including the third training inputs and a third set of target outputs including the third target outputs, wherein the second object recognition MLM is trained to recognize objects in images of the second ROI type. The method of claim 10 , wherein the first training image is a medical image of a patient.

13. A system for detecting a target object, comprising: Memory devices; as well as one or more processing devices communicatively coupled to the memory device for: applying an input image to a first machine learning model (MLM) to identify a plurality of regions of interest (ROIs) within the input image, wherein the MLM is trained to identify the plurality of ROIs based at least in part on a plurality of reference features detected within the input image and one or more geometric relationships between each reference feature and one or more locations associated with one of a plurality of ROI types; as well as Each of the identified multiple ROIs is provided as input to a corresponding second MLM among a plurality of second MLMs to obtain an output of the corresponding second MLM, wherein the corresponding second MLM corresponds to one of the multiple ROI types, and the output of the corresponding second MLM indicates one or more objects within the ROI of the corresponding ROI type.

14. The system of claim 13, wherein the input image is a multidimensional image of a first dimension, and the first MLM is trained to detect the plurality of reference features based at least in part on processing a plurality of cross-sectional images associated with the multidimensional image, each of the plurality of cross-sectional images having a second dimension and representing a cross-section of the input image, and wherein the second dimension is lower than the first dimension.

15. The system of claim 13, wherein the first MLM is trained based on a plurality of training images having reference features of a type common to the type of the reference features within the input image.

16. The system of claim 13, wherein the input image is a medical image of a patient, and the plurality of reference features are associated with one or more bones of the patient depicted by the input image. 17 . The system of claim 16 , wherein a first ROI of the plurality of ROIs includes a representation of at least a portion of a first organ of the patient and does not include a representation of any portion of a second organ of the patient.

18. The system of claim 17, wherein the one or more processing devices comprises a graphics processing unit.

19. A non-transitory computer-readable medium having stored thereon instructions for target object detection, wherein the instructions, when executed by a processing device, cause the processing device to: applying an input image to a first machine learning model (MLM) to identify a plurality of regions of interest (ROIs) within the input image, wherein the first MLM is trained to identify the plurality of ROIs based at least in part on a plurality of reference features detected within the input image and one or more geometric relationships between each reference feature and one or more locations associated with one of a plurality of ROI types; and Each of the identified multiple ROIs is provided as input to a corresponding second MLM among a plurality of second MLMs to obtain an output of the corresponding second MLM, wherein the corresponding second MLM corresponds to one of the multiple ROI types, and the output of the corresponding second MLM indicates one or more objects within the ROI of the corresponding ROI type.

20. The computer-readable medium of claim 19, wherein the input image is a multidimensional image of a first dimension, the first MLM being trained to detect the plurality of reference features based at least in part on processing a plurality of cross-sectional images associated with the multidimensional image, each of the plurality of cross-sectional images having a second dimension and representing a cross-section of the input image, and wherein the second dimension is lower than the first dimension.

21. The computer-readable medium of claim 19, wherein the first MLM is trained based on a plurality of training images having reference features of a type common to the type of the reference features within the input image.

22. The computer-readable medium of claim 19, wherein the input image is a medical image of a patient and the plurality of reference features are associated with one or more bones of the patient depicted by the input image.

23. The computer-readable medium of claim 22, wherein a first ROI of the plurality of ROIs includes a representation of at least a portion of a first organ of the patient and does not include a representation of any portion of a second organ of the patient.

Images