Object detection using optical flow with dynamic regions of interest in image stream processing

Abstract

Apparatus, systems, and techniques for efficiently deploying machine learning to detect and classify moving objects in image streams are disclosed. A set of machine learning models with different input sizes can be used to process various regions of interest in multiple image streams in parallel. Both the machine learning models and their inputs can be dynamically selected based on the size of the region of interest.

Description

Technical Field

[0001] At least one embodiment relates to processing resources for performing and facilitating artificial intelligence. For example, according to various new techniques described herein, at least one embodiment relates to the efficient deployment of machine learning for the detection and classification of moving objects in an image stream. Background Technology

[0002] Machine learning is commonly used in image processing pipelines and applications, such as performing the recognition of objects depicted within images. In some applications, object recognition can be performed in real time, particularly in applications involving the identification and monitoring of moving objects. Such applications can be used in industrial safety settings, public safety settings, surveillance systems, traffic monitoring applications, scientific research, autonomous driving systems, robotic automation, and a variety of other settings. Moving object detection may be machine learning-based and may involve training a computational system—using training images and other training data—to identify common features, arrangements, or patterns in an image that may contribute to object recognition. Training can be supervised or unsupervised. Machine learning models can use various computational algorithms, such as decision tree algorithms (or other rule-based algorithms), artificial neural networks, etc. During the inference phase, new images are input into the trained machine learning model, and the patterns and features learned during training can be used to identify various target objects of interest (e.g., vehicles in road images). Attached Figure Description

[0003] Figure 1A This is a block diagram of an example computer system that uses machine learning to perform efficient motion vector-based detection and classification of moving objects, according to at least some embodiments.

[0004] Figure 1B It is an example computing device according to at least one embodiment that can support efficient motion vector-based detection and classification of moving objects;

[0005] Figure 2 An example data stream is shown during the detection and classification of moving objects using a machine learning model with motion vectors and multiple input sizes, according to at least one embodiment;

[0006] Figure 3 The diagram schematically depicts the use of motion vectors to identify a region of interest according to at least one embodiment;

[0007] Figure 4A -C illustrates the selection of a motion vector-based detection model according to at least one embodiment; Figure 4A This illustrates the case where the size of the region of interest is smaller than the minimum input size of the detection model. Figure 4BThis illustrates the size of the region of interest between the input sizes of the two detection models; Figure 4C This illustrates the case where the size of the region of interest is larger than the maximum input size of the detection model.

[0008] Figure 5 This is a flowchart of an example method for effectively detecting and classifying moving objects using a machine learning model with motion vectors and multiple input sizes, according to at least one embodiment;

[0009] Figure 6 This is a flowchart of an example method for training one or more classifier machine learning models to effectively detect and classify moving objects according to at least one embodiment;

[0010] Figure 7A The inference and / or training logic according to at least one embodiment is illustrated;

[0011] Figure 7B The inference and / or training logic according to at least one embodiment is illustrated;

[0012] Figure 8 The training and deployment of a neural network according to at least one embodiment are illustrated;

[0013] Figure 9 This is an example data flow diagram of an advanced computing pipeline according to at least one embodiment;

[0014] Figure 10 This is a system diagram of an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment. Detailed Implementation

[0015] Automatic object recognition is used in many technologies and industries, including but not limited to autonomous driving / parking systems, traffic / parking control and law enforcement, robotics, aerospace technology, and many other applications. In one example, object recognition can be used for traffic monitoring, which involves multiple cameras located at various intersections, roads, parking areas, and building entrances and exits. Each camera can produce a stream of images (e.g., video frames) that may (sometimes) or may not (at other times) capture depictions of stationary or moving objects. The image stream can then be processed using various computer-based object recognition techniques to identify the presence (or absence) of moving objects in the stream. For example, one or more machine learning models (MLMs) can be used to process the various frames of the stream, detecting the presence of vehicles (pedestrians, animals, or any other moving objects) within the camera's field of view, and can also determine the type of object (e.g., pedestrian, passenger car, truck, etc.), the brand of the vehicle (e.g., car manufacturer), the model of the vehicle, etc. A trained MLM can include a neural network of artificial neurons with a certain number of neurons in the input layer, such as n×m. The number of neurons can be correlated with the size of the input (e.g., in pixels) that one or more MLMs can process. For example, a black-and-white frame might have a single value (pixel brightness) associated with each pixel. A color frame might have multiple values ​​associated with each pixel (e.g., one value for each color, such as red, green, and blue). However, the frame size N×M can be different from (e.g., larger than) the MLM input size. Therefore, the preprocessing stage can perform frame rescaling, for example, from N×M pixels to n×m pixels, to match the frame size to the MLM input size. Rescaling can be performed on each new frame and consumes significant computational resources (e.g., processor and memory). Furthermore, when the MLM processes the rescaled frames as a whole, a large portion of these resources is used to process the portions of the frame that do not depict moving objects.

[0016] The aspects and embodiments of this disclosure address these and other technical challenges by disclosing methods and systems that identify reduced portions of a frame depicting a (e.g., moving) object and discard at least a portion of the static background. Since the reduced portion of the frame is selected for processing based on the actual location of the object, computational complexity is significantly reduced. Furthermore, multiple MLMs with different input sizes can be deployed to process the selected portion. This reduces the processing involved in rescaling, and for many frames, the need for rescaling can be completely eliminated.

[0017] More specifically, a given frame can be compared to a different frame (referred to herein as a reference frame), which may be a prior frame acquired by the same camera. A set of motion vectors can be determined for various pixels or pixel blocks within the given frame. These motion vectors characterize the movement of a specific pixel (or pixel block) between the reference frame and the given frame. For example, a pixel block depicting a portion of a vehicle's hood might move from position A in the reference frame to position B in the given frame over a specific time interval. The pixel clusters whose motion vectors are determined to be above a certain minimum level (e.g., above a noise level) can then be identified along with the region surrounding all these clusters. The size of this region can then be compared to the available detector MLM input size. If the region size is smaller than the minimum detector MLM input size, the region can be expanded (without rescaling) so that the expanded region's size matches the minimum detector MLM input size. The expanded region is then input into the corresponding detector MLM. If the region size is between the input sizes of two detector MLMs, the region can be similarly expanded to a larger input size. If the region size is larger than the maximum detector MLM input size, the region can be shrunk to the maximum detector MLM input size. The above process minimizes the need for rescaling. In some embodiments, the region may undergo additional (up or down) scaling to a larger or smaller input size, as described in more detail below.

[0018] As described, for each new frame, the selection of a region of interest (ROI) is performed dynamically and individually, along with the selection of the optimal detector MLM. When the selected ROI is processed by the selected detector MLM, it segments the region into separate sub-regions surrounding the different moving objects (e.g., different vehicles) detected within that region. The outputs of multiple detector MLMs can then be combined for concurrent processing by one or more classifier MLMs trained to determine or identify the type / brand / model / etc. of the detected objects. As a result, frames from different video streams are processed together, thereby improving throughput.

[0019] The advantages of the disclosed technology include, but are not limited to, improved power and processing efficiency due to the elimination or reduction of rescaling operations, centralized processing of the region of actual interest, and concurrent classification of data from different sources (e.g., video cameras). The freed-up processing resources can be used to deploy more advanced MLMs (e.g., MLMs with more processing layers) on the same computing platform, enabling more complete object detection and more accurate classification of detected objects.

[0020] System Architecture

[0021] Figure 1AThis is a block diagram of an example computer system 100 according to at least some embodiments, which uses machine learning to perform efficient detection and classification of moving objects based on motion vectors. Figure 1A As depicted, the computing system 100 may include a computing device 102, an image storage library 150 (shown as "Image Library 150" in the figure), and a training server 160 connected to a network 140. The network 140 may be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wireless network, a personal area network (PAN), or a combination thereof.

[0022] Computing device 102 may be a desktop computer, laptop computer, smartphone, tablet computer, server, or any suitable computing device capable of performing the techniques described herein. Computing device 102 may be configured to receive an image (frame) 101, which may be part of an image stream, such as a video feed generated by one or more cameras connected to computing device 102 via any suitable wired or wireless connection. In some embodiments, the video feed may be generated in real time. In some embodiments, the video feed may be generated, stored, and then retrieved at some point in the past for processing as described in this disclosure. For example, a device capable of generating a video feed (including image 101) may be a surveillance camera, video recorder, photographic equipment, scanner, autonomous vehicle sensing device (e.g., LiDAR, radar, long-range and mid-range cameras), etc. Image 101 may be an indoor image, an outdoor image, an image of a car environment, an image of a person or a group of people, an image of a natural or man-made (e.g., outdoor or indoor) landscape, or any other type of image. Image 101 can be in any digital (e.g., pixel-based or vector-based) format, including but not limited to JPEG, GIF, PNG, BMP, TIFF, CIB, DIMAP, NITF, etc. Image 101 can be stored (e.g., along with other images) in image storage 150. Additionally, according to some embodiments disclosed herein, image storage 150 can store training images 151 for training one or more MLM object detection and classification systems. Image storage 150 can be accessed directly by computing device 102 or (as shown) via network 140.

[0023] Image storage 150 may be a persistent storage device capable of storing images and their metadata. Image storage 150 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage-based disks, tape or hard disk drives, NAS, SAN, etc. Although described as separate from computing device 102, in at least one embodiment, image storage 150 may be part of computing device 102. In at least some embodiments, image storage 150 may be a network-attached file server, while in other embodiments, image storage 150 may be some other type of persistent storage, such as an object-oriented database, relational database, etc., which may be hosted by a server machine or one or more different machines coupled to computing device 102 via network 140.

[0024] Computing device 102 may include memory 104 communicatively coupled to one or more processing devices, such as one or more graphics processing units (GPUs) 110 and one or more central processing units (CPUs) 130. Memory 104 may store one or more additional processing modules and MLMs, such as motion vector-based segmentation 120, object detection model (ODM) 124, and object classification model (OCM) 126. Computing device 102 may host inference engine 122 to run motion vector-based segmentation 120, ODM 124, and OCM 126. Inference engine 122 may be executed by GPU 110 and / or CPU 130. Computing device 102 may use image 101 (or training image 151) as input to MV-based segmentation 120, which is configured to identify motion vectors in image 101 based on a comparison of image 101 with a reference image, as described in more detail below. Using motion vectors, computing device 102 can identify portions of image 101 depicting one or more moving objects and can select a suitable trained ODM 124 based on the size of the identified portions. Inference engine 122 can then apply the selected ODM 124 to portions of image 101 to identify depictions of individual moving objects in image 101. Multiple ODMs 124 can operate in parallel, with each ODM 124 processed on a selected specific image (e.g., in an image pipeline). The outputs of the various ODMs 124 can be combined into a single input to one or more OCMs 126, which can determine the type and other characteristics of the objects, such as brand, model, year, color, etc. In at least one embodiment, one or more of the ODMs 124 and / or OCMs 126 can be models including neural networks.

[0025] ODM 124 and / or OCM 126 can be trained by training server 160. In at least one embodiment, training server 160 may be part of computing device 102. In other embodiments, training server 160 may be directly or communicatively coupled to computing device 102 via network 140. Training server 160 may be (and / or include) rack server, router computer, personal computer, laptop computer, tablet computer, desktop computer, media center, or any combination thereof. Training server 160 may include training engine 162. In at least one embodiment, training engine 162 may generate one or more machine learning models (e.g., ODM 124 and / or OCM 126). The generated MLMs may be trained by training engine 162 using training data that may include training input 165 and corresponding target output 167. In at least one embodiment, different MLMs may be trained separately for different types of images. For example, an OCM for determining truck brands may be trained separately from an OCM for determining bus brands.

[0026] For training ODM 124, training input 165 may include one or more training images 151 and various metadata of the training images 151. Training images 151 may be large-scale images, such as panoramic images of a car environment, and may include various different types of objects, such as cars, trucks, buses, motorcycles, bicycles, pedestrians, etc. Region of Interest (ROI) annotation engine 163 can annotate the large-scale images with indications of various ROIs (e.g., bounding boxes or other boundary shapes, such as ellipses, cylinders, etc.), including individual objects (e.g., bounding boxes of cars, bounding boxes of buses, etc.). ROI annotation engine 163 can generate a target output 167, which may be the training input 165 with ROI annotations. Furthermore, training engine 162 can generate mapping data 166 (e.g., metadata) that associates training input 165 with the correct target output 167. During training of ODM 124, training engine 162 can identify patterns in training input 165 based on the desired target output 167 and train ODM 124 to perform ROI segmentation using the identified patterns. The predictive utility of the identified patterns can then be validated using additional training input / target output associations, and then used by ODM 124 in the inference phase for future processing of new images. For example, upon receiving a new image of an intersection, the trained ODM 124 can identify various ROIs, including a variety of vehicles, pedestrians, and other objects located at or near the intersection.

[0027] Similarly, to train one or more OCMs 126, the class annotation engine 164 can annotate the images of individual ROIs and generate corresponding categories for the ROIs using indications (e.g., pointers, bounding boxes, etc.) of various target objects (cars, trucks, buses, pedestrians, etc.) within the ROI. A category should be understood as any feature of an object that can be determined based on its appearance, such as the type of vehicle (car, truck, etc.) or animal (e.g., bird, cat, dog, etc.), the vehicle's manufacturer (or maker), the vehicle's model / year / color, the animal's breed, etc. In at least one embodiment, training input 165 can be generated from a large-scale image set by acquiring subsets corresponding to different ROIs. The training engine 162 can also generate mapping data 166 that associates the class training inputs with the correct class target output 167. During the training of one or more OCMs 126, patterns in the class training input 165 can be identified based on the desired class target output 167 and used to perform object classification within the corresponding ROI using the identified class-level patterns. The identified patterns can then be validated using additional training (test) input / target output associations. Based on the recognition / verification pattern, the trained OCM 126 then processes new images during the inference phase. For example, upon receiving a new image 101, recognizing a portion of an object depicted in image 101, and performing ROI detection using ODM 124, the trained OCM 126 can be deployed to classify various objects within image 101.

[0028] In at least one embodiment, each or some of ODM 124 and / or OCM 126 may be implemented as a deep learning neural network with multiple levels of linear or nonlinear operations. For example, each or some of ODM 124 and / or OCM 126 may be a convolutional neural network, a recurrent neural network (RNN), a fully connected neural network, etc. In at least one embodiment, each or some of ODM 124 and / or OCM 126 may include multiple neurons, each receiving its input from other neurons or from an external source and producing an output by applying an activation function to the sum of (trainable) weighted input and bias values. In at least one embodiment, each or some of ODM 124 and / or OCM 126 may include multiple neurons arranged in layers, including an input layer, one or more hidden layers, and an output layer. Neurons from adjacent layers may be connected by weighted edges. Initially, some initial (e.g., random) values ​​may be assigned to the edge weights. For each training input 165, training engine 162 may cause each or some of ODM 124 and / or OCM 126 to produce an output. The training engine 137 can then compare the observed output with the desired target output 167. The resulting error or mismatch, such as the difference between the desired target output 167 and the actual output of the neural network, can be backpropagated through the respective neural networks, and the weights in the neural networks can be adjusted to make the actual output closer to the target output. This adjustment can be repeated until the output error for a given training input 165 meets a predetermined condition (e.g., below a predetermined value). Subsequently, different training inputs 165 can be selected, new outputs generated, and a series of new adjustments implemented until the individual neural networks are trained to acceptable accuracy.

[0029] Figure 1BAn example computing device 102, according to at least one embodiment, can support efficient detection and classification of moving objects based on motion vectors. In at least one embodiment, MV-based segmentation 120 and inference engine 122 can be executed by one or more GPUs 110 and object detection and classification can be performed using ODM 124 and / or OCM 126. In at least one embodiment, GPU 110 includes multiple cores 111, each core capable of executing multiple threads 112. Each core can run multiple threads 112 concurrently (e.g., in parallel). In at least one embodiment, thread 112 can access register 113. Register 113 can be thread-specific registers, which can access registers limited to the corresponding thread. Furthermore, shared register 114 can be accessed by all threads of the core. In at least one embodiment, each core 111 can include a scheduler 115 to distribute computational tasks and processes among different threads 112 of the core 111. Dispatch unit 116 can perform scheduled tasks on appropriate threads using the correct private registers 113 and shared registers 114. The computing device 102 may include input / output components 134 to facilitate information exchange with one or more users or developers.

[0030] In at least one embodiment, GPU 110 may have a (high-speed) cache 118, to which multiple cores 111 may share access. Furthermore, computing device 102 may include GPU memory 119, where GPU 110 may store intermediate and / or final results (outputs) of various computations performed by GPU 110. After completing a specific task, GPU 110 (or CPU 130) may move the output to (main) memory 104. In at least one embodiment, CPU 130 may execute processes involving serial computation tasks (assigned by inference engine 122), while GPU 110 may execute tasks suitable for parallel processing (e.g., multiplying the input of a neural node by weights and adding bias). In at least one embodiment, inference engine 122 may determine which processes should be executed on GPU 110 and which should be executed on CPU 130. In other embodiments, CPU 130 may determine which processes should be executed on GPU 110 and which should be executed on CPU 130.

[0031] Figure 2 An example data stream 200 is illustrated during the detection and classification of a moving object using a machine learning model with motion vectors and multiple input sizes, according to at least one embodiment. In at least one embodiment, the data stream 200 can be... Figure 1A and Figure 1B This is implemented using the inference engine 122 of the computing device 102. For example... Figure 2As shown, the data can originate from multiple sources 201-k (depicting n sources), which can be any video or camera, storage of previously acquired video / photographic images, synthetic video sources (e.g., video game applications), or any other suitable source of imaging data. Each source 201-k can generate an image stream 202-k (for simplicity, only one image from each stream is depicted). Images 202-k can have different vantage points, fields of view, resolutions, frame rates, aspect ratios, etc. For example, source 201-1 can generate an image 202-1 of size 720x486 pixels, source 201-2 can generate an image 202-2 of size 1920x1080 pixels, source 201-3 can generate an image 202-2 of size 1280x720 pixels, and so on. In at least one embodiment, the image 202-k may undergo decoding / preprocessing 204, which may include trimming, image sharpening, blur / noise / artifact removal, compression, resampling, normalization, upsampling, or any combination of these operations.

[0032] The image 202-k output by the decoding / preprocessing 204 can be input into the motion vector (MV) determination module 206. The MV determination module 206 can identify how the pixels of a given image are shifted relative to a reference image. Figure 3 A schematic depiction illustrates the use of motion vectors to identify regions of interest in motion according to at least one embodiment. Reference image 302 is used to identify pixel displacements in image 304. Image 304 may be captured at time t, while reference image 302 may be captured at a previous time t-τ (although in some embodiments, reference image 302 may be captured at a later time t+τ). MV determination module 206 can identify pixels or pixel blocks in image 304, such as block 306, and can further identify pixels or pixel blocks in reference image 302, such as block 308, which image the same region of a moving object (e.g., a vehicle). For example, MV determination module 206 can identify the pixel intensity (or multiple pixel intensities, if the image is a color image) I(x, y) of block 306 in image 304 and the pixel intensity I of block 308 in reference image 302. REF If (x-Δx, y-Δy) are the same (within a certain tolerance), then I(x, y) ≈ I REF (x-Δx, y-Δy), so it is possible to depict the same object or a part of the same object.

[0033] In various situations, pixel intensities in two images may not be exactly the same because the brightness (and color) of pixels can change as a moving object changes its position relative to the camera. As a result, various error evaluation procedures can be deployed. For example, for each pixel x in a block assumed to belong to a moving object... i ,yk This can form a difference δI(x) i ,y k )=I(x i ,y)-I REF (x i -Δx,y k -Δy). And the quadratic cost function (or any other suitable loss function) can be calculated, for example, Cost = ∑ j ∑ k [δI(x i ,y k )] 2 If the cost function is within a margin (e.g., empirically determined) mar relative to the total (or average) pixel intensity of the block, then I = ∑ j ∑ k I(x i ,y k For example, such that Cost / I ≤ mar, block 306 in image 304 and block 308 in reference image 302 can be identified as depictions of the same object. The values ​​Δx and Δy, representing the displacement of the block between the two images, represent the motion vector 310 of block 306 (in... Figure 3 The amplified components, for example, ΔR = (Δx, Δy). The determination of the motion vector ΔR and the size of block 306 can itself be performed as part of the optimization process, for example, by minimizing the cost function of various blocks described as the same object. In some embodiments, the cost function can be a binary cross-entropy cost function, a mean squared error cost function, a weighted mean squared error cost function, a mean absolute error cost function, a Huber cost function, a Hinge cost function, a multi-class cross-entropy cost function, a Kullback-Liebler cost function, etc.

[0034] In some embodiments, to reduce the amount of processing performed in identifying motion vectors, the MV determination module 206 may limit candidate blocks 308 in the reference image 302 to a certain vicinity of block 306 in the image 304. The size of the vicinity may be determined by the time difference τ between the reference image 302 and the image 304 and the maximum expected velocity v of the moving object in the specific environment. max To determine, |ΔR|=v max For example, the maximum speed in a parking lot might be 20 mph, while the maximum speed at a city intersection might be 40-50 mph, and so on. Blocks 306 and 308 can have any suitable pixel size, such as 32x32 pixels, 8x16 pixels, 4x4 pixels, 2x3 pixels, or in some cases, as small as a single pixel.

[0035] Continue to refer to Figure 2A set of motion vectors identified for a given image (e.g., image 202-1) can be used by segmentation module 208 to select detection model 210-k for image processing. Detection model 210-k can be... Figure 1A ODM 126. Figure 4A -C illustrates the selection of a motion vector-based detection model according to at least one embodiment. The segmentation module 208 can access the set of motion vectors {ΔR} and select pixels (or pixel blocks) associated with motion vectors ΔR that are greater than the value corresponding to noise. Such pixels (or pixel blocks) are assumed to belong to the depiction of a moving object. The segmentation module 208 can then identify regions of interest 402 (in) that surround some or all of such pixels (or pixel blocks). Figure 4A (Depicted with dashed lines in the middle). In some embodiments, the region of interest can be the smallest bounding box of such pixels, for example, whose lower left corner has coordinates (x, y). min ,x min And the upper right corner has coordinates (x) max ,y max The bounding box of x, where x min (x max ) is the minimum (maximum) x-coordinate of the pixel associated with the description of any moving object, and y min (y max Similarly, this refers to the minimum (maximum) y-coordinate of such pixels. Although for ease of viewing, region of interest 402 is... Figure 4A (and Figure 4B -C) is depicted as occupying the lower left corner of image 401, but region of interest 402 can be located anywhere within image 401.

[0036] Multiple available detector models 210-k (e.g., ODM 126) can have different input sizes, m k ×n k The selection of detector mode 210-k can be achieved by comparing various available input sizes m. k ×n k To perform this operation, use the size of the region of interest 402, for example, (x max -x min )×(y max -y min ). Figure 4A This illustrates a case where the size of the region of interest 402 is smaller than the minimum detection model input size, for example, smaller than the first model input size 404, m1×n1. In this case, the size of the region of interest can be expanded to m1×n1, for example, by adding pixels depicting non-moving objects to one or more sides of the region of interest 402, such as... Figure 4AAs shown by the solid arrow in the image. In some embodiments, the added pixels may be pixels from the same image 202-1 or composite pixels, such as filler pixels, with some fixed intensity. Figure 4B The diagram illustrates the size of the region of interest 402 between two detection model input sizes, for example, between a first model input size of 404m1×n1 and a second model input size of 406m2×n2. In this case, the size of the region of interest can be increased to m2×n2, for example, in conjunction with the above. Figure 1B Described in the same way. Figure 4C The diagram illustrates a case where the size of the region of interest 402 is larger than the maximum detection model input size, for example, larger than the second model input size 406, m²×n² (if two detector models 2¹⁰-k are deployed). In this case, the region of interest 402 can be reduced to the maximum model input size, such as... Figure 4C As shown by the dashed arrow in the image.

[0037] Figure 4A -C is intended as an illustration of an embodiment where the region of interest is expanded to the next lowest detector model input size to minimize the need for rescaling. In some embodiments, a set of threshold sizes may be defined between available model input sizes. For example, a threshold size m may be defined between a first model input size m1×n1 and a second model input size m2×n2. T ×n T This allows the region of interest 402 to be between m1×n1 and m T ×n T Between these, the region of interest is reduced to m1×n1 (e.g., Figure 4B (as shown by the dashed arrow in the image), and if the size of the region of interest 402 is in m T ×n T If the region of interest is between m²×n² and m²×n², then the region of interest is expanded (or filled) to m²×n². The threshold size m can be chosen based on overall computational efficiency. T ×n T For example, in regions of interest smaller than m... T ×n T In the case of [the region of interest] 402, shrinking it to m1×n1 and processing the shrunken image with a smaller input size model, instead of expanding the region and processing the expanded image with a larger input size model (which may result in longer processing time), is likely to be more computationally efficient. For sizes greater than m... T ×n T For the region of interest 402, expanding rather than shrinking might be more efficient. Furthermore, a larger input size might result in a more accurate model. Therefore, the threshold size m... T ×n TThe determination can take all such metrics into account and can be performed based on application-specific standards (including user-specific standards).

[0038] It can identify any number of thresholds m T ×n T In some embodiments, for a system with N detector models of different input sizes, the number of different thresholds can be N-1. In some cases, the horizontal m (or vertical n) size of the region of interest 402 may be larger than the horizontal m of a particular detector model 210-k. k (or vertical n) k The input size is smaller than the vertical n (or horizontal m) of the region of interest 402, but the vertical n of the same detector model 210-k is smaller. k (or horizontal m) k The size of the region of interest (ROI) 402 can be adjusted. There are several ways to handle this situation. For example, if one dimension of the ROI 402 is even higher than the corresponding dimension of the detector model 210-k, the size of the ROI can be extended to the next largest input size model, such as detector model 210-(k+1). In some cases, the ROI 402 can be extended (or padded) along smaller dimensions, for example, Make the new size With detector model 210-k(m k :n k The input size has the same aspect ratio. The expanded (filled) region of interest can then be scaled down to the input size of detector model 210-k. In some embodiments, region of interest 402 can be scaled down (or scaled up) differently along different dimensions (so that resizing or padding is not required). Because this anisotropic rescaling alters the relative horizontal and / or vertical size of various objects, this technique can be used when detector model 210-k has been trained on a variety of images, including those depicting objects whose size has been modified compared to their natural size (e.g., stretched or squeezed).

[0039] Return to reference Figure 2The selected (and in some cases rescaled) regions of interest (ROIs) can be guided by the segmentation module 208 for processing by one of the detector models 210-k. For example, entries (ROIs) in a corresponding batch can be assigned to each detector model 210-k for processing, and these entries (ROIs) are fed into the model. More specifically, new entries can be placed at the back end of the corresponding batch, where entries retrieved from the batch are processed by the detector model 210-k in the order of their placement history. In some embodiments, entries in a batch can include actual pixel values ​​stored in an input buffer. In some embodiments, entries in a batch can be metadata identifying that the region of interest 402 is the corresponding complete image. For example, metadata can include an identifier of the image, coordinates of the bounding box of the region of interest 402, the time the image was captured, and so on. In some embodiments, processing of entries in a batch can be performed in a pipeline manner. More specifically, if the detector model 210-k is a neural network model, the first layer of neurons in the detector model 210-k can perform computation of the first entry in the batch during a first computation cycle (or a set of cycles), and may perform computation of the second entry during a second computation cycle, and so on. Meanwhile, during the second computation cycle, the second layer of neurons may be processing the first entry, and so on. In some embodiments, the processing of each detector model 210-k may be performed by one or more GPUs by separate processing threads (e.g., in parallel).

[0040] Each of detector models 210-1…210-N can output an indication of the detected object 220 in the corresponding entry. This indication can have any suitable form, such as a coordinate set, bounding box, etc. The bounding box can be rectangular, elliptical, or can have any other suitable form. For example, if object A partially occludes object B, the indication for object B can include the bounding box surrounding object B, where a portion of the bounding box is removed (e.g., the portion occluded by object A). In some embodiments, the indication can include any number of connected (or otherwise associated) boxes or pixel regions, pixel-by-pixel lists, etc. There can be any number of detected objects 220 within a specific entry processed by each detector model 210-k. In some embodiments, detector model 210-k can identify sub-regions within region of interest 402 corresponding to various moving objects. In some embodiments, detector model 210-k can identify any additional information about the object, such as the object's high-level type / category, such as a vehicle or a person.

[0041] The indication of detected object 220 can be received by combiner (tracker) 230. Combiner 230 can create a data structure that includes data (e.g., pixel data) for each sub-region corresponding to objects detected by all or at least some of the detector models 210-1…210-N. More specifically, the data structure created by combiner 230 during the j-th calculation cycle (or a set of cycles) can include the detected object 220, whose identification was output by detector models 210-1…210-N during the previous (e.g., the j-1th) calculation cycle. For example, the data structure can include concatenations of pixel values ​​for various sub-regions corresponding to the detected object 220. Additionally, the data structure can include various metadata that identifies each sub-region as generated by the corresponding source 201-k, the location of the sub-region within image 202-k, the time of image 202 capture, physical location, resolution, field of view of the corresponding source 201-k, etc. At least some metadata can be generated by segmentation module 208 (e.g., Figure 2 (As shown by the dashed line in the image) is provided based on the complete image 202-k.

[0042] A data structure can be provided to one or more classifier models 240-k. Classifier models 240-k can include any number of models trained to independently (e.g., in parallel) or sequentially determine various attributes of the detected object 220. For example, classifier model 240-1 can determine the type of the object (e.g., SUV, pickup truck, passenger car, van, bus, semi-rail, non-vehicle object, etc.) and classifier model 240-2 can determine the brand and model of the vehicle. In some embodiments, such as when different classifier models operate in parallel, some classifier models (e.g., models 240-1 and 240-2, as shown in the dashed box) can receive and process the same data structure provided by combiner 230. In some embodiments, such as when different classifier models operate sequentially, one classifier model (e.g., model 240-1) can output a set of features (e.g., feature vectors) representing the object's appearance, and another classifier model (e.g., model 240-2) may be processing this set of features to determine the final classification. Although in Figure 2The diagram depicts two classifier models 240-k, the number of which is not limited and can be one, three, or any other number of models. Classifier models 240-k generate inference output 250, which may include some of all categories determined by detector model 210-k and classifier model 240-k. Inference output 250 may be provided to users, applications (e.g., surveillance applications, traffic control applications, etc.), stored in local storage or the cloud, etc. Inference output 250 may be indexed or provided with information from combiner 230 to reference the location of classified objects, the identification of specific images 202-k of captured classified objects (and any reference images), the time the images were captured, etc. In some embodiments, one or more categories (e.g., high-level object types) may be determined by detector model 210-k. In some embodiments, all categories may be obtained through detector model 210-k.

[0043] In some embodiments, combiner 230 may manage the computation pipeline of classifier models 240-k. For example, under certain conditions, the number of objects in most images may be small (e.g., off-peak traffic), so most regions of interest are relatively narrow and thus directed to detector models with low input sizes (e.g., detector model 210-1). Under other conditions (e.g., peak traffic), the number of objects may be large, most regions of interest are wide, and they are accordingly directed to detector models with high input sizes (e.g., detector models 210-N) for processing. As a result, combiner 230 may receive more objects detected by some detector models than by others. In this case, combiner 230 may combine objects detected by different detector models only for a portion of all computation cycles. During other cycles, combiner 230 may be clearing pipeline backlogs created by one (or more) detector models. For example, combiner 230 may access an overflow buffer storing unprocessed objects previously detected by these models and use P previous outputs of those models to generate data structures. Before restoring the combined processing of different detector models 210-k, the combiner 230 may repeat the process as needed to clear the backlog.

[0044] Figure 5 and Figure 6This is a flowchart of corresponding example methods 500 and 600, according to some embodiments of this disclosure, that facilitate the detection and classification of moving objects in an image stream. Methods 500 and 600 may be executed by one or more processing units (e.g., CPU and / or GPU), which may include (or communicate with) one or more memory devices. In at least one embodiment, methods 500 and 600 may be executed by a processing unit of computing device 102 or training server 160. In at least one embodiment, methods 500 and 600 may be executed by multiple processing threads (e.g., CPU threads and / or GPU threads), each thread performing the operation of one or more individual functions, routines, subroutines, or methods. In at least one embodiment, the processing threads implementing method 500 (and similarly, method 600) may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 500 (and similarly, method 600) may execute asynchronously relative to each other. Figure 5 and 6 Compared to the order shown, the various operations of methods 500 and 600 can be executed in different orders. Some operations of the methods can be executed concurrently with other operations. In at least one embodiment, Figure 5 and 6 One or more of the operations shown may not always be performed.

[0045] Figure 5 This is a flowchart of an example method 500 for effectively detecting and classifying moving objects using a machine learning model with motion vectors and multiple input sizes, according to at least one embodiment. Method 500 can be performed in traffic control and monitoring, medical imaging, autonomous driving systems, industrial environments, research environments, and many other environments. In block 510, a processing unit performing method 500 can identify a first image and a first reference image. Furthermore, the processing unit can identify a second (third, etc.) image and a second (third, etc.) reference image. The terms first, second, etc., should be understood as merely identifiers and do not presuppose any temporal or contextual order. The first (second, etc.) image and the first (second, etc.) reference image can be portions (e.g., frames) of corresponding image streams, such as a video frame stream or photographic images captured at specific time intervals, e.g., periodic time intervals. More specifically, the first image and the first reference image can be obtained by a first camera, while the second (third, etc.) image and the second (third, etc.) reference image can be obtained by a second (third, etc.) camera. The field of view of the first camera can be different from the field of view of the second (third, etc.) camera.

[0046] In block 520, the processing unit may determine a first (second, etc.) plurality of motion vectors of a first (second, etc.) image. Each of the first (second, etc.) plurality of motion vectors may characterize a displacement of the first (second, etc.) image relative to one or more pixels of a first (second, etc.) reference image. For example, the first (second, etc.) image may be segmented into a grid of pixel blocks, and for at least some pixel blocks, similar reference blocks may be located in the first (second, etc.) reference image. Some pixel blocks may be located in the first image at the same (or substantially the same) locations as similar blocks in the first (second, etc.) reference image. Such blocks depict stationary objects (or portions of such objects) and may be associated with zero motion vectors. Depictions of moving objects may be associated with non-zero motion vectors, each motion vector characterizing the displacement of a pixel or pixel block. To identify such motion vectors, in some embodiments, a grid of multiple block sizes may be used, for example, a 16x16 pixel block grid and a 4x4 pixel block grid. In some embodiments, an initial coarse grid (e.g., a grid of 32x32 pixel blocks, 64x64 pixel blocks, etc.) may be deployed first to eliminate regions in a first (second, etc.) image that depict stationary objects and include pixel blocks identical to the corresponding pixel blocks in the first (second, etc.) reference image. Once a mismatch is detected in one or more pixel blocks of the initial grid, the corresponding block may be further subdivided into finer pixel blocks, e.g., 8x8 pixel blocks, 4x4 pixel blocks, etc. The finer pixel blocks may then be moved along various directions (e.g., 8, 12, 16, etc.) by a number of pixels (e.g., integer or fractional pixels) until a best match with the corresponding block in the reference image is found.

[0047] In box 530, method 500 can continue, whereby the processing unit uses first (second, etc.) multiple motion vectors to identify a first (second, etc.) region of interest in the first (second, etc.) image, the first (second, etc.) region of interest depicting one or more objects in the first (second, etc.) image. For example, the first (second, etc.) region of interest could be... Figure 4A -C Region of Interest 402. In some embodiments, the first (second, etc.) region of interest may include pixels or pixel blocks of the first (second, etc.) image associated with non-zero motion vectors and exclude pixels or pixel blocks associated with zero motion vectors, thereby depicting a stationary object. In block 540, the processing unit may use the size of the first (second, etc.) region of interest to select a first (second, etc.) MLM from a plurality of MLMs. Each of the plurality of MLMs may have a different input size. This can be combined as follows Figure 4A -C performs the MLM selection as described.

[0048] In box 550, the processing unit may further select a first (second-class) portion of the first (second-class) image based on the size of the first (second-class) region of interest and the input size corresponding to the first MLM, wherein the first (second-class) portion includes the first (second-class) region of interest. For example, the first (second-class) portion may include region of interest 402 and additional segments of the first (second-class) image such that the size of the first (second-class) portion matches the size of one or more MLMs (e.g., detector model 210-k), which may be an MLM with a minimum input size that exceeds (or matches) the size of the region of interest.

[0049] At box 560, method 500 may continue, whereby the processing unit uses the processing output of the first (second class) MLM based on the first (second class) portion to detect the presence of one or more objects in the first (second class) image. In some embodiments, box 560 may further include classifying one or more detected objects. In some embodiments, classifying moving objects may include multiple operations, such as Figure 6 As shown in the labeled section. More specifically, in selectable box 562, before the first (second) portion is processed by the first (second) MLM, the processing unit can rescale the first (second) portion of the first (second) image to match the input size corresponding to the first (second) MLM. In particular, the first (second) portion of the first (second) image can be selected to have a size different from the input size corresponding to the first (second) MLM. For example, as Figure 4C As shown, if the size of the region of interest 402 exceeds the input size corresponding to the selected MLM, the region of interest 402 can be reduced to that input size.

[0050] At box 564, method 500 may include processing a first (second-class) selected portion of a first (second-class) image by a first (second-class) MLM. In some embodiments, the output of the processing of the first (second-class) MLM on the first (second-class) portion may include bounding boxes for each of one or more objects in the first (second-class) image. In some embodiments, classifying objects in the first image is performed concurrently with classifying one or more objects in the second (third-class) image. More specifically, as shown at box 566, classifying one or more objects may include processing a combined input by one or more classifier MLMs. The combined input may include the output of the processing of the first portion by the first MLM and the output of the processing of the second portion by the second (third-class) MLM.

[0051] In some embodiments, one or more classifier MLMs are trained to determine the type of one or more objects represented in the combined input. The output of one or more classifier MLMs may include one or more inferences corresponding to i) vehicle type, ii) vehicle brand, or iii) vehicle model. Such characteristics may be determined for at least one of at least one of the objects in at least one of the first image or the second (third, etc.) image. In some embodiments, the combined input to one or more classifier MLMs may be one of multiple combined inputs to the input pipeline. In such embodiments, one or more classifier MLMs may be configured to perform pipeline processing of the multiple combined inputs. In some embodiments, any or all processing operations, including processing of the first (second, etc.) portion of the combined input by the first (second, etc.) MLM, may be performed using one or more GPUs.

[0052] Figure 6 This is a flowchart of an example method 600 for training one or more classifier machine learning models to efficiently detect and classify objects according to at least one embodiment. At box 610, the processing unit performing method 600 may obtain multiple training images, each depicting one or more objects (e.g., objects in motion). At box 620, method 600 may proceed to crop one or more portions from each of the multiple images, each portion including a depiction of an object. At box 630, method 600 may continue with the processing unit creating a combined data structure including data from each cropped portion. The combined data structure may also include various metadata associated with the cropped portion, which may include the time the image was captured, the location of the camera that generated the image, the camera resolution, etc. At box 640, method 600 may include processing the combined data structure using one or more classifier models to predict characteristics of at least some of the objects depicted in the multiple images. At box 650, method 600 may include adjusting the parameters of one or more classifier models based on the difference between predicted features of the objects and target features of the objects.

[0053] Reasoning and training logic

[0054] Figure 7A Inference and / or training logic 715 is shown for performing inference and / or training operations associated with one or more embodiments.

[0055] In at least one embodiment, inference and / or training logic 715 may include, but is not limited to, code and / or data storage 701 for storing forward and / or output weights and / or input / output data, and / or other parameters configuring neurons or layers of a neural network trained for and / or used for inference in one or more embodiments. In at least one embodiment, training logic 715 may include or be coupled to code and / or data storage 701 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic, including integer and / or floating-point units (collectively referred to as arithmetic logic units (ALUs) or simple circuits). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, code and / or data storage 701 stores weight parameters and / or input / output data of each layer of a neural network trained or used in one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inference using one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 701 may be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0056] In at least one embodiment, any portion of the code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 701 may be a cache memory, dynamic random-addressable memory (“DRAM”), static random-addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 701 is internal or external to the processor, for example, or composed of DRAM, SRAM, flash memory, or some other storage type, may depend on the available on-chip or off-chip storage space, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0057] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 705 for storing backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, during training and / or inference using one or more embodiments, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of a neural network trained or used in one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 705 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).

[0058] In at least one embodiment, code (such as graph code) causes the architecture of the neural network corresponding to that code to load weights or other parameter information into the processor ALU. In at least one embodiment, any portion of the code and / or data storage 705 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice between the code and / or data storage 705 being internal or external to the processor, for example, whether it consists of DRAM, SRAM, flash memory, or some other type of storage, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the data batch size used in the inference and / or training of the neural network, or some combination of these factors.

[0059] In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be separate storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be combined storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data storage 701 and code and / or data storage 705 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0060] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 710 (including integer and / or floating-point units) for performing logical and / or mathematical operations at least in part based on or instructed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from layers or neurons within a neural network) stored in activation storage 720, which are functions of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, activation is activated in response to execution instructions or other code, linear algebraic and / or matrix-based mathematical generation performed by ALU 710, and the activation is stored in activation storage 720. The weight values ​​stored in code and / or data storage 705 and / or data storage 701 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters. Any or all of these can be stored in code and / or data storage 705 or code and / or data storage 701 or another on-chip or off-chip memory.

[0061] In at least one embodiment, one or more processors or other hardware logic devices or circuits include one or more ALUs 710, while in another embodiment, one or more ALUs 710 may be located outside the processor or other hardware logic device or the circuitry that uses them (e.g., a coprocessor). In at least one embodiment, one or more ALUs 710 may be included within an execution unit of a processor, or otherwise included in a group of ALUs accessible by the execution unit of the processor, which may be within the same processor or distributed among different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may share a processor or other hardware logic device or circuitry, while in another embodiment, they may be located in different processors or other hardware logic devices or circuitry, or in some combination of the same and different processors or other hardware logic devices or circuitry. In at least one embodiment, any portion of activation storage 720 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be retrieved and / or processed using the processor’s fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0062] In at least one embodiment, the active memory 720 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 720 may be wholly or partially located inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory 720 is internal to or external to the processor may depend on the available on-chip or off-chip storage, the latency requirements for training and / or inference functions, the batch size of data used in inference and / or training the neural network, or some combination of these factors. For example, it may include DRAM, SRAM, flash memory, or some other memory type.

[0063] In at least one embodiment, Figure 7A The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7AThe inference and / or training logic 715 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware or other hardware (such as field programmable gate array (“FPGA”)).

[0064] Figure 7B Inference and / or training logic 715 according to at least one embodiment is illustrated. In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, hardware logic, wherein computational resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., field-programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 7B In at least one embodiment shown, each of code and / or data storage 701 and code and / or data storage 705 is associated with dedicated computing resources (e.g., computing hardware 702 and computing hardware 706), respectively. In at least one embodiment, each of computing hardware 702 and computing hardware 706 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) only on the information stored in code and / or data storage 701 and code and / or data storage 705, respectively, and the results of the function execution are stored in activation storage 720.

[0065] In at least one embodiment, each of the code and / or data storage 701 and 705 and the corresponding computing hardware 702 and 706 corresponds to a different layer of the neural network, such that activations obtained from one storage / computation pair 701 / 702 of the code and / or data storage 701 and computing hardware 702 provide input as input to the next storage / computation pair 705 / 706 of the code and / or data storage 705 and computing hardware 706, in order to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / computation pair 701 / 702 and 705 / 706 may correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic 715 after or in parallel with the storage / computation pairs 701 / 702 and 705 / 706.

[0066] Neural network training and deployment

[0067] Figure 8 Training and deployment of a deep neural network according to at least one embodiment are illustrated. In at least one embodiment, an untrained neural network 806 is trained using a training dataset 802. In at least one embodiment, the training framework 804 is the PyTorch framework, while in other embodiments, the training framework 804 is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training frameworks. In at least one embodiment, the training framework 804 trains the untrained neural network 806 and enables it to be trained using the processing resources described herein to generate a trained neural network 808. In at least one embodiment, the weights may be randomly selected or pre-trained using a deep belief network. In at least one embodiment, training may be performed in a supervised, partially supervised, or unsupervised manner.

[0068] In at least one embodiment, supervised learning is used to train an untrained neural network 806, wherein training dataset 802 includes inputs paired with desired outputs for input, or wherein training dataset 802 includes inputs with known outputs and neural network 806 is manually graded output. In at least one embodiment, the untrained neural network 806 is trained in a supervised manner, and inputs from training dataset 802 are processed, and the resulting output is compared with a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through the untrained neural network 806. In at least one embodiment, training framework 804 adjusts the weights controlling the untrained neural network 806. In at least one embodiment, training framework 804 includes tools for monitoring the degree to which the untrained neural network 806 converges to a model (e.g., a trained neural network 808) adapted to generate the correct answer (e.g., result 814) based on input data (e.g., a new dataset 812). In at least one embodiment, training framework 804 repeatedly trains the untrained neural network 806 while adjusting the weights to improve the output of the untrained neural network 806 using a loss function and tuning algorithms (e.g., stochastic gradient descent). In at least one embodiment, the training framework 804 trains the untrained neural network 806 until the untrained neural network 806 reaches the desired accuracy. In at least one embodiment, the trained neural network 808 can then be deployed to perform any number of machine learning operations.

[0069] In at least one embodiment, unsupervised learning is used to train an untrained neural network 806, wherein the untrained neural network 806 attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised learning training dataset 802 will include input data without any associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 806 can learn groupings within the training dataset 802 and can determine how each input relates to the untrained dataset 802. In at least one embodiment, unsupervised training can be used to generate a self-organizing graph in the trained neural network 808, which is capable of performing operations useful for reducing the dimensionality of the new dataset 812. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows the identification of data points in the new dataset 812 that deviate from the normal patterns of the new dataset 812.

[0070] In at least one embodiment, semi-supervised learning can be used, a technique in which a mixture of labeled and unlabeled data is included in the training dataset 802. In at least one embodiment, the training framework 804 can be used to perform incremental learning, for example, through transfer learning techniques. In at least one embodiment, incremental learning enables the trained neural network 808 to adapt to a new dataset 812 without forgetting the knowledge injected into the trained neural network 808 during initial training.

[0071] Reference Figure 9 , Figure 9 This is an example data flow diagram of process 900 for generating and deploying a processing and inference pipeline according to at least one embodiment. In at least one embodiment, process 900 may be deployed to perform game name recognition analysis and inference on user feedback data at one or more facilities 902, such as a data center.

[0072] In at least one embodiment, process 900 may be executed within training system 904 and / or deployment system 906. In at least one embodiment, training system 904 may be used to train, deploy, and implement machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use with deployment system 906. In at least one embodiment, deployment system 906 may be configured to offload processing and computing resources in a distributed computing environment to reduce infrastructure requirements of facility 902. In at least one embodiment, deployment system 906 may provide a pipeline platform for selecting, customizing, and implementing virtual instruments for use with computing devices at facility 902. In at least one embodiment, virtual instruments may include software-defined applications for performing one or more processing operations on feedback data. In at least one embodiment, one or more applications in the pipeline may use or invoke services of deployment system 906 (e.g., inference, visualization, computation, AI, etc.) during application execution.

[0073] In at least one embodiment, some applications used in the advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at facility 902 using feedback data 908 (e.g., feedback data) stored at facility 902 or feedback data 908 from another or more facilities, or a combination thereof. In at least one embodiment, training system 904 may be used to provide applications, services, and / or other resources to generate a deployable machine learning model for the operation of deployment system 906.

[0074] In at least one embodiment, the model registry 924 may be supported by an object storage system that supports version control and object metadata. In at least one embodiment, it may be available from within a cloud platform via, for example, cloud storage (e.g., Figure 10 The system uses a Cloud 1026-compatible Application Programming Interface (API) to access object storage. In at least one embodiment, machine learning models within the model registry 924 can be uploaded, listed, modified, or deleted by the developer or partner of the system interacting with the API. In at least one embodiment, the API can provide access to methods that allow users with appropriate credentials to associate models with applications, enabling the models to be executed as part of the containerized instantiation of the application.

[0075] In at least one embodiment, training pipeline 1004 ( Figure 10 This can include situations where facility 902 is training its own machine learning model or has an existing machine learning model that needs optimization or updating. In at least one embodiment, feedback data 908 can be received from various channels (such as forums, web forms, etc.). In at least one embodiment, once feedback data 908 is received, AI-assisted annotation 910 can be used to help generate annotations corresponding to the feedback data 908 for use as ground truth data for the machine learning model. In at least one embodiment, AI-assisted annotation 910 can include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that can be trained to generate annotations corresponding to certain types of feedback data 908 (e.g., from certain devices), and / or certain types of anomalies in the feedback data 908. In at least one embodiment, AI-assisted annotation 910 can then be used directly, or it can be adjusted or fine-tuned using annotation tools to generate ground truth data. In at least one embodiment, in some examples, labeled data 912 can be used as ground truth data for training the machine learning model. In at least one embodiment, AI-assisted annotation 910, labeled data 912, or a combination thereof can be used as ground truth data for training the machine learning model. In at least one embodiment, the trained machine learning model may be referred to as output model 916 and may be used by deployment system 906 as described herein.

[0076] In at least one embodiment, training pipeline 1004 ( Figure 10This can include situations where facility 902 requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 906, but facility 902 may not currently have such a machine learning model (or may not have an efficient or effective model optimized for this purpose). In at least one embodiment, an existing machine learning model can be selected from model registry 924. In at least one embodiment, model registry 924 can include machine learning models trained to perform various inference tasks on imaging data. In at least one embodiment, the machine learning model in model registry 924 can be trained on imaging data from a different facility (e.g., a remote facility) instead of facility 902. In at least one embodiment, the machine learning model may have already been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a specific location, training can be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of the imaging data off-site (e.g., in compliance with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model has been trained or partially trained at one location, a machine learning model can be added to a model registry 924. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be used in the model registry 924. In at least one embodiment, a machine learning model (referred to as output model 916) can then be selected from the model registry 924 and used in a deployment system 906 to perform one or more processing tasks for one or more applications of the deployment system.

[0077] In at least one embodiment, training pipeline 1004 ( Figure 10This can be used in scenarios including facility 902, which requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 906, but facility 902 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for this purpose). In at least one embodiment, the machine learning model selected from model registry 924 may not be fine-tuned or optimized for the feedback data 908 generated at facility 902 due to population differences, genetic variation, robustness, anomalous diversity of the training data, and / or other problems with the training data used to train the machine learning model. In at least one embodiment, AI-assisted annotation 910 can be used to help generate annotations corresponding to the feedback data 908 for use as ground-based data for retraining or updating the machine learning model. In at least one embodiment, labeled data 912 can be used as ground-based data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 914. In at least one embodiment, model training 914 (e.g., AI-assisted annotation 910, labeled data 912, or a combination thereof) can be used as ground-based data for retraining or updating the machine learning model.

[0078] In at least one embodiment, the deployment system 906 may include software 918, service 920, hardware 922, and / or other components, features, and functions. In at least one embodiment, the deployment system 906 may include a software "stack" such that software 918 can be built on top of service 920 and can be used to perform some or all of the processing tasks, and service 920 and software 918 can be built on top of hardware 922 and can be used to perform processing, storage, and / or other computing tasks of the deployment system 906.

[0079] In at least one embodiment, software 918 may include any number of different containers, each of which may perform an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in an advanced processing and inference pipeline. In at least one embodiment, for each type of computing device, there may be any number of containers that may perform data processing tasks on feedback data 908 (or other data types, such as those described herein). In at least one embodiment, in addition to the containers that receive and configure imaging data for use by each container and / or for use by facility 902 after processing through the pipeline, the advanced processing and inference pipeline may also be defined based on the selection of different containers desired or required for processing feedback data 908 (e.g., to convert output back to a usable data type for storage and display in facility 902). In at least one embodiment, a combination of containers within software 918 (e.g., which constitute a pipeline) may be referred to as a virtual instrument (as described in more detail herein), and the virtual instrument may utilize service 920 and hardware 922 to perform some or all of the processing tasks of the application instantiated in the container.

[0080] In at least one embodiment, data may be preprocessed as part of a data processing pipeline to prepare it for processing by one or more applications. In at least one embodiment, post-processing may be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare output data for the next application and / or to prepare output data for user transmission and / or use (e.g., as a response to an inference request). In at least one embodiment, the inference task may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include the output model 916 of the training system 904.

[0081] In at least one embodiment, the tasks of the data processing pipeline can be encapsulated in one or more containers, each container representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications can be published to a private (e.g., limited access) area of ​​a container registry (described in more detail herein), and trained or deployed models can be stored in a model registry 924 and associated with one or more applications. In at least one embodiment, an image of the application (e.g., a container image) can be used in the container registry, and once a user selects an image from the container registry for deployment in the pipeline, that image can be used to generate containers for instantiation of the application for use by the user's system.

[0082] In at least one embodiment, the developer can develop, publish, and store an application (e.g., as a container) for performing processing and / or inference on the provided data. In at least one embodiment, a software development kit (SDK) associated with the system can be used to perform development, publication, and / or storage (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application can be tested locally using the SDK (e.g., at a first facility, on data from a first facility), the SDK serving as a system (e.g.,...). Figure 10 System 1000 may support at least some services 920. In at least one embodiment, once verified by system 1000 (e.g., for accuracy, etc.), the application becomes available in the container registry for users (e.g., hospitals, clinics, laboratories, healthcare providers, etc.) to select and / or implement one or more processing tasks on data at the user's facility (e.g., a second facility).

[0083] In at least one embodiment, the developer can then share the application or container over a network for the system (e.g., Figure 10 The system 900 allows for user access and use. In at least one embodiment, a completed and validated application or container may be stored in a container registry, and the associated machine learning model may be stored in a model registry 924. In at least one embodiment, a requesting entity (which provides an inference or image processing request) may browse the container registry and / or model registry 924 for applications, containers, datasets, machine learning models, etc., select the desired combination of elements to include in the data processing pipeline, and submit a processing request. In at least one embodiment, the request may include input data necessary to execute the request, and / or may include a selection of the application and / or machine learning model to be executed when the request is processed. In at least one embodiment, the request may then be passed to one or more components of the deployment system 906 (e.g., the cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 906 may include referencing elements (e.g., applications, containers, models, etc.) selected from the container registry and / or model registry 924. In at least one embodiment, once results are generated through the pipeline, the results may be returned to the user for reference (e.g., for viewing in a suite of viewing applications executed on a local machine, local workstation, or terminal).

[0084] In at least one embodiment, service 920 may be utilized to assist in processing or executing applications or containers in the pipeline. In at least one embodiment, service 920 may include computing services, collaborative content creation services, simulation services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, service 920 may provide functionality common to one or more applications in software 918, thus abstracting functionality into services that can be invoked or utilized by applications. In at least one embodiment, the functionality provided by service 920 can operate dynamically and more efficiently, while also allowing applications to process data in parallel (e.g., using...). Figure 10 The parallel computing platform 1030 in the system can be scaled well. In at least one embodiment, it is not required that each application providing the same functionality as the shared service 920 must have a corresponding instance of the service 920, but rather that the service 920 can be shared between and among various applications. In at least one embodiment, as a non-limiting example, the service may include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included, which can provide the ability to train and / or retrain machine learning models.

[0085] In at least one embodiment, where service 920 includes an AI service (e.g., an inference service), as part of application execution, one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) can be executed by invoking (e.g., as an API call) the inference service (e.g., an inference server) to execute one or more machine learning models or their processing. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, the application can invoke the inference service to execute the machine learning models for performing one or more processing operations associated with the segmentation task. In at least one embodiment, the software 918 implementing the high-level processing and inference pipeline can be pipelined because each application can invoke the same inference service to execute one or more inference tasks.

[0086] In at least one embodiment, hardware 922 may include a GPU, CPU, graphics card, AI / deep learning system (e.g., an AI supercomputer, such as NVIDIA's DGX supercomputer system), cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 922 may be used to provide efficient, specially designed support for software 918 and services 920 deployed in system 906. In at least one embodiment, GPU processing may be used to perform local processing (e.g., at facility 902) within the AI / deep learning system, in the cloud system, and / or other processing components of system 906 to improve the efficiency, accuracy, and performance of game name recognition.

[0087] In at least one embodiment, as a non-limiting example, with respect to deep learning, machine learning and / or high-performance computing, simulation and vision computing, software 918 and / or service 920 may be optimized for GPU processing. In at least one embodiment, at least some of the computing environments of deployment system 906 and / or training system 904 may be executed in a data center, one or more supercomputers, or high-performance computing systems with GPU-optimized software (e.g., the hardware and software combination of NVIDIA DGX systems). In at least one embodiment, as described herein, hardware 922 may include any number of GPUs that can be invoked to perform data processing in parallel. In at least one embodiment, the cloud platform may also include GPU-optimized execution for deep learning tasks, GPU processing for machine learning tasks, or other computing tasks. In at least one embodiment, the cloud platform (e.g., NVIDIA's NGC) may be executed using an AI / deep learning supercomputer and / or GPU-optimized software (e.g., as provided on NVIDIA's DGX systems) as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform may integrate application container cluster systems or coordination systems (e.g., KUBERNETES) across multiple GPUs to achieve seamless scaling and load balancing.

[0088] Figure 10 This is a system diagram of an example system 1000 for generating and deploying a deployment pipeline according to at least one embodiment. In at least one embodiment, system 1000 can be used to implement Figure 9 The process 900 and / or other processes include advanced processing and inference pipelines. In at least one embodiment, system 1000 may include a training system 904 and a deployment system 906. In at least one embodiment, the training system 904 and deployment system 906 may be implemented using software 918, services 920, and / or hardware 922, as described herein.

[0089] In at least one embodiment, system 1000 (e.g., training system 904 and / or deployment system 906) may be implemented in a cloud computing environment (e.g., using cloud 1026). In at least one embodiment, system 1000 may be implemented locally (regarding facilities) or as a combination of cloud computing resources and local computing resources. In at least one embodiment, access to APIs in cloud 1026 may be restricted to authorized users by establishing security measures or protocols. In at least one embodiment, the security protocol may include a network token, which may be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and may carry appropriate authorization. In at least one embodiment, the API of the virtual instrument (described herein) or other instances of system 1000 may be restricted to a set of public IPs that have been audited or authorized for interaction.

[0090] In at least one embodiment, the various components of system 1000 may communicate with each other and among themselves using any of a variety of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communication between facilities and components of system 1000 (e.g., for sending inference requests, for receiving the results of inference requests, etc.) may be transmitted via one or more data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0091] In at least one embodiment, similar to the description herein. Figure 9 As described, training system 904 can execute training pipeline 1004. In at least one embodiment, where deployment system 906 uses one or more machine learning models in deployment pipeline 1010, training pipeline 1004 can be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1006 (e.g., without retraining or updating). In at least one embodiment, as a result of training pipeline 1004, output model 916 can be generated. In at least one embodiment, training pipeline 1004 can include any number of processing steps, AI-assisted annotation 910, tagging or annotating feedback data 908 to generate labeled data 912, selecting a model from a model registry, model training 914, training, retraining, or updating the model, and / or other processing steps. In at least one embodiment, different training pipelines 1004 can be used for different machine learning models used by deployment system 906. In at least one embodiment, similar to the approach regarding... Figure 9 The training pipeline 1004 described in the first example can be used for the first machine learning model, similar to the one described above. Figure 9The training pipeline 1004 described in the second example can be used for a second machine learning model, similar to the one described above. Figure 9 The training pipeline 1004 of the third example described can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 904 can be used according to the requirements of each respective machine learning model. In at least one embodiment, one or more machine learning models may have already been trained and are ready for deployment, so the training system 904 may not perform any processing on the machine learning models, and the machine learning models may be implemented by the deployment system 906.

[0092] In at least one embodiment, according to an embodiment or example, one or more output models 916 and / or pre-trained models 1006 may include any type of machine learning model. In at least one embodiment, and not limited thereto, the machine learning model used by system 1000 may include models using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recursion, perceptrons, long / short-term memory (LSTM), Bi-LSTM, Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0093] In at least one embodiment, training pipeline 1004 may include AI-assisted annotation. In at least one embodiment, labeled data 912 (e.g., conventional annotation) can be generated using any number of techniques. In at least one embodiment, in some examples, labels or other annotations can be generated by drawing programs (e.g., annotation programs), computer-aided design (CAD) programs, tagging programs, another type of application suitable for generating annotations or labels for ground reality, and / or can be hand-drawn. In at least one embodiment, ground reality data can be synthetically generated (e.g., generated from computer models or renderings), realistically generated (e.g., designed and generated from real-world data), automatically generated by machines (e.g., extracting features from data using feature analysis and learning, and then generating labels), manually annotated (e.g., taggers or annotation experts, defining the placement of labels), and / or combinations thereof. In at least one embodiment, for each instance of feedback data 908 (or other data types used by the machine learning model), there may be corresponding ground reality data generated by training system 904. In at least one embodiment, AI-assisted annotation can be performed as part of deployment pipeline 1010; supplementing or replacing AI-assisted annotation included in training pipeline 1004. In at least one embodiment, system 1000 may include a multi-layer platform, which may include a software layer (e.g., software 918) of a diagnostic application (or other application type) that can perform one or more medical imaging and diagnostic functions.

[0094] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or certified API that can invoke (e.g., call) an application or container from an external environment (e.g., facility 902). In at least one embodiment, the application may then invoke or execute one or more services 920 to perform computational, AI, or visualization tasks associated with their respective applications, and the software 918 and / or service 920 may utilize hardware 922 to perform processing tasks efficiently and effectively.

[0095] In at least one embodiment, deployment system 906 may execute deployment pipeline 1010. In at least one embodiment, deployment pipeline 1010 may include any number of applications, which may be sequential, non-sequential, or otherwise applied to feedback data (and / or other data types) – including AI-assisted annotations, as described above. In at least one embodiment, as described herein, deployment pipeline 1010 for an individual device may be referred to as a virtual instrument for the device. In at least one embodiment, for a single device, there may be more than one deployment pipeline 1010, depending on the desired information from the data generated from the device.

[0096] In at least one embodiment, the applications that can be used to deploy pipeline 1010 may include any application that can perform processing tasks on feedback data or other data from the device. In at least one embodiment, since various applications can share common image operations, in some embodiments, data augmentation libraries (e.g., as one of services 920) may be used to accelerate these operations. In at least one embodiment, to avoid bottlenecks in traditional processing methods that rely on CPU processing, parallel computing platform 1030 may be used for GPU acceleration of these processing tasks.

[0097] In at least one embodiment, deployment system 906 may include user interface 1014 (e.g., graphical user interface, web interface, etc.) which may be used to select applications to be included in deployment pipeline 1010, deploy applications, modify or change applications or their parameters or configurations, use and interact with deployment pipeline 1010 during setup and / or deployment, and / or otherwise interact with deployment system 906. In at least one embodiment, although not shown with respect to training system 904, user interface 1014 (or different user interfaces) may be used to select models to be used in deployment system 906, to select models to be trained or retrained in training system 904, and / or to otherwise interact with training system 904.

[0098] In at least one embodiment, in addition to the application coordination system 1028, a pipeline manager 1012 may be used to manage interactions between applications or containers deployed through the pipeline 1010 and services 920 and / or hardware 922. In at least one embodiment, the pipeline manager 1012 may be configured to facilitate interactions from application to application, from application to service 920, and / or from application or service to hardware 922. In at least one embodiment, although shown as included in software 918, this is not intended to be limiting, and in some examples, the pipeline manager 1012 may be included in service 920. In at least one embodiment, the application coordination system 1028 (e.g., Kubernetes, DOCKER, etc.) may include a container coordination system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline 1010 (e.g., rebuilding applications, splitting applications, etc.) with individual containers, each application can execute in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0099] In at least one embodiment, each application and / or container (or its image) can be developed, modified, and deployed independently (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer). This allows focus on the tasks of a single application and / or container without being hindered by the tasks of another application or container. In at least one embodiment, the pipeline manager 1012 and the application coordination system 1028 can facilitate communication and collaboration between different containers or applications. In at least one embodiment, the application coordination system 1028 and / or the pipeline manager 1012 can facilitate communication and resource sharing between and within each application or container, provided that the expected inputs and / or outputs of each container or application are known to the system (e.g., based on the application or container's architecture). In at least one embodiment, since one or more applications or containers in the deployment pipeline 1010 can share the same services and resources, the application coordination system 1028 can coordinate, load balance, and determine the sharing of services or resources between and within the various applications or containers. In at least one embodiment, the scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. Therefore, in at least one embodiment, taking into account the system's needs and availability, the scheduler can allocate resources to different applications and distribute resources between and among applications. In some examples, the scheduler (and / or other components of the application coordination system 1028) can determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as Quality of Service (QoS), the urgency of data output (e.g., to determine whether to perform real-time processing or delayed processing), etc.

[0100] In at least one embodiment, service 920, utilized and shared by applications or containers in deployment system 906, may include computing service 1016, collaborative content creation service 1017, AI service 1018, simulation service 1019, visualization service 1020, and / or other service types. In at least one embodiment, an application may invoke (e.g., execute) one or more services 920 to perform processing operations for the application. In at least one embodiment, an application may utilize computing service 1016 to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more computing services 1016 may be utilized to perform parallel processing (e.g., using parallel computing platform 1030) to process data substantially simultaneously through one or more applications and / or one or more tasks of a single application. In at least one embodiment, parallel computing platform 1030 (e.g., NVIDIA's CUDA) may implement general-purpose computing on a GPU (GPGPU) (e.g., GPU 1022). In at least one embodiment, the software layer of parallel computing platform 1030 may provide access to the GPU's virtual instruction set and parallel computing elements to execute computing kernels. In at least one embodiment, the parallel computing platform 1030 may include memory, and in some embodiments, memory may be shared between and within multiple containers, and / or between and within different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within containers to use the same data from a shared memory segment of the parallel computing platform 1030 (e.g., where multiple different stages of one or more applications are processing the same information). In at least one embodiment, instead of copying data and moving it to different locations in memory (e.g., read / write operations), the same data in the same memory location can be used for any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, this information about the new location of the data can be stored and shared between applications because the resulting data from processing is used to generate new data. In at least one embodiment, the location of the data, and the location of the updated or modified data, may be part of the definition of how the payload in the container is understood.

[0101] In at least one embodiment, AI service 1018 may be used to perform an inference service for executing a machine learning model associated with the application (e.g., a task to perform one or more processing tasks of the application). In at least one embodiment, AI service 1018 may utilize AI system 1024 to execute a machine learning model (e.g., a neural network such as a CNN) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, the application deploying pipeline 1010 may use one or more output models 916 from training system 904 and / or other models from the application to perform inference on imaging data (e.g., DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.). In at least one embodiment, two or more examples of inference using application coordination system 1028 (e.g., a scheduler) may be available. In at least one embodiment, the first category may include a high-priority / low-latency path that can implement a higher service level protocol, such as for performing inference on urgent requests in emergency situations or for radiologists during diagnostic procedures. In at least one embodiment, the second category may include standard priority paths, which can be used for requests that may not be urgent or for situations where analysis can be performed at a later time. In at least one embodiment, the application coordination system 1028 may allocate resources (e.g., services 920 and / or hardware 922) based on priority paths for different inference tasks of the AI ​​service 1018.

[0102] In at least one embodiment, shared memory may be installed into AI service 1018 in system 1000. In at least one embodiment, shared memory may operate as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a set of API instances of deployment system 906 may receive the request and may select one or more instances (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request may be fed into a database, and if not already in the cache, a machine learning model may be located from model registry 924. A verification step may ensure that an appropriate machine learning model is loaded into the cache (e.g., shared memory), and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not already running or there are not enough instances of the application, a scheduler (e.g., the scheduler of pipeline manager 1012) may be used to start the application referenced in the request. In at least one embodiment, if an inference server has not yet been started to execute the model, an inference server may be started. In at least one embodiment, any number of inference servers may be started for each model. In at least one embodiment, in a pull model that clusters inference servers, the model can be cached whenever load balancing is favorable. In at least one embodiment, the inference servers can be statically loaded into the corresponding distributed servers.

[0103] In at least one embodiment, an inference server running in a container can be used to perform inference. In at least one embodiment, an instance of the inference server can be associated with a model (and optionally multiple versions of the model). In at least one embodiment, if an instance of the inference server does not exist when a request to perform inference on the model is received, a new instance can be loaded. In at least one embodiment, when the inference server is started, a model can be passed to the inference server, allowing the same container to be used to serve different models, as long as the inference server runs as different instances.

[0104] In at least one embodiment, during application execution, an inference request for a given application can be received, and a container (e.g., an instance of a hosted inference server) can be loaded (if not already loaded), and a launcher can be invoked. In at least one embodiment, preprocessing logic within the container can (e.g., using a CPU and / or GPU) load, decode, and / or perform any additional preprocessing on the incoming data. In at least one embodiment, once the data is ready for inference, the container can infer the data as needed. In at least one embodiment, this can include a single inference call for an image (e.g., a hand X-ray) or can request inference for hundreds of images (e.g., a chest CT scan). In at least one embodiment, the application can summarize the results before completion, which may include, but is not limited to, a single confidence score, pixel-level segmentation, voxel-level segmentation, generating visualizations, or generating text to summarize the results. In at least one embodiment, different priorities can be assigned to different models or applications. For example, some models may have a real-time (TAT less than 1 minute) priority, while other models may have a lower priority (e.g., TAT less than 10 minutes). In at least one embodiment, model execution time can be measured from the requesting agency or entity, and may include cooperative network traversal time and inference service execution time.

[0105] In at least one embodiment, the transfer of requests between service 920 and the inference application can be hidden behind a software development kit (SDK) and robust transfer can be provided via queues. In at least one embodiment, requests are placed in queues via an API for individual application / tenant ID combinations, and the SDK pulls requests from the queues and provides them to the application. In at least one embodiment, the name of the queue can be provided in the environment where the SDK picks up the queue. In at least one embodiment, asynchronous communication via queues may be useful because it allows any instance of the application to pick up work as it becomes available. In at least one embodiment, results can be sent back via queues to ensure no data loss. In at least one embodiment, queues can also provide the ability to partition work, as the highest priority work can go into a queue connected to a majority of instances of the application, while the lowest priority work can go into a queue connected to a single instance that processes tasks in the order they are received. In at least one embodiment, the application can run on a GPU-accelerated instance generated in cloud 1026, and the inference service can perform inference on the GPU.

[0106] In at least one embodiment, visualization service 1020 can be used to generate visualizations for viewing the output of application and / or deployment pipeline 1010. In at least one embodiment, visualization service 1020 can utilize GPU 1022 to generate visualizations. In at least one embodiment, visualization service 1020 can implement rendering effects such as ray tracing or other light transport simulation techniques to generate higher quality visualizations. In at least one embodiment, visualizations can include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slicing, virtual reality display, augmented reality display, etc. In at least one embodiment, a virtualized environment can be used to generate virtual interactive displays or environments (e.g., virtual environments) for system users (e.g., doctors, nurses, radiologists, etc.) to interact with. In at least one embodiment, visualization service 1020 can include an internal visualizer, cinematic and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0107] In at least one embodiment, hardware 922 may include GPU 1022, AI system 1024, cloud 1026, and / or any other hardware for executing training system 904 and / or deployment system 906. In at least one embodiment, GPU 1022 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that can be used to perform processing tasks for any feature or function of computing service 1016, collaborative content creation service 1017, AI service 1018, simulation service 1019, visualization service 1020, other services, and / or software 918. For example, for AI service 1018, GPU 1022 may be used to perform preprocessing on imaging data (or other data types used by machine learning models), postprocessing on the output of machine learning models, and / or inference (e.g., to execute machine learning models). In at least one embodiment, cloud 1026, AI system 1024, and / or other components of system 1000 may use GPU 1022. In at least one embodiment, cloud 1026 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1024 may use a GPU, and one or more AI systems 1024 may be used to perform cloud 1026 (or at least part of a task for deep learning or inference). Similarly, although hardware 922 is shown as a discrete component, this is not intended to be limiting, and any component of hardware 922 may be combined with or utilized by any other component of hardware 922.

[0108] In at least one embodiment, AI system 1024 may include a specially built computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, in addition to CPU, RAM, memory, and / or other components, features, or functions, AI system 1024 (e.g., NVIDIA's DGX) may also include GPU-optimized software (e.g., a software stack) that can be performed using multiple GPUs 1022. In at least one embodiment, one or more AI systems 1024 may be implemented in a cloud 1026 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1000.

[0109] In at least one embodiment, cloud 1026 may include GPU-accelerated infrastructure (e.g., NVIDIA's NGC) that can provide a GPU-optimized platform for performing processing tasks of system 1000. In at least one embodiment, cloud 1026 may include AI system 1024 for performing one or more AI-based tasks of system 1000 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1026 may be integrated with application coordination system 1028 utilizing multiple GPUs to achieve seamless scaling and load balancing between and within applications and services 920. In at least one embodiment, as described herein, cloud 1026 may be responsible for performing at least some of the services 920 of system 1000, including computing service 1016, AI service 1018, and / or visualization service 1020. In at least one embodiment, cloud 1026 may perform large and small batch inference (e.g., perform NVIDIA's TENSOR RT), provide accelerated parallel computing APIs and platform 1030 (e.g., NVIDIA's CUDA), perform application coordination system 1028 (e.g., KUBERNETES), provide graphics rendering APIs and platform (e.g., for ray tracing, 2D graphics, 3D graphics and / or other rendering techniques to produce higher quality cinematic effects), and / or provide other functionalities for system 1000.

[0110] In at least one embodiment, to protect patient confidentiality (e.g., in cases of off-site use of patient data or records), cloud 1026 may include a registry—e.g., a deep learning container registry. In at least one embodiment, the registry may store containers for instantiating applications that can perform preprocessing, post-processing, or other processing tasks on patient data. In at least one embodiment, cloud 1026 may receive data, including patient data and sensor data in containers, perform the requested processing only on the sensor data in those containers, and then forward the output and / or visualization to appropriate parties and / or devices (e.g., local medical devices for visualization or diagnosis), without extracting, storing, or otherwise accessing the patient data. In at least one embodiment, the confidentiality of patient data is maintained in accordance with HIPAA and / or other data regulations.

[0111] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0112] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar references, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (referring to a physical connection where not modified) should be interpreted as partially or wholly contained, attached to, or joined together, even with some intervention. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0113] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” means multiple items). In at least one embodiment, the number of items in the multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” rather than “based on only”.

[0114] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is executed jointly on one or more processors via hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within transient signal transceivers. In at least one embodiment, code (e.g., executable code or source code) is stored on one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media lack all the code, but the multiple non-transitory computer-readable storage media collectively store all the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0115] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the operations of the processes described herein, either individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating in different ways, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0116] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not constitute a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0117] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference as if each reference were individually and specifically indicated to be incorporated herein by reference and the entire contents of which are described herein.

[0118] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0119] Unless otherwise expressly stated, it will be understood that throughout this specification, terms such as “processing,” “computing,” “determining,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that process and / or convert data represented as physical quantities (e.g., electrons) in the registers and / or memory of the computing system into other data represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0120] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or intermittently, sequentially, or in parallel. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0121] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an application programming interface, or an inter-process communication mechanism.

[0122] While the description herein illustrates exemplary embodiments of the described technologies, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for descriptive purposes, various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0123] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A method comprising: Identify the first image and the first reference image; Determine a first plurality of motion vectors MV, each of the first plurality of MVs representing the displacement of one or more pixels of the first image relative to the first reference image; The first region of the first image is identified using the first plurality of MVs, the first region including a depiction of one or more objects in the first image; Based on the size of the first region, a first MLM of multiple machine learning model MLMs is selected, each of the multiple MLMs corresponding to a corresponding one of multiple input sizes; Based on the size of the first region and the input size corresponding to the first MLM, a first portion of the first image is selected, wherein the first portion includes the first region; as well as The presence of one or more objects in the first image is detected based on the output of the processing of the first portion using the first MLM.

2. The method as described in claim 1, wherein, The first portion of the first image is selected to have a size equal to the input size corresponding to the first MLM.

3. The method as described in claim 1, wherein, The first portion of the first image is selected to have a size different from the input size of the first MLM, and the method further includes: Before the first portion is processed by the first MLM, the first portion of the first image is rescaled to match the input size corresponding to the first MLM.

4. The method of claim 1, wherein, The output of the processing of the first portion by the first MLM includes bounding boxes for at least one of the one or more objects in the first image.

5. The method of claim 1, further comprising: Identify the second image and the second reference image; Determine a second plurality of motion vectors MV, each of the second plurality of MV representing the displacement of one or more pixels of the second image relative to the second reference image; The second region of the second image is identified using the second plurality of MVs, the second region including the depiction of one or more objects in the second image; The second MLM among the plurality of machine learning models is selected based on the size of the second region, wherein the input size corresponding to the second MLM is different from the input size corresponding to the first MLM; Based on the size of the second region and the input size corresponding to the second MLM, a second portion of the second image is selected, wherein the second portion includes the second region; as well as The presence of one or more objects in the second image is detected by using the output of the processing of the second part using the second MLM.

6. The method of claim 5, further comprising: The objects in the first image and the objects in the second image are classified in the following way: The combined input is processed using one or more classifiers (MLMs), wherein the combined input includes: The output of the processing of the first part using the first MLM, and The output of the processing of the second part using the second MLM, and wherein one or more classifier MLMs are trained to determine the type of one or more objects depicted in the combined input.

7. The method of claim 6, wherein the output generated using the one or more classifiers MLM includes one or more inferences corresponding to at least one of the one or more objects in at least one of the first image or the second image: i) vehicle type, ii) vehicle brand or iii) vehicle model.

8. The method of claim 6, wherein, The combined input processed by the one or more classifiers MLM is one of a plurality of combined inputs, and wherein the one or more classifiers MLM are configured to perform pipeline processing of the plurality of combined inputs.

9. The method of claim 5, wherein, The first image and the first reference image are acquired by a first camera, wherein the second image and the second reference image are acquired by a second camera, and wherein the field of view of the first camera is different from the field of view of the second camera.

10. The method of claim 1, wherein, Processing the first portion using the first MLM includes performing one or more calculations associated with the first MLM using one or more graphics processing units.

11. A system comprising: Memory devices; as well as One or more processing devices, communicatively coupled to the memory device, are used for: Identify the first image and the first reference image; Determine a first plurality of motion vectors MV, each of the first plurality of MVs representing the displacement of one or more pixels of the first image relative to the first reference image; The first region of the first image is identified using the first plurality of MVs, the first region including a depiction of one or more objects in the first image; Based on the size of the first region, a first MLM of multiple machine learning model MLMs is selected, each of the multiple MLMs corresponding to a corresponding one of multiple input sizes; Based on the size of the first region and the input size of the first MLM, a first portion of the first image is selected, wherein the first portion includes the first region; as well as The presence of one or more objects in the first image is detected based on the output of the processing of the first portion using the first MLM.

12. The system of claim 11, wherein, The first portion of the first image is selected to have a size equal to the input size corresponding to the first MLM.

13. The system of claim 11, wherein, The first portion of the first image is selected to have a size different from the input size of the first MLM, and wherein the one or more processing devices are further configured to: Before processing the first portion using the first MLM, the first portion of the first image is rescaled to match the input size corresponding to the first MLM.

14. The system of claim 11, wherein, The one or more processing devices are also used for: Identify the second image and the second reference image; A second plurality of motion vectors MV are determined, each of the second plurality of MVs representing the displacement of one or more pixels of the second image relative to the second reference image; The second region of the second image is identified using the second plurality of MVs, the second region including the depiction of one or more objects in the second image; Based on the size of the second region, a second MLM is selected from the plurality of machine learning models MLM, wherein the input size corresponding to the second MLM is different from the input size corresponding to the first MLM; Based on the size of the second region and the input size of the second MLM, a second portion of the second image is selected, wherein the second portion includes the second region; as well as The presence of one or more objects in the second image is detected by using the output of the processing of the second part using the second MLM.

15. The system of claim 14, wherein, The one or more processing devices are also used for: The combined input is processed using one or more classifiers (MLMs), wherein the combined input includes: The output of the processing of the first part using the first MLM, and The output of the processing of the second part using the second MLM, wherein the one or more classifier MLMs are trained to determine the type of one or more objects depicted in the combined input.

16. The system of claim 15, wherein the output generated using the one or more classifiers MLM includes one or more inferences corresponding to one of the one or more objects in at least one of the first image or the second image: i) vehicle type, ii) vehicle brand or iii) vehicle model.

17. The system of claim 11, wherein, The one or more processing devices include one or more graphics processing units.

18. A non-transitory computer-readable medium storing instructions thereon, wherein the instructions, when executed by a processing device, cause the processing device to: Identify the first image and the first reference image; Determine a first plurality of motion vectors MV, each of the first plurality of MVs representing the displacement of one or more pixels of the first image relative to the first reference image; The first region of the first image is identified using the first plurality of MVs, the first region including a depiction of one or more objects in the first image; Based on the size of the first region, a first MLM of multiple machine learning model MLMs is selected, each of the multiple MLMs corresponding to a corresponding one of multiple input sizes; Based on the size of the first region and the input size corresponding to the first MLM, a first portion of the first image is selected, wherein the first portion includes the first region; as well as The presence of one or more objects in the first image is detected based on the output of the processing of the first portion using the first MLM.

19. The computer-readable medium of claim 18, wherein, The instruction also causes the processing device to: Identify the second image and the second reference image; A second plurality of motion vectors MV are determined, each of the second plurality of MVs representing the displacement of one or more pixels of the second image relative to the second reference image; The second region of the second image is identified using the second plurality of MVs, the second region including the depiction of one or more objects in the second image; Based on the size of the second region, a second MLM is selected from the plurality of machine learning models MLM, wherein the input size corresponding to the second MLM is different from the input size corresponding to the first MLM; Based on the size of the second region and the input size of the second MLM, a second portion of the second image is selected, wherein the second portion includes the second region; as well as The presence of one or more objects in the second image is detected based on the output of the processing of the second part using the second MLM.

20. The computer-readable medium of claim 19, wherein the instructions further cause the processing apparatus to: The combined input is processed using one or more classifiers (MLMs), wherein the combined input includes: The output of the processing of the first part using the first MLM, and The output of the processing of the second part using the second MLM, and wherein one or more classifier MLMs are trained to determine the type of one or more objects depicted in the combined input.

Images