Training an object detection model using transfer learning

Abstract

To train an object detection model using transfer learning.SOLUTION: A method 600 of training a machine learning model and updating the machine learning model trained to remove a mask head comprises: a step 610 of generating training data for the machine learning model that includes generating a training input including an image showing an object and generating a target output including an instruction of a bounding box, mask data and a class related to the shown object for the training input; a step 616 of providing the training data for training the machine learning model about a set of training inputs including the generated training input and a set of the target outputs including the generated target output; a step 618 of identifying a head of the trained machine learning model corresponding to prediction of the mask data for the given input image; and a step 620 of updating the machine learning model trained to remove the identified head.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] At least one example performs operations for training an object detection model using transfer learning and relates to processing resources used to facilitate. For example, at least one example provides one or more computing systems for training an object detection model for detecting objects of a target class shown in one or more images using transfer learning according to various novel techniques described herein, and relates to a processor or computing system used to enable.

Background Art

[0002] Machine learning is often applied to image processing, such as the identification of objects shown in an image. Object identification can be used in medical imaging, scientific research, autonomous driving systems, robotic automation, security applications, law enforcement activities, and many other settings. Machine learning involves training a computing system to identify patterns in images that can facilitate object detection using training images and other training data. The training can be with or without a teacher. Machine learning models can use various calculation algorithms such as decision tree algorithms (or other rule-based algorithms), artificial neural networks, etc. During the inference phase, a new image is input into the trained machine learning model, and various target objects of interest (e.g., vehicles in an image of a road) can be identified using the patterns and features identified during training.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

[0004] Various embodiments of this disclosure will be described with reference to the drawings. [Brief explanation of the drawing]

[0005] [Figure 1] This is a block diagram of an exemplary system architecture, based on at least one embodiment. [Figure 2] This is a block diagram of an exemplary training data generator and an exemplary training engine according to at least one embodiment. [Figure 3] This is a block diagram of an exemplary object detection engine, according to at least one embodiment. [Figure 4A] This figure shows an exemplary trained object detection model based on at least one embodiment. [Figure 4B] This figure shows an exemplary trained object detection model updated to remove mask heads, according to at least one embodiment. [Figure 5A] This is a flowchart illustrating an exemplary method for training a machine learning model to detect objects of a target class, using at least one embodiment. [Figure 5B]This is a flowchart illustrating an exemplary method using a machine learning model trained to detect objects of a target class, with at least one embodiment. [Figure 6] This is a flowchart illustrating an exemplary method for training a machine learning model and updating the trained machine learning model to remove the mask head, in at least one embodiment. [Figure 7A] This figure shows the inference and / or training logic according to at least one embodiment. [Figure 7B] This figure shows the inference and / or training logic according to at least one embodiment. [Figure 8] This figure shows an exemplary data center system according to at least one embodiment. [Figure 9] This figure shows a computer system according to at least one embodiment. [Figure 10] This figure shows a computer system according to at least one embodiment. [Figure 11] This figure shows at least a portion of a graphics processor according to one or more embodiments. [Figure 12] This figure shows at least a portion of a graphics processor according to one or more embodiments. [Figure 13] This is an exemplary data flow diagram for an advanced computing pipeline, with at least one implementation example. [Figure 14] This is a system diagram for an exemplary system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, with at least one embodiment. [Figure 15A] This is a data flow diagram for the process of training a machine learning model, with at least one example. [Figure 15B]This figure shows a client-server architecture for extending annotation tools with a pre-trained annotation model, based on at least one embodiment. [Figure 16A] This figure shows an example of an autonomous vehicle, based on at least one embodiment. [Figure 16B] This figure shows an example of camera location and field of view for the autonomous vehicle shown in Figure 16A, based on at least one embodiment. [Figure 16C] This figure shows an exemplary system architecture for an autonomous vehicle, according to at least one embodiment, as shown in Figure 16A. [Figure 16D] This figure shows a system for communication between (one or more) cloud-based servers and the autonomous vehicle shown in Figure 16A, according to at least one embodiment. [Modes for carrying out the invention]

[0006] Accurately detecting and classifying objects in images representing various environments is a challenging task. Advances have been made in machine learning models trained to detect objects in a given input image. However, the accuracy of object detection and classification provided by a machine learning model depends on the data used to train the model. In one example, a smart surveillance system could use a machine learning model to detect objects of a target class (e.g., "people" or "human" class) in images captured by a camera (e.g., a surveillance camera, a camera for an autonomous vehicle, etc.). In addition to detecting and classifying objects shown in a given input image, a machine learning model can be trained to determine one or more characteristics associated with the detected object. As in the previous example, a machine learning model used by a smart surveillance system to detect objects of a target class can also be trained to predict the position of the detected object in a given input image (e.g., relative to other objects shown in the given input image).

[0007] To train a model to detect objects of a target class with a high degree of accuracy (e.g., 95% or more), training data can be generated based on a large number of images (e.g., thousands, or in some cases, millions) (referred to herein as training images). In some systems, the data used to train the model (referred to herein as training data) may include indications of the region of each training image containing an object (e.g., bounding box), indications of whether the object in that region corresponds to the target class, and additional data (e.g., mask data) indicating the object's location (e.g., posture, position, or orientation) or shape. Collecting a considerable number of images suitable for use in training a model to detect and classify objects of a target class can take a considerable amount of time (e.g., months, or in some cases, years). Furthermore, accurately determining the labeled data for each image (e.g., the region of each image containing an object, the class associated with the object, and the mask data associated with the object) can take an even greater amount of time. Some systems rely solely on highly trusted entities to accurately determine and provide labeled data for training images. For example, some systems rely on humans to provide region, class, and additional data indications for objects shown in each training image. However, obtaining labeled data for each training image in such systems can be excessively costly and time-consuming, as the highly trusted entity providing the labeled data must examine thousands, if not millions, of images to determine and specify the labeled data for each image.

[0008] In some cases, the labeled data can be generated based on a smaller number of training images (e.g., dozens or hundreds), and / or based on decisions made by entities that are not highly reliable entities. In such a system, a machine learning model can first be trained to detect and classify objects contained in a given input image with a low degree of accuracy (e.g., less than 95%). During the introduction, the model can be retrained based on feedback provided for the data determined based on one or more outputs of the model (e.g., the region of a given input image containing the detected object, and / or the class determined for the detected object in a given input image, etc.). Finally, the model can be retrained while being introduced to detect and classify objects with a high degree of accuracy. However, retraining the model to detect and classify objects with a high degree of accuracy can, in some cases, consume time and computing resources to train the model, and additional time and resources are consumed to retrain the model during the interference phase, so it may require a much larger amount of time and computing resources.

[0009] Embodiments of the present disclosure address the above and other deficiencies by providing a transfer learning technique for training an object detection model to detect objects associated with a target class in a given input image. A first machine learning model (also called a teacher model) can be trained (e.g., by a training data generator and / or training engine) to detect one or more objects shown in a given input image. In some embodiments, the objects shown in a given input image may correspond to at least one of several (e.g., tens, hundreds, etc.) different classes. The teacher model can be trained using first training data, which may include a training input containing one or more images and a target output containing labeled data such as data associated with each object shown in each of the set of images. In some embodiments, the data associated with each object may include an indication of a region in one or more images containing the object, an indication of a class (i.e., a class among several different classes) associated with the object, and / or mask data associated with the object. The mask data refers to data (e.g., a two-dimensional bit array) indicating whether one or more pixels (or groups of pixels) for an image correspond to a certain object. In some embodiments, the images and data associated with the objects shown in the images may be obtained from a publicly available repository or database containing a large number of different image and object datasets that can be used to train an object detection machine learning model. The training model may be trained using the first training data to detect one or more objects shown in a given set of input images and, for each detected object, to predict at least mask data associated with each detected object. In some additional embodiments, the training model may be further trained to predict regions in the images (e.g., bounding boxes) from a given set of input images that contain the shown objects, and / or the class (i.e., a class among several different classes) corresponding to the detected objects.

[0010] Once a teacher model is trained using the first training data, the trained teacher model may be used to generate second training data for training a second machine learning model (called a student model) to detect objects of a target class indicated in a given input image. A set of images may be provided as input to the teacher model. In some embodiments, each of the sets of images to be used to generate the second training data for training the student model may be selected (for example, from a domain-specific or organization-specific repository or database). Object data associated with objects detected in each of the sets of images provided as input to the teacher model may be determined based on one or more acquired outputs of the teacher model. In some embodiments, the object data may include mask data associated with each detected object. In some additional embodiments, the object data may further include, for each detected object, data indicating the image region containing the detected object, and / or a class indication associated with the detected object.

[0011] To train a student model related to a target class of objects, second training data including one or more outputs of a teacher model may be used. In particular, in some embodiments, the second training data may include a training input including a set of images and a target output including mask data related to each object detected in the set of images. In some embodiments, the training data generator may use the obtained output of the teacher model to obtain mask data related to each object. The target output of the second training data may also include an indication of whether the class associated with each object detected in the set of images corresponds to the target class. For example, as described above, one or more outputs of the teacher model may include an indication of the class associated with an object detected in a given input image included in the second training data. In some embodiments, the training data generator may determine whether the class associated with each detected object corresponds to the target class based on the obtained output of the teacher model. The target output of the second training data may further include ground truth data related to each object detected in the set of images. The ground truth data may indicate a region of the image (e.g., a bounding box) that includes each detected object. In some embodiments, the training data generator may obtain ground truth data from a database including indications of one or more bounding boxes related to an image in the set of images (e.g., for use instead of the bounding box data provided by the teacher model for higher accuracy). In some embodiments, the database may be a domain-specific or organization-specific database including the set of images. Each bounding box related to an image may be provided by an approved bounding box authority entity of the computing platform or by a user.

[0012] The second training data can be used to train a student model to predict bounding boxes and mask data associated with objects detected in a given input image, for a given input image. The student model can also be trained to predict whether the classes associated with objects detected in a given input image correspond to a target class. As described above, the teacher model can be trained to predict multiple classes for objects detected in a given input image. Since the student model is trained to predict a specific class of objects (i.e., the target class) rather than multiple object classes, the student model may provide more accurate predictions than the teacher model.

[0013] In some cases, a trained student model may be a multi-head machine learning model. For example, a trained student model may include a first head for predicting bounding boxes associated with objects detected in a given image, a second head for predicting classes associated with the detected objects, and a third head for predicting mask data associated with the detected objects. An object detection engine (e.g., a computing device, a cloud computing platform) may, in some cases, identify a head of the student model (called a mask head) corresponding to predicting mask data associated with the detected objects, and can update the student model to remove the identified head. After the mask head has been removed from the student model, the updated student model can be used to predict bounding boxes and classes associated with objects detected in a given input image. By initially including a mask head in the student model, the object detection and classification predictions of the updated student model may be more accurate than those of an object detection model trained using training data that does not include mask data associated with objects shown in the provided training images. Furthermore, the inference speed associated with the student model can be significantly improved (e.g., 10-20%) after the mask head is removed from the student model, and the model size associated with the student model can be significantly reduced. Thus, in some cases, the updated student model can be transmitted over the network to edge devices and / or one or more endpoint devices (e.g., smart surveillance cameras, autonomous vehicles) for use in object detection.

[0014] Aspects and embodiments of this disclosure provide techniques for training object detection models using transfer learning. Using a large number of images (i.e., from publicly available repositories or databases) representing various classes of objects, training data can be used to train a teacher model to make predictions with a sufficient degree of accuracy. Image data may be obtained from domain-specific or organization-specific repositories or databases and used as input to the teacher model to obtain predictions, which can then be used to train a student model to increase predictive accuracy for specific, focused, or particular image classes. Thus, the student model can be trained to make predictions with a (more) higher degree of accuracy (e.g., 95% or higher) without obtaining labeled data for training from experts or other accepted authorities. Furthermore, embodiments of the present disclosure provide the ability to provide object detection models to be used in edge devices and / or endpoint devices (e.g., smart surveillance cameras, autonomous vehicles, etc.), which are trained to detect target class objects with a high degree of accuracy and to satisfy size constraints and inference speed conditions associated with edge devices and / or endpoint devices.

[0015] System Architecture Figure 1 is a block diagram of an exemplary system architecture 100 according to at least one embodiment. The system architecture 100 (also referred to herein as the “system”) includes a computing device 102, data stores 112A-N (collectively and individually referred to as data stores 112), and server machines 130, 140, and / or 150. In implementations, the network 110 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network or a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and / or combinations thereof.

[0016] Computing device 102 may be a desktop computer, laptop computer, smartphone, tablet computer, server, or any suitable computing device capable of carrying out the techniques described herein. In some embodiments, computing device 102 may be a computing device of a cloud computing platform. For example, computing device 102 may be a server machine of a cloud computing platform, or a component thereof. In such embodiments, computing device 102 may be coupled to one or more edge devices (not shown) via network 110. An edge device refers to a computing device that enables communication between computing devices at the boundary of two networks. For example, an edge device may be connected to computing device 102, data stores 112A-N, server machine 130, server machine 140, and / or server machine 150 via network 110, and may be connected to one or more endpoint devices (not shown) via another network. In such an example, the edge device can enable communication between computing device 102, data stores 112A-N, server machine 130, server machine 140, and / or server machine 150 and one or more client devices. In other or similar embodiments, computing device 102 may be an edge device or a component thereof. For example, computing device 102 may facilitate communication between data stores 112A-N, server machines 130, server machine 140, and / or server machine 150, which are connected to computing device 102 via network 110, and one or more client devices connected to computing device 102 via another network.

[0017] In further or similar embodiments, computing device 102 may be an endpoint device or a component thereof. For example, computing device 102 may be a device or a component thereof, but is not limited to, a television, smartphone, cellular phone, personal digital assistant (PDA), portable media player, netbook, laptop computer, e-book reader, tablet computer, desktop computer, set-top box, gaming console, autonomous vehicle, surveillance device, etc. In such embodiments, computing device 102 may be connected to data stores 112A-N, server machine 130, server machine 140 and / or server machine 150 via network 110. In other or similar embodiments, computing device 102 may be connected via a network to edge devices (not shown) of system 100, and edge devices of system 100 may be connected via network 110 to data stores 112A-N, server machines 130, server machines 140 and / or server machines 150.

[0018] The computing device 102 may include memory 104. Memory 104 may include one or more volatile and / or non-volatile memory devices configured to store data. In some embodiments, the computing device 102 may include an object detection engine 151. The object detection engine 151 may be configured to detect one or more objects shown in an image (e.g., image 106) and, in some embodiments, to obtain data (e.g., object data 108) associated with one or more detected objects. For example, the object detection engine 151 may be configured to provide image 106 as input to a trained object detection model (e.g., model 160) and determine object data 108 associated with image 106 based on one or more outputs of the trained object detection model. While the implementations of this disclosure are described in relation to an object detection model, it should be noted that the implementations can generally be applied to any type of machine learning model. Further details regarding the object detection engine 151 and the object detection model are provided herein.

[0019] As described above, the computing device 102 may, in some embodiments, be an endpoint device or a component thereof. In such embodiments, the computing device 102 may include an audiovisual component that can generate audio data and / or visual data. In some embodiments, the audiovisual component may include an image capture device (e.g., a camera) for capturing and generating an image 106, and may generate image data and / or video data associated with the generated image 106. In other or similar embodiments, the computing device 102 may, as described above, be an edge device or a component thereof. In such embodiments, the computing device 102 may receive an image 106 from an endpoint device including an audiovisual component (i.e., via network 110 or another network). Also as described above, the computing device 102 may, in some embodiments, be a server machine (e.g., for a cloud computing platform) or a component thereof. In such embodiments, the computing device 102 may receive images 106 from an endpoint device including audio-visual components, and / or edge devices connected to that endpoint device (i.e., via network 110 or another network).

[0020] In some implementations, data stores 112A-N are persistent storage capable of storing content items (e.g., images), data associated with the stored content items (e.g., object data), and data structures for tagging, organizing, and indexing the content items and / or object data. Data store 112 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage-based disks, tapes or hard drives, NAS, SAN, etc. In some implementations, data store 112 may be a network-attached file server, but in other embodiments, data store 112 may be some other type of persistent storage, such as an object-oriented database or relational database, hosted by a compute 102, or one or more different machines coupled to the compute device 102 via a network 110.

[0021] As shown in Figure 1, the system 100 may include a plurality of data stores 112 in some embodiments. In some embodiments, a first data store (e.g., data store 112A) may be configured to store data accessible only to computing device 102, server machine 130, server machine 140, and / or server machine 150. For example, data store 112A may be or include a domain-specific or organization-specific repository or database. In some embodiments, computing device 102, server machine 130, server machine 140, and / or server machine 150 may be able to access data store 112A only via network 110, which may be a private network. In other or similar embodiments, the data stored in data store 112A may be encrypted and accessible to computing device 102, server machine 130, server machine 140, and / or server machine 150 via an encryption mechanism (e.g., a private encryption key). In additional or alternative embodiments, a second data store (e.g., data store 112B) may be configured to store data accessible to any device accessible to data store 112B via any network. For example, data store 112B may be or include a publicly accessible repository or database. In some embodiments, data store 112B may be a publicly accessible data store accessible to any device via a public network. In additional or alternative embodiments, system 100 may include data store 112 configured to store first data accessible only to computing device 102, server machine 130, server machine 140, and / or server machine 150 (e.g., via a private network 110, via an encryption mechanism, etc.) and second data accessible to devices connected to the data store via another network (e.g., a public network).In additional or alternative embodiments, system 100 may include only a single data store 112 configured to store data accessible only to computing device 102, server machine 130, server machine 140, and / or server machine 150 (for example, via a private network 110, via an encryption mechanism, etc.). In such embodiments, data store 112 may store data retrieved from a publicly accessible data store (for example, by computing device 102, training data generator 131, training engine 141, etc.).

[0022] The server machine 130 may include a training data generator 131 capable of generating training data (e.g., a set of training inputs and a set of target outputs) for training ML models 160A-N. The training data may be based on images stored in a data store (e.g., data store 112A) that is or contains a domain-specific or organization-specific repository or database, or in the private portion of data store 112, and / or on images stored in a data store (e.g., data store 112B) that is or contains a publicly accessible repository or database, or in the publicly accessible portion of data store 112. For example, the training data generator 131 may generate training data for a supervised machine learning model (e.g., a supervised object detection model) based on images stored in data store 112B, images stored in the publicly accessible portion of data store 112, or images retrieved from a publicly accessible data store (not shown). In another example, the training data generator 131 may generate training data for a student machine learning model (e.g., a student object detection model) based on images stored in data store 112A, images stored in the private portion of data store 112, or images from a single private data store 112, and based on one or more outputs of the teacher object detection model, as described above. Further details regarding the generation of training data for the teacher object detection model and the student object detection model are provided with reference to Figure 2.

[0023] The server machine 140 may include a training engine 141. The training engine 141 may train machine learning models 160A-N using training data from the training set generator 131. Machine learning models 160A-N may refer to model artifacts created by the training engine 141 using training data that includes training inputs and corresponding target outputs (the correct answers for each training input). The training engine 141 may find patterns in the training data that map the training inputs to target outputs (the answers to be predicted) and provide machine learning models 160A-N that capture these patterns. Machine learning models 160A-N may consist of, for example, single-level linear or nonlinear operations (e.g., support vector machines (SVM)). A machine, or deep network, may be a machine learning model consisting of multiple levels of nonlinear operations. One example of a deep network is a neural network with one or more hidden layers, and such a machine learning model can be trained by adjusting the weights of the neural network according to, for example, a backpropagation learning algorithm. For convenience, the remainder of this disclosure refers to the implementation form as a neural network, but some implementation forms may employ an SVM or other type of learning machine instead of, or in addition to, a neural network. In some embodiments, training data may be obtained by a training data generator 131 hosted by a server machine 130. For example, a training engine 141 may obtain from the training set generator 131 first training data for training a teacher object detection model and second training data for training a student object detection model. Further details regarding training object detection models (e.g., models 160A-N) are provided with respect to Figure 2.

[0024] Server 150 may include an object detection engine 151 that provides one or more images as input to a trained machine learning model 160A-N in order to obtain one or more outputs. In some embodiments, as described above, one or more images are stored in the data store 112 or a private portion of the data store 112. For example, trained machine learning model 160A may be a trained teacher object detection model. In such an embodiment, the object detection engine 151 may provide one or more images as input to the trained machine learning model 160A in order to obtain one or more first outputs. A training data generator 131 may use one or more first outputs of machine learning model 160A to generate training data for training a student object detection model, according to embodiments provided herein. In another embodiment, trained machine learning model 160B may be a trained student object detection model. In such an example, the object detection engine 151 may provide one or more images 106 acquired by the computing device 102 as input to a trained machine learning model 160B to obtain one or more second outputs. The object detection engine 151 may use one or more second outputs to detect objects shown in one or more images 106 and to determine object data 108 associated with one or more detected objects. Further details regarding the object detection engine 151 are provided with respect to Figure 3.

[0025] In some implementations, the computing device 102, data store 112, and / or server machines 130-150 may be one or more computing devices (such as rack-mount servers, router computers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, and desktop computers), data stores (such as hard disks, memory, and databases), networks, software components, and / or hardware components, which may be used to enable object detection based on an image (e.g., image 106). Note that in some other implementations, the functionality of computing device 102, server machines 130, 140, and / or 150 may be provided by fewer machines. For example, in some implementations, server machines 130 and 140 may be integrated into a single machine, while in other implementations, server machines 130, 140, and 150 may be integrated into multiple machines. Furthermore, in some implementations, one or more of the server machines 130, 140, and 150 may be incorporated into computing device 102. Generally, the functions described in the implementations as being performed by computing device 102 and / or server machines 130, 140, and 150 may also be performed on one or more edge devices (not shown) and / or client devices (not shown), where appropriate. In addition, functionality attributed to a particular component may be performed by different or more components working together. Computing device 102 and / or server machines 130, 140, and 150 may also be accessed as services provided to other systems or devices through appropriate application programming interfaces.

[0026] Figure 2 is a block diagram of an exemplary training data generator 131 and an exemplary training engine 141 according to at least one embodiment. The training data generator 131 may include a teacher model training data generator 210 and a student model training data generator 220. The training engine 141 may include a teacher model training module 230 and a student model training module 232. As previously described, the training data generator 131 may reside in a server machine such as server machine 130 in Figure 1, which is either part of or separate from computing device 102. The training engine 141 may reside in server machine 130 or another server machine such as server machine 140, which is either part of or separate from computing device 102.

[0027] In some embodiments, the teacher model training data generator 210 may be configured to generate training data for training a teacher object detection model (e.g., Model 160A), and the student model training data generator 220 may be configured to generate training data for training a student object detection model (e.g., Model 160B). As shown in Figure 2, the training data generator 131 may be connected to a data store 250. The data store 250 may be configured to store data used by the teacher model training data generator 210 to generate training data for training the teacher object detection model. For example, the data store 250 may be configured to store one or more training images 252, and for each training image 252, training image region of interest (ROI) data 254, training image mask data 256, and / or training image object data 258 associated with the training image 252. In some embodiments, each training image 252 may represent an object associated with a particular class of a set of several different object classes. In an exemplary example, training image 252 may show a first object corresponding to the human class and a second object corresponding to the animal class (for example, the dog class).

[0028] The training image ROI data 254 can represent each region of each training image 252 that represents each object. In some embodiments, the training image ROI data 254 can correspond to bounding boxes or other bounding shapes (e.g., ellipsoidal, ellipsoidal, cylindrical, etc.) that represent the regions of the training image 252 that represent each object. According to the previous example, the training image ROI data 254 associated with the exemplary training image 252 may include a first bounding box that represents a first region of the training image 252 that represents a first object, and a second bounding box that represents a second region of the training image 252 that represents a second object.

[0029] The training image mask data 256 may refer to data (for example, a two-dimensional (2D) bit array) indicating whether one or more pixels (or groups of pixels) for each training image 252 correspond to a particular object. As in the previous example, the training image mask data 256 associated with the exemplary training image 252 may include an indication of a first group of pixels corresponding to a first object and an indication of a second group of pixels corresponding to a second object.

[0030] The training image object data 258 may refer to data indicating one or more characteristics associated with each object shown in each training image 252. In some embodiments, the training image object data 258 may include data indicating the class (i.e., one of several classes) associated with the shown object. For example, the training image object data 258 associated with an exemplary training image 252 may include data indicating that a first object shown in training image 252 is associated with a first class (e.g., the human class) and a second object shown in training image 252 is associated with a second class (e.g., the animal class). In additional or alternative embodiments, the training image object data 258 may include data indicating other characteristics associated with each shown object, such as the object's position (e.g., orientation) or shape.

[0031] In some embodiments, training image 252 may be included in a set of images that can be used to train an object detection model. For example, training image 252 may be included in a set of publicly accessible images that can be retrieved from a publicly accessible data store (e.g., data store 112B) or a publicly accessible portion of a data store (e.g., data store 112) and used to train an object detection model. In some embodiments, each of the image sets may be associated with image data, which may also be included in a publicly accessible data store or a publicly accessible portion of a data store. In some embodiments, each of the image sets, and the image data associated with each of the image sets, may be provided by one or more users of the object detection platform. For example, a user of the object detection platform may provide images showing one or more objects (i.e., via their respective client devices). The user may also provide ROI data, mask data, and object data indications associated with each object shown in the provided images (i.e., via the graphical user interface of their respective client devices). In another example, a first user of the object detection platform may provide an image showing one or more objects (i.e., via a first client device associated with the first user), and a second user may provide ROI data, mask data, and object data indications associated with each object shown in the image provided by the first user (i.e., via a graphical user interface of a second client device associated with the second user). In some embodiments, a data retrieval module 212 of the teacher model training data generator 210 may retrieve training images 252 to be used to train a teacher object detection model from a publicly accessible data store or a publicly accessible portion of the data store, according to embodiments provided herein.

[0032] In some embodiments, data store 250 may correspond to a publicly accessible data store, such as data store 112B, as described with respect to Figure 1. In other or similar embodiments, data store 250 may correspond to a publicly accessible portion of data store 112A. The data retrieval module 212 may retrieve training images 252, training image ROI data 254, training image mask data 256, and / or training image object data 258 from data store 250 (i.e., from data store 112A or data store 112B). In other or similar embodiments, data store 250 may correspond to data store 112 (e.g., data store 112A) that is accessible only via a private network and / or via an encryption mechanism. In such embodiments, the data retrieval module 212 may retrieve training images 252, training image ROI data 254, training image mask data 256, and / or training image object data 258 from a publicly accessible data store (not shown) and the data store. In such an embodiment, the data retrieval module 212 can retrieve training images 252, training image ROI data 254, training image mask data 256, and / or training image object data 258 from the data store 250.

[0033] The training data generator module 214 may generate training data to be used to train a training object detection model in response to the data extraction module 212 extracting training images 252, training image ROI data 254, training image mask data 256, and / or training image object data 258. In some embodiments, the training data may include a set of training inputs and a set of target outputs. The set of training inputs may include one or more training images 252 extracted by the data extraction module 212 as described above. In some embodiments, the training data generator module 214 may apply one or more image transformations to one or more training images 252 extracted by the data extraction module 212. For example, the training images 252 extracted by the data extraction module 212 may be associated with a certain amount of image noise. The training data generator module 214 may apply one or more image transformations to the extracted training images 252 to generate modified training images. The corrected training images may contain a different amount of image noise (e.g., less image noise) than the extracted training images 252. The training data generator module 214 may include the corrected training images in the set of training images 252 according to the embodiment described above. The set of target outputs may include training image ROI data 254, training image mask data 256, and / or training image object data 258. In response to having generated a set of training inputs and a set of target outputs, the training data generator module 214 may generate a mapping between the set of training inputs and the set of target outputs in order to generate the teacher model training data 272.

[0034] In some embodiments, the teacher model training data generator 210 may store the teacher model training data 272 in a data store 270. The data store 270 may be a domain-specific or organization-specific repository or database or a data store containing such a repository or database (e.g., data store 112A), or a private portion of data store 112, accessible to computing devices via a private network and / or through an encryption mechanism. The data store 270 may be accessible to the training data generator 131 and / or the training engine 141 according to the embodiment described with respect to Figure 1. In additional or alternative embodiments, the teacher model training data generator 210 may provide the generated mappings to the teacher model training module 230 of the training engine 141 for training a teacher object detection model.

[0035] In response to acquiring training data 272 (i.e., from the teacher model training data generator 210 or from the data store 270), the teacher model training engine 230 may use the training data 272 to train the teacher object detection model. In some embodiments, the teacher object detection model may be trained to detect one or more objects of a given class shown in a given input image, and for each detected object, predict mask data and / or ROI data associated with the detected object. In some embodiments, the teacher object detection model may also be trained to predict object data (e.g., object class, other characteristic data associated with the object, etc.) for each object detected in a given input image. In other or similar embodiments, the teacher object detection model may be trained to detect one or more objects of a single class shown in a given input image, and for each detected object, predict mask data, ROI data, object class data, and / or other object characteristic data associated with the detected object. In some embodiments, in response to training a teacher object detection model, the training engine 141 may store the trained teacher object detection model as the teacher model 274 in the data store 270. The student model training data generator 220 may use the trained teacher model 274 to generate training data for training the student object detection model, according to embodiments provided herein.

[0036] The data store 270 may store one or more training images 276 that should be used to train a student object detection model to detect objects associated with a target class. For example, the target class may correspond to a people class. In such an example, one or more training images 276 in the data store 270 may represent one or more objects associated with a people class. In some embodiments, each object shown in each training image 276 may be associated with a different characteristic (e.g., a person in a different environment, a person located in a different position, etc.). Thus, one or more training images 276 may be used to train a student object detection model to detect objects that are associated with a people class and also associated with a different characteristic. In another example, one or more training images 276 in the data store 270 may represent one or more objects that are not associated with a people class but correspond to one or more characteristics similar to those of objects associated with a people class (e.g., an object shown in training image 276 is associated with a similar location or environment to objects in a people class). Therefore, one or more training images 276 can be used to train a student object detection model to detect objects that correspond to one or more characteristics associated with the people class, but are not associated with the people class.

[0037] The teacher inference module 222 may retrieve one or more training images 276 from the data store 270 and provide one or more training images 276 as input to the trained teacher model 274. The teacher output module 224 may retrieve one or more outputs of the trained teacher model 274 and determine object data associated with each input training image 276 from one or more retrieved outputs. In some embodiments, the determined object data for each training image 274 may include output image ROI data 278, output image mask data 280, and / or output image characteristic data 282. The output image ROI data 278 associated with each training image 276 may be a region of each training image 276 that indicates a particular object. The output image mask data 280 may indicate mask data associated with a particular object. The output image characteristic data 282 may indicate one or more characteristics associated with a particular object, such as the object's class, object's location, or object's shape. In some embodiments, the teacher output module 224 may store output image ROI data 278, output image mask data 280, and / or output image characteristic data 282 in the data store 270.

[0038] The training data generator module 226 may generate training data to be used to train a student object detection model to detect one or more objects associated with a target class for a given input image. In some embodiments, the training data may include a set of training inputs and a set of target outputs. The set of training inputs may include one or more training images 276 provided as input to the trained teacher model 274 as described above. The set of target outputs may include at least output image mask data 280 determined from one or more outputs of the trained teacher model 274. In some embodiments, the set of target outputs may include output image mask data 280 and output image characteristic data 282.

[0039] In some embodiments, the training data generator module 226 may generate updated image characteristic data based on output image characteristic data 282 determined from one or more outputs of the trained teacher model 274. For example, the training image 276 provided as input to the trained teacher model 274 may represent a first object associated with a first class (e.g., the people class) and a second object associated with a second class (e.g., the animals class). The teacher output module 224 may obtain one or more outputs from the trained teacher model 274 showing the first and second objects detected in a given input image, and output image characteristic data 282 indicating that the first object is associated with a first class and the second object is associated with a second class. The training data generator module 226 may determine whether the first class and / or the second class correspond to a target class and, based on that determination, may generate updated image object data. For example, if the target class is the people class, the training data generator module 226 may determine that a first class corresponds to the target class and generate updated image characteristic data to indicate that a first object corresponds to the target class. The training data generator module 226 may also determine that a second class does not correspond to the target class and generate updated image characteristic data to indicate that a second object does not correspond to the target class. In some embodiments, the training data generator module 226 may include updated object data in the set of target outputs instead of output image characteristic data 282.

[0040] In some embodiments, the set of target outputs may further include image ground truth data 284. Image ground truth data 284 may indicate regions of each training image 276 that contain objects detected by the trained teacher model 274. For example, ground truth data 284 may include one or more bounding box indications related to the training image 276 obtained from an accepted bounding box authority entity or user of a computing device or object detection platform. In some embodiments, ground truth data 284 may be obtained from an accepted bounding box authority entity or user before or after one or more training images 276 are provided as input to the trained teacher model 274. In exemplary examples, the image ground truth data 284 may correspond to the output image ROI data 278, except that the bounding box of the image ground truth data 284 may more accurately identify regions of image 276 representing specific objects than the bounding box of the output image ROI data 278. In some additional or alternative embodiments, the image ground truth data 284 may represent the classes of objects detected by the trained teacher model 274. In some embodiments, the target output set may include the classes of objects represented by the image ground truth data 284 instead of the classes of objects represented by the output image characteristic data 282, as described above. In some embodiments, the training data generation module 226 may generate updated image characteristic data based on the object classes represented by the image ground truth data 284, according to the embodiments described earlier.

[0041] In response to generating a set of training inputs and a set of target outputs for each training image 276, the training data generator module 226 may generate a mapping between the training input set and the set of target outputs to generate student model training data 286. In some embodiments, the student model training data generator 220 may store the student model training data 286 in the data store 270. In other or similar embodiments, the student model training data generator 220 may transmit the student model training data 286 to the training engine 141. In response to receiving the training data 286 (i.e., from the student model training data generator 220 or from the data store 270), the student model training module 232 may use the training data 286 to train a student object detection model. The student object detection model may be trained to detect one or more objects of a target class shown in a given input image for a given input, and for each detected object, predict mask data, ROI data, and / or characteristic data associated with the detected object. In some embodiments, the training engine 141 may provide a trained student object detection model to an object detection engine, such as the object detection engine 151 in Figure 1.

[0042] Figure 3 is a block diagram of an exemplary object detection engine 310 according to at least one embodiment. In some embodiments, the object detection engine 310 may correspond to the object detection engine 151 described with respect to Figure 1. As shown in Figure 3, the object detection engine 310 may include an input image component 312, an object data component 314, a model head component 316, and / or a model update component 318. In some embodiments, the object detection engine 310 may be coupled to a memory 320. In some embodiments, the object detection engine 310 may reside in a computing device 102. In such embodiments, the memory 320 may correspond to the memory 104 described with respect to Figure 1. In other or similar embodiments, the object detection engine 310 may reside in a server 150. In such embodiments, the memory 320 may correspond to memory in a data store (e.g., data store 112), memory 104, or memory in another memory device associated with the system 100.

[0043] The input image component 312 may be configured to acquire an image (for example, image 106) and to provide the acquired image as input to a trained object detection model 322 stored in memory 320. In some embodiments, the trained object detection model 322 may correspond to a student object detection model trained by the training engine 141 using training data generated by the training data generator 131, as described with respect to Figures 1 and 2. In other or similar embodiments, the trained object detection model 322 may correspond to another trained object detection model that is not trained by the training engine 141 using training data generated by the training data generator 131.

[0044] As described with respect to Figure 1, the computing device 102 may, in some embodiments, be a computing device of a cloud computing platform. In such embodiments, the computing device 102 may be coupled to one or more edge devices (e.g., edge device 330), as shown in Figure 3, which are each coupled to one or more endpoint devices (e.g., endpoint devices 332A-N, collectively and individually referred to herein as endpoint device 332). In some embodiments, one or more audio-visual and / or sensor components of endpoint device 332 may generate an image 106, as described above, and transmit the image 106 to edge device 330 (e.g., via a network). Edge device 330 may transmit the received image 106 to computing device 102 (e.g., via network 110). In such embodiments, computing device 102 may transmit the image 106 to input image component 312 (e.g., via network 110 or the bus of computing device 102). In other or similar embodiments, computing device 102 may be an edge device 330 or a component thereof, as also described above. In such embodiments, edge device 330 may receive image 106 from endpoint device 332 (e.g., via a network) and transmit image 106 to input image component 312 (e.g., via network 110 or the bus of computing device 102). In other or similar embodiments, computing device 102 may be one or more endpoint devices 332A-N or a component thereof, as previously described. In such embodiments, one or more endpoint devices 332A-N may generate image 106 and transmit image 106 to input image component 312 (e.g., via network, network 110, or the bus of computing device 102).

[0045] In response to receiving image 106, the input image component 312 may, in some embodiments, provide image 106 as input to the trained object detection model 322. In other or similar embodiments, the input image component 312 may apply one or more image transformations (for example, to reduce the amount of noise contained in image 106) as described above to generate a modified image and provide the modified image as input to the trained object detection model 322. The object data component 314 may acquire one or more outputs of the trained object detection model 322 and determine object data 108 based on one or more acquired outputs. In some embodiments, the object data 108 determined based on one or more acquired outputs may correspond to one or more objects detected in a given input image 106 (or modified input image). For example, the object data 108 may include indications of regions of image 106 (e.g., bounding boxes) containing the detected objects. In some additional or alternative embodiments, the object data 108 may further include mask data related to the detected object. In some additional or alternative embodiments, the object data 108 may further include data indicating the class and / or one or more characteristics related to the detected object.

[0046] In some embodiments, one or more outputs of the trained object detection model 322 may include indications of multiple regions of image 106 and indications of confidence levels for each region, where the detected object is located. The object data component 314 may determine that a particular region of image 106 contains a detected object by determining that the confidence level associated with that particular region of image 106 satisfies a confidence criterion level (for example, the confidence level exceeds a threshold). In response to the determination that a particular region of image 106 satisfies a confidence criterion level, the object data component 314 may include the indication of the particular region of image 106 in memory 320 along with the object data 108. In additional or alternative embodiments, one or more outputs of the trained object detection model 322 may include multiple sets of mask data and indications of confidence levels for each set of mask data associated with a detected object. In response to the determination that a particular set of mask data satisfies a confidence criterion level, the object data component 314 may include the indication of the particular mask data in memory 320 along with the object data 108. In additional or alternative embodiments, one or more outputs of the trained object detection model 322 may include multiple classes and / or characteristics, and an indication of the level of confidence for which each class and / or characteristic corresponds to an object detected. In response to determining that a particular class and / or characteristic satisfies a level of confidence criterion, the object data component 314 may include the indication of that particular class and / or characteristic in memory 320 along with the object data 108.

[0047] In some embodiments, the object data component 314 may determine, based on the object data 108, whether an object detected in a given input image 106 corresponds to a target class. For example, in some embodiments, the object data 108 may indicate a class associated with the detected object, as described above. The object data component 312 may compare the indicated class with a target class to determine whether the detected object corresponds to a target class. In another embodiment, the object data 108 may include data indicating whether the detected object corresponds to a target class, according to the embodiments described above. In such an embodiment, the object data component 312 may determine, based on the included data, whether the detected object corresponds to a target class. In some embodiments, the object data component 314 may update the object data 108 to include an indication of whether the detected object corresponds to a target class. The object data component 314 may transmit the object data 108 to the computing device 102. In some embodiments, the object data component 312 may, as an addition or alternative, transmit a notification to the computing device 102 indicating whether the detected object corresponds to a target class.

[0048] As described above, in some embodiments, the trained object detection model 322 (for example, the trained student object detection model in Figure 2) may be trained to predict ROI data, image mask data, and / or image characteristic data associated with a given input image. In such embodiments, the trained object detection model 322 may be a multi-head model, where each head of the multi-head model is used to predict a specific type of data associated with an object detected in a given input image. For example, the trained object detection model 322 may include a first head corresponding to predicting ROI data associated with the detected object, a second head corresponding to predicting mask data associated with the detected object, and / or a third head corresponding to predicting characteristic data associated with the detected object, such as the class of the detected object. As described above, in some embodiments, a considerable number of training images (e.g., training image 276) may be used to train the object detection model. For example, in some systems, hundreds, thousands, or in some cases, millions of images may be used to train the object detection model. In view of the above, a multi-head object detection model 322 trained on a considerable number of images may, in some embodiments, consume a considerable amount of system resources (e.g., memory space, processing resources, etc.). In some embodiments, the object detection engine 310 may remove one or more heads (e.g., mask heads) of the trained multi-head object detection model 322 before and / or during inference in order to reduce the amount of system resources consumed by the object detection model 322.

[0049] Figure 4A shows an exemplary trained multi-head object detection model 322 according to at least one embodiment. As shown in Figure 4A, model 322 may include at least an ROI head 412 and a mask head 414. Model 322 may be similar to a neural network, as shown in Figure 4A, but it should be noted that embodiments of the present disclosure may be applied to any type of machine learning model. An input image component 312 may provide an image 106 as input to model 322 according to the embodiments described above. The input image 106 may be provided to both the ROI head 412 and the mask head 414 of model 322. Model 322 may provide one or more outputs based on a given input image 106, as described above. In some embodiments, the provided outputs may include an ROI head output 416 and a mask head output 418. The ROI head output 416 may be provided based on inference performed according to the ROI head 412. The mask head output 418 may be provided based on the inference performed according to the mask head 414.

[0050] Referring again to Figure 3, in some embodiments, the model head component 316 and model update component 318 of the object detection engine 310 may remove mask heads from the object detection model 322. For example, the model head component 316 may identify one or more heads (e.g., mask head 414) of the model 322 that correspond to providing a specific output related to a given input image (e.g., predicting mask data). In response to the model head component 316 identifying the corresponding heads of the model 322, the model update component 318 may update the model 322 to remove one or more identified heads. Figure 4B shows an updated trained object detection model 324 updated to remove mask head 414, according to at least one embodiment. As shown in Figure 4B, the model update component 318 may remove mask head 414 from the model 322 to produce the updated model 324. Therefore, in some embodiments, Model 324 may provide an ROI head output 416 and may not provide a mask head output 418. Figures 3 and 4A to 4B include embodiments relating to the removal of the mask head 414 from Model 322, but it should be noted that embodiments of this disclosure may be applied to the removal of any head of Model 322.

[0051] Referring again to Figure 3, in some embodiments, the object detection engine 310 may transmit the updated object detection model 324 to the computing device 102 (for example, via the network 110 or the bus of the computing device 102). As described above, the computing device 102 may, in some embodiments, be a cloud computing platform or a component thereof. In such embodiments, the computing device 102 may transmit the updated object detection model 324 to the edge device 330 (for example, via the network 110). In some embodiments, the edge device 330 may use the updated object detection model 324 to perform object detection based on images 106 generated by endpoint devices 332A-N. In other or similar embodiments, the edge device 330 may transmit the updated object detection model 324 to endpoint devices 332A-N. As also described above, the computing device 102 may be the edge device 330 or a component thereof. In such embodiments, the edge device 330 may use the updated object detection model 324 to perform object detection and / or transmit the updated object detection model 324 to the endpoint device 332A. As also described above, the computing device 102 may be one or more endpoint devices 332A-N or components thereof. In such embodiments, one or more of the endpoint devices 332A-N may use the updated object detection model 324 to perform object detection according to the embodiments described above.

[0052] Figures 5A-5B and 6 are flowcharts of exemplary methods 500, 550, and 600 relating to training an object detection model, respectively, according to at least several embodiments. In at least one embodiment, methods 500, 550, and / or 600 may be implemented by computing device 102, server machine 130, server machine 140, server machine 150, one or more edge devices, one or more endpoint devices, or any other computing device, or a combination of multiple computing devices. Methods 500, 550, and / or 600 may be implemented by one or more processing units (e.g., CPU and / or GPU) which may include (or communicate with) one or more memory devices. In at least one embodiment, methods 500, 550, and / or 600 may be implemented by multiple processing threads (e.g., CPU threads and / or GPU threads), each thread executing one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, processing threads implementing methods 500, 550, and / or 600 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, processing threads implementing methods 500, 550, and / or 600 may run asynchronously from one another. Various operations of methods 500, 550, and / or 600 may be performed in a different order than the order shown in Figures 5A-5B and 6. Some operations of the methods may be performed in parallel with other operations. In at least one embodiment, one or more operations shown in Figures 5A-5B and 6 may not always be performed.

[0053] Figure 5A shows a flowchart of an exemplary method 500 for training a machine learning model to detect objects of a target class, according to at least one embodiment. In some embodiments, one or more operations of method 500 may be carried out by one or more components or modules of the training data generator 131 as described herein. A processing unit carrying out method 500 may, in block 510, identify a first set of images containing multiple objects of multiple classes. In some embodiments, the processing unit may retrieve the first set of images from the data store 270, as described previously.

[0054] In block 512, a processing unit implementing method 500 may provide a first set of images as input to a first machine learning model. The first machine learning model may be a trained object detection model trained to detect one or more objects of a plurality of classes shown in a given input image for a given input image. The trained object detection model may also be trained to predict, for each of the one or more detected objects, mask data and, in some embodiments, ROI data associated with each detected object. In additional or alternative embodiments, the trained object detection model may be trained to predict, for each detected object, a specific class of a plurality of classes associated with each detected object. In block 514, a processing unit implementing method 500 may determine object data associated with a first set of images from one or more outputs of the first machine learning model. The object data for each image in the first set of images may include mask data associated with each object detected in the respective image.

[0055] In block 516, a processing unit implementing method 500 may train a second machine learning model to detect objects of a target class in a second set of images using a first set of images and a portion of object data determined from one or more outputs of a first machine learning model. The second machine learning model may be a student object detection model. The processing unit may train the student object detection model using a training input and a target output. The training input may include a first set of images (i.e., those provided as input to the trained student object detection model). The target output may include mask data associated with each detected object in the first set of images, which is included in one or more outputs of the student object detection model. The target output may also include an indication of whether the class associated with each detected object in the first set of images corresponds to the target class. In some embodiments, the processing unit may determine whether a particular class associated with each detected object in the first set of images corresponds to the target class. The target output may include an indication of whether a particular class corresponds to the target class. In another embodiment, the target output may include a specific class designation associated with each object detected in the first set of images.

[0056] In some embodiments, the target output may also include ground truth data associated with each object detected in the first set of images. As described above, the ground truth data associated with each detected object may indicate the region of the image containing each detected object. The processing unit may identify the ground truth data using a database (e.g., in data store 270) containing indications of one or more ROIs (e.g., bounding boxes) associated with the image, according to the embodiments described earlier. Each of the ROIs included in the database may be provided by the platform's accepted ROI authority or by a user. In some embodiments, the processing unit may also use the database to identify the class associated with each object detected in the first set of images.

[0057] In some embodiments, the trained second machine learning model may be a multi-head machine learning model, as described above. In some embodiments, the processing unit may identify one or more heads of the second machine learning model corresponding to predicting mask data for a given input image, and may update the second machine learning model to remove one or more identified heads. In some embodiments, the processing unit may provide a second set of images as input to the second machine learning model and obtain one or more outputs of the second machine learning model. From one or more obtained outputs, the processing unit may determine additional object data associated with each of the second set of images. In some embodiments, the additional object data may include indications of regions of each image (e.g., bounding boxes) containing the detected objects in each image, and classes associated with the detected objects.

[0058] Figure 5B shows a flowchart of an exemplary method 550 using a machine learning model trained to detect objects of a target class, according to at least one embodiment. In some embodiments, one or more operations of method 550 may be carried out by one or more components or modules of the object detection engine 151, as described herein. A processing unit carrying out method 550 may, in block 552, provide a set of current images as input to the first machine learning model. In some embodiments, the set of current images may be generated by an audio-visual component (e.g., a camera) in or coupled to an endpoint device, edge device, or server, as described above.

[0059] A first machine learning model can be trained to detect objects of a target class in a given set of images. In some embodiments, the first machine learning model may correspond to a student object detection model as described above. In some embodiments, the first machine learning model can be trained according to the embodiments described earlier. For example, the first machine learning model can be trained using a training input, which includes a set of training images, and a target output for the training input. The target output may include ground truth data related to each object shown in each training image, for each training image in the set of training images. The ground truth data may indicate the region of each training image containing each object. In some embodiments, the ground truth data can be obtained using a database containing indications for one or more bounding boxes related to the set of training images. Each of the one or more bounding boxes may be provided by the platform's accepted bounding box authority entity and / or user.

[0060] The target output may also include mask data associated with each object shown in each training image. The mask data may be obtained based on one or more outputs of a second machine learning model. In some embodiments, the second machine learning model may correspond to a supervising model as described herein. For example, a set of training images may be provided as input to the second machine learning model. The second machine learning model may be trained to detect one or more objects of at least one of several classes shown in a given input image for a given input image, and to predict at least mask data associated with each of the one or more detected objects. The object data may be determined from one or more outputs of the second machine learning model as described above. The object data for each training image may include mask data associated with each object detected in the respective image. The target output may also include an indication, according to the embodiments described earlier, whether the class associated with each object shown in each training image corresponds to a target class.

[0061] A processing unit implementing Method 550 may, in block 554, acquire one or more outputs of the first machine learning model. A processing unit implementing Method 550 may, in block 556, determine object data associated with each of the current set of images based on one or more acquired outputs. In some embodiments, the determined object data for each image in the current set of images may include an indication of the region of each image containing the detected object in each image, and an indication of whether the detected object corresponds to a target class. In some embodiments, the object data may further include mask data associated with the detected object in each image. In some embodiments, the object data associated with each of the set of images may be determined by extracting one or more sets of object data from one or more outputs of the first machine learning model. Each of the one or more sets of object data may be associated with a confidence level, where the object data corresponds to the detected object in each image. The processing unit may determine whether the confidence level associated with each set of object data satisfies a confidence criterion level (e.g., exceeds a confidence level threshold). In response to determining that the level of confidence associated with each set of object data satisfies the level of confidence criteria, the processing unit may determine that the set of object data corresponds to a detected object.

[0062] Figure 6 shows a flowchart of an exemplary method, according to at least one embodiment, for training a machine learning model and updating the trained machine learning model to remove the mask head. In some embodiments, one or more operations of Method 600 may be carried out by one or more components or modules of the training data generator 131 and / or training engine 141 as described herein. A processing unit carrying out Method 600 may identify and generate training data for a machine learning model in block 610.

[0063] In block 612, a processing unit implementing method 600 may generate a training input which includes an image showing an object. In block 614, a processing unit implementing method 600 may generate a target output for the training input. The target output may include a bounding box associated with the indicated object, mask data associated with the indicated object, and a class indication associated with the indicated object. In some embodiments, the processing unit may generate the target output by providing the image showing the object as input to an additional machine learning model which is trained to detect one or more objects shown in a given input image and to predict at least mask data associated with each of the one or more detected objects. In some embodiments, the additional machine learning model is further trained to predict the class associated with each detected object. The processing unit may use the additional machine learning model to obtain mask data (and, in some embodiments, class indications) associated with the objects shown in the training input image.

[0064] In additional or alternative embodiments, the processing unit may generate the target output by acquiring ground truth data associated with the images. As described above, the ground truth data may include bounding boxes associated with the indicated objects and may be acquired from a database that stores bounding box indications associated with the objects shown in the set of images. The bounding box indications may be provided by the platform's accepted bounding box authority entity or by a user.

[0065] In block 616, a processing unit implementing method 600 may provide training data for training a machine learning model with respect to (i) a set of training inputs including a generated training input, and (ii) a set of target outputs including a generated target output. In block 618, a processing unit implementing method 600 may identify one or more heads of the trained machine learning model corresponding to predicting mask data for a given input image. In block 620, a processing unit implementing method 600 may update the trained machine learning model to remove one or more identified heads.

[0066] In some embodiments, a processing unit implementing Method 600, or other processing units, may provide a set of images as input to an updated, trained machine learning model and obtain one or more outputs of the updated, trained machine learning model. From one or more outputs, the processing unit may determine object data associated with each of the sets of images. The object data may include an indication of the region of each image containing the object detected in each image, and a class associated with the detected object. In some embodiments, a processing unit implementing Method 600, or other processing units, may transmit the updated, trained machine learning model over a network to at least one of an edge device or an endpoint device.

[0067] Reasoning and training logic Figure 7A shows the inference and / or training logic 715 used to perform the inference and / or training operations associated with one or more embodiments. Further details regarding the inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B.

[0068] In at least one embodiment, the inference and / or training logic 715 may include, but not limited to, code and / or data storage 701 for storing forward and / or output weights and / or input / output data, and / or other parameters, for constituting neurons or layers of a neural network used for training and / or inference in one or more embodiments. In at least one embodiment, the training logic 715 may include, or be coupled to, code and / or data storage 701 for storing graph code or other software for controlling timing and / or sequence, and weight and / or other parameter information should be loaded into the code and / or data storage 701 to constitute logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, the code, such as graph code, loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, the code and / or data storage 701 stores the weight parameters and / or input / output data of each layer of the neural network being trained or used in conjunction with one or more embodiments during the forward propagation of input / output data and / or weight parameters during training and / or inference using the embodiments of one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 701 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0069] In at least one embodiment, any portion of the code and / or data storage 701 may be inside or outside one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or code and / or data storage 701 may be cache memory, dynamic randomly addressable memory ("DRAM"), static randomly addressable memory ("SRAM"), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the selection of whether the code and / or code and / or data storage 701 is inside or outside the processor, or whether it consists of DRAM, SRAM, flash, or some other type of storage, may depend on available storage, on-chip vs. off-chip, latency requirements of the training and / or inference functions being performed, batch size of data used in neural network inference and / or training, or any combination of these factors.

[0070] In at least one embodiment, the inference and / or training logic 715 may include code and / or data storage 705 for storing backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network used to train and / or infer in one or more embodiments, but not limited to. In at least one embodiment, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of the neural network used to train or in conjunction with one or more embodiments during backward propagation of input / output data and / or weight parameters during training and / or inference using one or more embodiments. In at least one embodiment, the training logic 715 may include, or be coupled to, code and / or data storage 705 for storing graph code or other software for controlling timing and / or sequence, and weight and / or other parameter information should be loaded into the code and / or data storage 705 to constitute logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, the code, such as graph code, loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, any portion of the code and / or data storage 705 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 705 may be inside or outside one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage.In at least one embodiment, the selection of whether the code and / or data storage 705 is, for example, internal or external to the processor, or whether it consists of DRAM, SRAM, flash or some other type of storage, may depend on the available storage, on-chip vs. off-chip, latency requirements of the training and / or inference functions being performed, the batch size of the data used in the neural network inference and / or training, or any combination of these factors.

[0071] In at least one embodiment, the code and / or data storage 701 and the code and / or data storage 705 may be separate storage structures. In at least one embodiment, the code and / or data storage 701 and the code and / or data storage 705 may be the same storage structure. In at least one embodiment, the code and / or data storage 701 and the code and / or data storage 705 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of the code and / or data storage 701 and the code and / or data storage 705 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0072] In at least one embodiment, the inference and / or training logic 715 may include, but not limited to, one or more arithmetic logic units ("ALUs") 710, including integer and / or floating-point units, for performing logical and / or mathematical operations that are at least partially based on or shown by training and / or inference code (e.g., graph code), the result of which activations (e.g., output values ​​from layers or neurons in a neural network) stored in activation storage 720, and these activations are functions of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, the activation stored in the activation storage 720 is generated according to linear algebra and / or matrix-based mathematics performed by (one or more) ALU 710 in response to the execution of an instruction or other code, and the weight values ​​stored in the code and / or data storage 705 and / or code and / or data storage 701 are used as operands along with other values ​​such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in the code and / or data storage 705 or code and / or data storage 701, or in other on-chip or off-chip storage.

[0073] In at least one embodiment, the (one or more) ALU 710 are contained within one or more processors or other hardware logic devices or circuits, but in another embodiment, the (one or more) ALU 710 may be outside of the processors or other hardware logic devices or circuits (e.g., coprocessors) that use them. In at least one embodiment, the ALU 710 may be contained within an execution unit of a processor, or otherwise contained within a bank of ALUs accessible by execution units of a processor, either within the same processor or distributed across different types of processors (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, the code and / or data storage 701, the code and / or data storage 705, and the activation storage 720 may be on the same processor or other hardware logic device or circuit, but in another embodiment, they may be in different processors or other hardware logic devices or circuits, or in some combination of the same processor or other hardware logic devices or circuits and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of the activated storage 720 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be fetched and / or processed using the processor's fetch, decode, schedule, execute, retirement, and / or other logic circuits.

[0074] In at least one embodiment, the activated storage 720 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the activated storage 720 may be entirely or partially located within or outside one or more processors or other logic circuits. In at least one embodiment, the selection of whether the activated storage 720 is, for example, inside or outside the processor, or whether it consists of DRAM, SRAM, flash, or some other type of storage, may depend on the available storage, on-chip vs. off-chip, latency requirements of the training and / or inference functions being performed, batch size of data used in neural network inference and / or training, or any combination of these factors. In at least one embodiment, the inference and / or training logic 715 shown in Figure 7A may be used in conjunction with an application-specific integrated circuit ("ASIC"), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore®, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the inference and / or training logic 715 shown in Figure 7A may be used in conjunction with other hardware, such as a central processing unit ("CPU"), a graphics processing unit ("GPU"), or a field-programmable gate array ("FPGA").

[0075] Figure 7B shows the inference and / or training logic 715 in at least one or more embodiments. In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, hardware logic in which computational resources are dedicated or, otherwise, used only in conjunction with weight values ​​or other information corresponding to one or more layers of neurons in a neural network. In at least one embodiment, the inference and / or training logic 715 shown in Figure 7B may be used in conjunction with application-specific integrated circuits (ASICs), such as a Tensorflow® processing unit from Google, an Inference Processing Unit (IPU) from Graphcore®, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the inference and / or training logic 715 shown in Figure 7B may be used in conjunction with other hardware, such as a central processing unit (CPU) hardware, a graphics processing unit (GPU) hardware, or a field-programmable gate array (FPGA). In at least one embodiment, the inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which may be used to store code (e.g., graph code), weight values, and / or other information including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In at least one embodiment shown in Figure 7B, each of the code and / or data storage 701 and code and / or data storage 705 is associated with a dedicated computing resource, such as compute hardware 702 and compute hardware 706, respectively. In at least one embodiment, each of the compute hardware 702 and compute hardware 706 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on the information stored in the code and / or data storage 701 and code and / or data storage 705, respectively, and the results are stored in the activation storage 720.

[0076] In at least one embodiment, each of the code and / or data storages 701 and 705 and the corresponding compute hardware 702 and 706 correspond to different layers of a neural network, thereby the activation resulting from one “storage / compute pair 701 / 702” of the code and / or data storage 701 and compute hardware 702 is provided as input to the “storage / compute pair 705 / 706” of the code and / or data storage 705 and compute hardware 706 in order to mirror the conceptual organization of the neural network. In at least one embodiment, each of the storage / compute pairs 701 / 702 and 705 / 706 may correspond to two or more neural network layers. In at least one embodiment, additional storage / compute pairs (not shown) may be included in the inference and / or training logic 715 after or in parallel with the storage / compute pairs 701 / 702 and 705 / 706.

[0077] Data center Figure 8 shows an exemplary data center 800 in which at least one embodiment may be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0078] In at least one embodiment, as shown in Figure 8, the data center infrastructure layer 810 may include a resource orchestrator 812, grouped computing resources 814, and node computing resources ("node CRs") 816(1) to 816(N), where "N" represents any positive integer. In at least one embodiment, the node CRs 816(1) to 816(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and cooling modules, etc. In at least one embodiment, one or more nodes CR from among nodes CR816(1) to 816(N) may be servers having one or more of the computing resources described above.

[0079] In at least one embodiment, the grouped computing resources 814 may include separate groupings of node CRs housed in one or more racks (not shown), or many racks housed in a data center at various geographical locations (also not shown). A separate grouping of node CRs within the grouped computing resources 814 may include grouped compute resources, network resources, memory resources, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs, including CPUs or processors, may be grouped in one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0080] In at least one embodiment, the resource orchestrator 812 may constitute or otherwise control one or more nodes CR816(1) to 816(N) and / or a grouped computing resource 814. In at least one embodiment, the resource orchestrator 812 may include a software design infrastructure ("SDI") management entity for the data center 800. In at least one embodiment, the resource orchestrator may include hardware, software, or any combination thereof.

[0081] In at least one embodiment, as shown in Figure 8, the framework layer 820 includes a job scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, the framework layer 820 may include a framework for supporting software 832 of the software layer 830 and / or one or more applications 842 of the application layer 840. In at least one embodiment, the software 832 or (one or more) applications 842 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. In at least one embodiment, the framework layer 820 may be a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which can utilize the distributed file system 828 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 822 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 800. In at least one embodiment, the configuration manager 824 may be able to configure different layers, such as the software layer 830 and the framework layer 820, which includes Spark and a distributed file system 828 to support large-scale data processing. In at least one embodiment, the resource manager 826 may be able to manage clustered or grouped computing resources that are mapped or allocated to support the distributed file system 828 and the job scheduler 822. In at least one embodiment, the clustered or grouped computing resources may include a grouped computing resource 814 in the data center infrastructure layer 810.In at least one embodiment, the resource manager 826 may work in conjunction with the resource orchestrator 812 to manage these mapped or allocated computing resources.

[0082] In at least one embodiment, the software 832 contained within the software layer 830 may include software used by nodes CR816(1) to 816(N), grouped computing resources 814, and / or at least a portion of the distributed file system 828 of the framework layer 820. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0083] In at least one embodiment, one or more applications 842 contained within the application layer 840 may include one or more types of applications used by nodes CR816(1) to 816(N), grouped computing resources 814, and / or at least a portion of the distributed file system 828 of the framework layer 820. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive compute, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0084] In at least one embodiment, any of the configuration manager 824, resource manager 826, and resource orchestrator 812 may implement any number and type of self-correcting actions based on any amount and type of data obtained in any technically feasible manner. In at least one embodiment, the self-correcting actions may relieve the data center operator of data center 800 of having to determine potentially faulty configurations and potentially avoid underutilized and / or underperforming portions of the data center.

[0085] In at least one embodiment, the data center 800 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by computing weight parameters according to a neural network architecture using the software and computing resources described above with respect to the data center 800. In at least one embodiment, a trained machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to the data center 800 by using weight parameters computed through one or more training techniques described herein.

[0086] In at least one embodiment, the data center may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, or other hardware to perform training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as services that enable users to train information or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0087] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, the inference and / or training logic 715 may be used in the system of Figure 8 for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0088] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0089] Computer system Figure 9 is a block diagram showing an exemplary computer system, which may be a system 900 having interconnected devices and components, a system-on-a-chip (SOC), or any combination thereof, formed together with a processor that may include an execution unit for executing instructions, according to at least one embodiment. In at least one embodiment, the computer system 900 may include components such as a processor 902 for employing an execution unit including logic for implementing algorithms for process data, as described herein, but not limited to the embodiments described herein. In at least one embodiment, the computer system 900 may include a processor such as the PENTIUM® processor family, Xeon®, Itanium®, XScale®, and / or StrongARM®, Intel® Core®, or Intel® Nervana® microprocessors, available from Intel Corporation in Santa Clara, California, but other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.) may also be used. In at least one embodiment, the computer system 900 may run a version of the WINDOWS® operating system available from Microsoft Corporation in Redmond, Washington, but other operating systems (e.g., UNIX® and Linux®), embedded software, and / or graphical user interfaces may also be used.

[0090] The embodiments may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor ("DSP"), a system-on-a-chip, a network computer ("NetPC"), a set-top box, a network hub, a wide-area network ("WAN") switch, or any other system capable of implementing one or more instructions according to at least one embodiment.

[0091] In at least one embodiment, the computer system 900 may include, but is not limited to, a processor 902, which may include, but is not limited to, one or more execution units 908 for performing machine learning model training and / or inference by the techniques described herein. In at least one embodiment, the computer system 900 is a single-processor desktop or server system, but in another embodiment, the computer system 900 may be a multi-processor system. In at least one embodiment, the processor 902 may include, but is not limited to, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor implementing a combination of instruction sets, or any other processor device, such as a digital signal processor. In at least one embodiment, the processor 902 may be coupled to a processor bus 910, which may transmit data signals between the processor 902 and other components in the computer system 900.

[0092] In at least one embodiment, the processor 902 may include, but is not limited to, a level 1 ("L1") internal cache memory ("cache") 904. In at least one embodiment, the processor 902 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory may reside outside the processor 902. Other embodiments may also include a combination of both internal and external caches, depending on the specific implementation and requirements. In at least one embodiment, the register file 906 may store different types of data in various registers, including, but is not limited to, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0093] In at least one embodiment, but not limited to, an execution unit 908 containing logic for performing integer and floating-point arithmetic is also present in the processor 902. In at least one embodiment, the processor 902 may also include a microcode ("u-code") read-only memory ("ROM") for storing microcode for several macro instructions. In at least one embodiment, the execution unit 908 may include logic for handling a packed instruction set 909. In at least one embodiment, by including the packed instruction set 909, along with the associated circuit elements for executing the instructions, in the instruction set of the general-purpose processor 902, arithmetic used by many multimedia applications can be performed using packed data in the general-purpose processor 902. In one or more embodiments, many multimedia applications can be accelerated and run more efficiently by using the full width of the processor's data bus to perform arithmetic on packed data, which may eliminate the need to transfer smaller units of data across the processor's data bus to perform one or more arithmetic operations, one data element at a time.

[0094] In at least one embodiment, the execution unit 908 may also be used in a microcontroller, embedded processor, graphics device, DSP, and other types of logic circuits. In at least one embodiment, the computer system 900 may include, but is not limited to, memory 920. In at least one embodiment, memory 920 may be implemented as a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or other memory device. In at least one embodiment, memory 920 may store (one or more) instructions 919 and / or data 921, which are represented by data signals that can be executed by the processor 902.

[0095] In at least one embodiment, a system logic chip may be coupled to a processor bus 910 and memory 920. In at least one embodiment, the system logic chip may include, but is not limited to, a memory controller hub ("MCH") 916, and the processor 902 may communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 may provide a high-bandwidth memory path 918 to memory 920 for instruction and data storage, as well as for the storage of graphics commands, data, and textures. In at least one embodiment, the MCH 916 may direct data signals between the processor 902, memory 920, and other components in the computer system 900, and bridge data signals between the processor bus 910, memory 920, and system I / O 922. In at least one embodiment, the system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH916 may be coupled to memory 920 through a high-bandwidth memory path 918, and the graphics / video card 912 may be coupled to the MCH916 through an Accelerated Graphics Port ("AGP") interconnect 914.

[0096] In at least one embodiment, the computer system 900 may use a system I / O 922, which is a proprietary hub interface bus for coupling the MCH 916 to the I / O controller hub ("ICH") 930. In at least one embodiment, the ICH 930 may provide direct connectivity to several I / O devices via a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus for connecting peripherals to memory 920, a chipset, and a processor 902. Examples may include, but are not limited to, an audio controller 929, a firmware hub ("Flash BIOS") 928, a wireless transceiver 926, data storage 924, a legacy I / O controller 923 including a user input and keyboard interface 925, a serial expansion port 927 such as a Universal Serial Bus ("USB"), and a network controller 934. The data storage 924 may comprise a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0097] In at least one embodiment, Figure 9 shows a system including interconnected hardware devices or “chips,” while in other embodiments, Figure 9 may show an exemplary system-on-a-chip (“SoC”). In at least one embodiment, devices may be interconnected by proprietary interconnects, standard interconnects (e.g., PCIe), or any combination thereof. In at least one embodiment, one or more components of the computer system 900 are interconnected using a compute express link (CXL) interconnect.

[0098] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, the inference and / or training logic 715 may be used in the system of Figure 9 for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0099] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0100] Figure 10 is a block diagram showing an electronic device 1000 for utilizing a processor 1010 according to at least one embodiment. In at least one embodiment, the electronic device 1000 may be, for example, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a telephone, an embedded computer, or any other suitable electronic device.

[0101] In at least one embodiment, system 1000 may include a processor 1010 communicably coupled to any number or type of preferred components, peripherals, modules, or devices, but not limited to these. In at least one embodiment, the processor 1010 is coupled using a bus or interface such as a 1°C bus, a System Management Bus ("SMBus"), a Low Pin Count (LPC) bus, a Serial Peripheral Interface ("SPI"), a High Definition Audio ("HDA") bus, a Serial Advance Technology Attachment ("SATA") bus, a Universal Serial Bus ("USB") (versions 1, 2, or 3), or a Universal Asynchronous Receiver / Transmitter ("UART") bus. In at least one embodiment, Figure 10 shows a system including interconnected hardware devices or “chips,” while in other embodiments, Figure 10 may show an exemplary system-on-a-chip (“SoC”). In at least one embodiment, the devices shown in Figure 10 may be interconnected by proprietary interconnects, standard interconnects (e.g., PCIe), or any combination thereof. In at least one embodiment, one or more components of Figure 10 are interconnected using a Compute Express Link (CXL) interconnect.

[0102] In at least one embodiment, Figure 10 includes a display 1024, a touchscreen 1025, a touchpad 1030, a Near Field Communication ("NFC") unit 1045, a sensor hub 1040, a thermal sensor 1046, an Express Chipset ("EC") 1035, a Trusted Platform Module ("TPM") 1038, a BIOS / firmware / flash memory ("BIOS,FW flash") 1022, a DSP 1060, a drive 1020 such as a Solid State Disk ("SSD") or Hard Disk Drive ("HDD"), a Wireless Local Area Network ("WLAN") unit 1050, a Bluetooth unit 1052, and a Wireless Wide Area Network ("WWAN") unit. The components may include a network (1056), a Global Positioning System (GPS) (1055), a camera such as a USB 3.0 camera ("USB 3.0 camera") (1054), and / or a Low Power Double Data Rate ("LPDDR") memory unit ("LPDDR3") (1015) implemented, for example, in the LPDDR3 standard. Each of these components may be implemented in any preferred manner.

[0103] In at least one embodiment, other components may be communicatively coupled to the processor 1010 through the components described above. In at least one embodiment, the accelerometer 1041, the ambient light sensor ("ALS") 1042, the compass 1043, and the gyroscope 1044 may be communicatively coupled to the sensor hub 1040. In at least one embodiment, the thermal sensor 1039, the fan 1037, the keyboard 1046, and the touchpad 1030 may be communicatively coupled to the EC 1035. In at least one embodiment, the speaker 1063, the headphones 1064, and the microphone ("mic") 1065 may be communicatively coupled to the audio unit ("audio codec and class d amplifier") 1062, and the audio unit 1062 may be communicatively coupled to the DSP 1060. In at least one embodiment, the audio unit 1064 may include, for example, an audio coder / decoder ("codec") and a Class D amplifier. In at least one embodiment, a SIM card ("SIM") 1057 may be communicatively coupled to the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and the Bluetooth unit 1052, as well as the WWAN unit 1056, may be implemented in a Next Generation Form Factor ("NGFF").

[0104] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, the inference and / or training logic 715 may be used in the system of Figure 10 for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0105] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0106] Figure 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1100 includes one or more processors 1102 and one or more graphics processors 1108, and may be a single-processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 1102 or processor cores 1107. In at least one embodiment, system 1100 is a processing platform embedded in a system-on-a-chip (SoC) integrated circuit for use in a mobile device, handheld device, or embedded device.

[0107] In at least one embodiment, system 1100 may include, or may be incorporated within, a server-based gaming platform, a game console including a game and media console, a mobile gaming console, a handheld game console, or an online game console. In at least one embodiment, system 1100 is a mobile phone, a smartphone, a tablet computing device, or a mobile internet device. In at least one embodiment, processing system 1100 may also include, can be coupled with, or may be incorporated within, wearable devices such as a smartwatch wearable device, a smart eyewear device, an augmented reality device, or a virtual reality device. In at least one embodiment, processing system 1100 is a television or set-top box device having one or more processors 1102 and a graphical interface produced by one or more graphics processors 1108.

[0108] In at least one embodiment, one or more processors 1102 each include one or more processor cores 1107 for processing instructions that perform actions for the system and user software when executed. In at least one embodiment, each of the one or more processor cores 1107 is configured to process a particular instruction set 1109. In at least one embodiment, the instruction set 1109 may facilitate computing via Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or Very Long Instruction Words (VLIW). In at least one embodiment, each processor core 1107 may process a different instruction set 1109, and the instruction set 1109 may include instructions to facilitate emulation of other instruction sets. In at least one embodiment, the processor core 1107 may also include other processing devices, such as a digital signal processor (DSP).

[0109] In at least one embodiment, the processor 1102 includes cache memory 1104. In at least one embodiment, the processor 1102 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory is shared among the various components of the processor 1102. In at least one embodiment, the processor 1102 also uses an external cache (e.g., a Level 3 (L3) cache or a Last Level Cache (LLC)) (not shown), and the external cache may be shared among the processor cores 1107 using known cache coherency techniques. In at least one embodiment, additionally, a register file 1106 is included in the processor 1102, and the register file 1106 may include different types of registers for storing different types of data (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers). In at least one embodiment, the register file 1106 may include general-purpose registers or other registers.

[0110] In at least one embodiment, one or more processors 1102 are coupled with one or more interface buses 1110 to transmit communication signals, such as addresses, data, or control signals, between the processors 1102 and other components in the system 1100. In at least one embodiment, the interface bus 1110 in one embodiment may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. In at least one embodiment, the interface 1110 is not limited to the DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), a memory bus, or other types of interface buses. In at least one embodiment, the (one or more) processors 1102 include an integrated memory controller 1116 and a platform controller hub 1130. In at least one embodiment, the memory controller 1116 facilitates communication between the memory device and other components of the system 1100, and the platform controller hub (PCH) 1130 provides connectivity to I / O devices via the local I / O bus.

[0111] In at least one embodiment, the memory device 1120 may be a dynamic random-access memory (DRAM) device, a static random-access memory (SRAM) device, a flash memory device, a phase-change memory device, or any other memory device having performance suitable for acting as process memory. In at least one embodiment, the memory device 1120 may act as system memory for the system 1100 to store data 1122 and instructions 1121 for use when one or more processors 1102 execute an application or process. In at least one embodiment, the memory controller 1116 may also be coupled to an optional external graphics processor 1112, which may communicate with one or more graphics processors 1108 in the processor 1102 to perform graphics and media operations. In at least one embodiment, the display device 1111 may be connected to one or more processors 1102. In at least one embodiment, the display device 1111 may include one or more internal display devices, such as those found in mobile electronic devices or laptop devices, or external display devices attached via a display interface (e.g., DisplayPort). In at least one embodiment, the display device 1111 may include a head-mounted display (HMD), such as a stereoscopic display device for use in virtual reality (VR) or augmented reality (AR) applications.

[0112] In at least one embodiment, the platform controller hub 1130 enables peripherals to connect to the memory device 1120 and processor 1102 via a high-speed I / O bus. In at least one embodiment, the I / O peripherals include, but are not limited to, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, a touch sensor 1125, and a data storage device 1124 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 may be connected via a storage interface (e.g., SATA) or via a peripheral bus such as a peripheral component interconnect bus (e.g., PCI, PCI Express). In at least one embodiment, the touch sensor 1125 may include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 1126 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or Long-Term Evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1128 enables communication with system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 1134 may enable network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus 1110. In at least one embodiment, the audio controller 1146 is a multi-channel high-definition audio controller.In at least one embodiment, system 1100 includes an optional legacy I / O controller 1140 for connecting legacy devices (e.g., Personal System 2 (PS / 2)) to the system. In at least one embodiment, platform controller hub 1130 can also connect to one or more Universal Serial Bus (USB) controller 1142-connected input devices, such as a keyboard and mouse combination 1143, a camera 1144, or other USB input devices.

[0113] In at least one embodiment, instances of the memory controller 1116 and the platform controller hub 1130 may be integrated into a discreet external graphics processor, such as an external graphics processor 1112. In at least one embodiment, the platform controller hub 1130 and / or the memory controller 1116 may be external to one or more processors 1102. For example, in at least one embodiment, the system 1100 may include an external memory controller 1116 and a platform controller hub 1130, which may be configured as a memory controller hub and peripheral controller hub within a system chipset communicating with (one or more) processors 1102.

[0114] The inference and / or training logic 715 is used to perform the inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, part or all of the inference and / or training logic 715 may be incorporated into the graphics processor 1500. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs embodied in the graphics processor. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than the logic shown in Figure 7A or Figure 7B. In at least one embodiment, weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that constitute the ALUs of the graphics processor for performing one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0115] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0116] Figure 12 is a block diagram of a processor 1200 having one or more processor cores 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208, according to at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores up to additional cores 1202N, represented by dashed boxes. In at least one embodiment, each of the processor cores 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cached units 1206.

[0117] In at least one embodiment, the internal cache units 1204A-1204N and the shared cache unit 1206 represent a cache memory hierarchy within the processor 1200. In at least one embodiment, the cache memory units 1204A-1204N may include at least one level of instruction and data cache within each processor core, and one or more levels of shared intermediate level caches such as Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, where the highest level cache prior to external memory is classified as LLC. In at least one embodiment, cache coherency logic maintains coherency among the various cache units 1206 and 1204A-1204N.

[0118] In at least one embodiment, the processor 1200 may also include a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, one or more bus controller units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1210 provides management functionality for various processor components. In at least one embodiment, the system agent core 1210 includes one or more integrated memory controllers 1214 for managing access to various external memory devices (not shown).

[0119] In at least one embodiment, one or more of the processor cores 1202A to 1202N include support for simultaneous multithreading. In at least one embodiment, the system agent core 1210 includes components for coordinating and operating the cores 1202A to 1202N during multithreaded processing. In at least one embodiment, the system agent core 1210 may additionally include a power control unit (PCU), the PCU including logic and components for regulating the power state of one or more of the processor cores 1202A to 1202N and the graphics processor 1208.

[0120] In at least one embodiment, the processor 1200 additionally includes a graphics processor 1208 for performing graphics processing operations. In at least one embodiment, the graphics processor 1208 is coupled with a system agent core 1210 which includes a shared cache unit 1206 and one or more integrated memory controllers 1214. In at least one embodiment, the system agent core 1210 also includes a display controller 1211 for driving graphics processor outputs to one or more coupled displays. In at least one embodiment, the display controller 1211 may also be a separate module coupled with the graphics processor 1208 via at least one interconnection, or it may be incorporated within the graphics processor 1208.

[0121] In at least one embodiment, a ring-based interconnect unit 1212 is used to connect the internal components of the processor 1200. In at least one embodiment, alternative interconnect units such as point-to-point interconnects, switching interconnects, or other techniques may be used. In at least one embodiment, the graphics processor 1208 is connected to the ring interconnect 1212 via an I / O link 1213.

[0122] In at least one embodiment, I / O link 1213 represents at least one of several types of I / O interconnects, including on-package I / O interconnects that facilitate communication between various processor components and high-performance embedded memory modules 1218, such as eDRAM modules. In at least one embodiment, each of the processor cores 1202A to 1202N and the graphics processor 1208 use the embedded memory module 1218 as a shared last-level cache.

[0123] In at least one embodiment, the processor cores 1202A to 1202N are homogeneous cores that execute a common instruction set architecture. In at least one embodiment, the processor cores 1202A to 1202N are heterogeneous in terms of instruction set architecture (ISA), where one or more of the processor cores 1202A to 1202N execute a common instruction set, and the other cores of one or more of the processor cores 1202A to 1202N execute a subset of the common instruction set or a different instruction set. In at least one embodiment, the processor cores 1202A to 1202N are heterogeneous in terms of microarchitecture, where one or more cores with relatively high power consumption are coupled with one or more power cores with lower power consumption. In at least one embodiment, the processor 1200 may be implemented on one or more chips or as an SoC integrated circuit.

[0124] The inference and / or training logic 715 is used to perform the inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 715 are provided below in conjunction with Figures 7A and / or 7B. In at least one embodiment, part or all of the inference and / or training logic 715 may be incorporated into the processor 1200. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more of the graphics processor 1512, (one or more) graphics cores 1202A to 1202N, or other components embodied in Figure 12. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed using logic other than the logic shown in Figure 7A or Figure 7B. In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that constitute the ALU of the graphics processor 1200 for implementing one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0125] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0126] Virtualized computing platform Figure 13 is an exemplary data flow diagram for a process 1300 that generates and deploys an image processing and inference pipeline, according to at least one embodiment. In at least one embodiment, process 1300 may be deployed in one or more facilities 1302 for use with imaging devices, processing devices, and / or other device types. Process 1300 may run within a training system 1304 and / or deployment system 1306. In at least one embodiment, training system 1304 may be used to carry out the training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use in deployment system 1306. In at least one embodiment, deployment system 1306 may be configured to offload processing and compute resources between distributed computing environments to reduce infrastructure requirements at facility 1302. In at least one embodiment, one or more applications in the pipeline may use or call services of the deployment system 1306 (e.g., inference, visualization, compute, AI, etc.) while the application is running.

[0127] In at least one embodiment, some of the applications used in the advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at facility 1302 using data 1308 (such as imaging data) generated at facility 1302 (and stored in one or more picture archiving and communication system (PACS) servers at facility 1302), or it may be trained using imaging or sequencing data 1308 from one or more other facilities, or a combination thereof. In at least one embodiment, the training system 1304 may be used to provide applications, services, and / or other resources for generating a working and deployable machine learning model for the deployment system 1306.

[0128] In at least one embodiment, the model registry 1324 may be backed up by object storage capable of supporting version control and object metadata. In at least one embodiment, the object storage may be accessible, for example, from within a cloud platform, through a cloud storage (e.g., cloud 1426 in Figure 14) compatible application programming interface (API). In at least one embodiment, machine learning models in the model registry 1324 may be uploaded, listed, modified, or deleted by a system developer or partner interacting with the API. In at least one embodiment, the API may provide access to a way that enables appropriately qualified users to associate models with applications, thereby allowing the models to run as part of a containerized instantiation of the application.

[0129] In at least one embodiment, the training pipeline 1404 (Figure 14) may include a scenario in which the facility 1302 is training its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, imaging data 1308 generated by (one or more) imaging devices, sequencing devices, and / or other device types may be received. In at least one embodiment, once the imaging data 1308 is received, AI-assisted annotation 1310 may be used to help generate annotations corresponding to the imaging data 1308 to be used as ground truth data for the machine learning model. In at least one embodiment, AI-assisted annotation 1310 may include one or more machine learning models (e.g., convolutional neural networks (CNNs)), and one or more machine learning models may be trained to generate annotations corresponding to several types of imaging data 1308 (e.g., from several devices). In at least one embodiment, the AI-assisted annotation 1310 may then be used directly to generate ground truth data, or it may be modified or fine-tuned using an annotation tool. In at least one embodiment, the AI-assisted annotation 1310, labeled clinic data 1312, or a combination thereof may be used as ground truth data for training a machine learning model. In at least one embodiment, the trained machine learning model may be referred to as output model 1316 and may be used by the deployment system 1306 as described herein.

[0130] In at least one embodiment, the training pipeline 1404 (Figure 14) may include a scenario in which facility 1302 requires a machine learning model to use when performing one or more processing tasks for one or more applications in the deployment system 1306, but facility 1302 may not currently have such a machine learning model (or may not have a model optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from the model registry 1324. In at least one embodiment, the model registry 1324 may include machine learning models trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, the machine learning models in the model registry 1324 may be trained on imaging data from a different facility than facility 1302 (e.g., a facility in a remote location). In at least one embodiment, the machine learning model may be trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when trained on imaging data from a specific location, training may be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of the imaging data outside the premises. In at least one embodiment, once a model is trained, or partially trained, at one location, the machine learning model may be added to the model registry 1324. In at least one embodiment, the machine learning model may then be retrained or updated at any number of other facilities, and the retrained or updated model may be made available in the model registry 1324. In at least one embodiment, a machine learning model may then be selected from the model registry 1324, which may be referred to as an output model 1316, and may be used in the deployment system 1306 to perform one or more processing tasks for one or more applications of the deployment system.

[0131] In at least one embodiment, the training pipeline 1404 (Figure 14), scenario may include the facility 1302 requiring a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 1306, but facility 1302 may not currently have such a machine learning model (or may not have an efficient or effective model optimized for such purposes). In at least one embodiment, a machine learning model selected from the model registry 1324 may not be fine-tuned or optimized for the imaging data 1308 generated at facility 1302 due to differences in the population, the robustness of the training data used to train the machine learning model, the diversity of anomalies in the training data, and / or other issues associated with the training data. In at least one embodiment, AI-assisted annotation 1310 may be used to help generate annotations corresponding to the imaging data 1308 to be used as ground truth data for retraining or updating the machine learning model. In at least one embodiment, labeled data 1312 may be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating a machine learning model may be referred to as model training 1314. In at least one embodiment, model training 1314, for example, AI-assisted annotation 1310, labeled clinic data 1312, or a combination thereof, may be used as ground truth data for retraining or updating a machine learning model. In at least one embodiment, the trained machine learning model may be referred to as output model 1316, which may be used by the deployment system 1306 as described herein.

[0132] In at least one embodiment, the deployment system 1306 may include software 1318, services 1320, hardware 1322, and / or other components, features, and functionalities. In at least one embodiment, the deployment system 1306 may include a software "stack" so that software 1318 can be built on top of services 1320 and use services 1320 to perform some or all of the processing tasks, and services 1320 and software 1318 can be built on top of hardware 1322 and use hardware 1322 to perform the processing, storage, and / or other compute tasks of the deployment system 1306. In at least one embodiment, software 1318 may include any number of different containers, each of which may perform the instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks in the advanced processing and inference pipeline (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.). In at least one embodiment, the advanced processing and inference pipeline may be defined based on the selection of different containers desired or required to process the imaging data 1308, in addition to a container that receives and configures the imaging data for use by each container and / or by facility 1302 after processing it through the pipeline (for example, to convert the output to an available data type). In at least one embodiment, the combination of containers in software 1318 (for example, building the pipeline) may be referred to as a virtual device (as described in more detail herein), which may leverage services 1320 and hardware 1322 to perform some or all of the processing tasks of applications instantiated in the containers.

[0133] In at least one embodiment, the data processing pipeline may receive input data (e.g., imaging data 1308) in a specific format in response to an inference request (e.g., a request from a user of the deployment system 1306). In at least one embodiment, the input data may represent one or more images, videos, and / or other data representations generated by one or more imaging devices. In at least one embodiment, the data may undergo preprocessing as part of the data processing pipeline to prepare the data for processing by one or more applications. In at least one embodiment, postprocessing may be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare output data for subsequent applications and / or output data for user transmission and / or use (e.g., as a response to an inference request). In at least one embodiment, the inference task may be performed by one or more machine learning models, such as a trained or deployed neural network, which may include the output model 1316 of the training system 1304.

[0134] In at least one embodiment, tasks in a data processing pipeline may be encapsulated in one or more containers, each representing a separate, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, a container or application may be exposed to a private (e.g., restricted access) area of ​​a container registry (described in more detail herein), and trained or deployed models may be stored in a model registry 1324 and associated with one or more applications. In at least one embodiment, an image of an application (e.g., a container image) may be available in the container registry, and once selected by a user from the container registry for deployment in a pipeline, the image may be used to generate a container for instantiation of the application for use by the user's system.

[0135] In at least one embodiment, a developer (e.g., a software developer, clinician, physician, etc.) may develop, publish, and store an application (e.g., as a container) for performing image processing and / or inference on supplied data. In at least one embodiment, development, publication, and / or storage may be performed using a software development kit (SDK) associated with the system (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application may be tested locally (e.g., at a first facility, against data from a first facility) using an SDK that can support at least some of the services 1320 as a system (e.g., system 1400 in Figure 14). In at least one embodiment, because a DICOM object may contain anywhere one to hundreds of images or other data types, and due to data variability, the developer may be responsible for managing the extraction and preparation of incoming data (e.g., setting up constructs for the application, incorporating preprocessing into the application, etc.). In at least one embodiment, once validated by System 1400 (for example, with respect to accuracy), the application may become available in a container registry for user selection and / or implementation to perform one or more processing tasks on data at the user's facility (for example, a second facility).

[0136] In at least one embodiment, the developer may then share the application or container over a network for access and use by users of the system (e.g., system 1400 in Figure 14). In at least one embodiment, the completed and validated application or container may be stored in a container registry, and the associated machine learning models may be stored in a model registry 1324. In at least one embodiment, a requesting entity providing an inference or image processing request may browse the container registry and / or model registry 1324 for applications, containers, datasets, machine learning models, etc., select a desired combination of elements to include in the data processing pipeline, and submit an imaging processing request. In at least one embodiment, the request may include input data (and, in some examples, associated patient data) necessary to perform the request, and / or include the selection of (one or more) applications and / or machine learning models to be executed when processing the request. In at least one embodiment, the request may then be passed to one or more components of the deployment system 1306 (e.g., the cloud) to perform the processing in the data processing pipeline. In at least one embodiment, processing by the deployment system 1306 may include referencing selected elements (e.g., applications, containers, models, etc.) from the container registry and / or model registry 1324. In at least one embodiment, once the pipeline has generated results, the results may be returned to the user for reference (e.g., for viewing in a viewing application suite running on a local, on-premises workstation or terminal).

[0137] In at least one embodiment, service 1320 may be utilized to assist in the processing or execution of an application or container in a pipeline. In at least one embodiment, service 1320 may include compute services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, service 1320 may provide functionality common to one or more applications in software 1318, and thus the functionality may be abstracted to a service that can be called or utilized by the applications. In at least one embodiment, the functionality provided by service 1320 may operate dynamically and more efficiently, and may scale well by enabling applications to process data in parallel (for example, using a parallel computing platform 1430 (Figure 14)). In at least one embodiment, service 1320 may be shared among and between various applications, rather than each application sharing the same functionality provided by service 1320 being required to have its own instance of service 1320. In at least one embodiment, the service may include, in non-limiting examples, an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included that can provide machine learning model training and / or retraining capability. In at least one embodiment, a data augmentation service may further be included that can provide GPU-accelerated data extraction, resizing, scaling, and / or other augmentation (e.g., DICOM, RIS, CIS, REST-compliant, RPC, raw, etc.). In at least one embodiment, a visualization service may be used that can add image rendering effects such as ray tracing, rasterization, denoising, and sharpening to add realism to two-dimensional (2D) and / or three-dimensional (3D) models.In at least one embodiment, a virtual instrument service may be included that provides beamforming, segmentation, inference, imaging, and / or support for other applications within the virtual instrument pipeline.

[0138] In at least one embodiment, if service 1320 includes an AI service (e.g., an inference service), one or more machine learning models may be executed by calling the inference service (e.g., an inference server) (e.g., as an API call) to execute the machine learning models, or their processing, as part of the application execution. In at least one embodiment, if another application includes one or more machine learning models for a segmentation task, the application may call the inference service to execute the machine learning models to perform one or more processing operations associated with the segmentation task. In at least one embodiment, software 1318 implementing an advanced processing and inference pipeline including a segmentation application and an anomaly detection application may be streamlined so that each application can call the same inference service to perform one or more inference tasks.

[0139] In at least one embodiment, the hardware 1322 may include a GPU, CPU, graphics card, AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX), cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 1322 may be used to provide efficient and dedicated support for the software 1318 and service 1320 in deployment system 1306. In at least one embodiment, the use of GPU processing for processing locally (e.g., at facility 1302) may be implemented within the AI / deep learning system, in the cloud system, and / or in other processing components of deployment system 1306 to improve the efficiency, accuracy, and effectiveness of image processing and generation. In at least one embodiment, the software 1318 and / or service 1320 may, as a non-limiting example, be optimized for GPU processing relating to deep learning, machine learning, and / or high-performance computing. In at least one embodiment, at least a portion of the computing environment of the deployment system 1306 and / or training system 1304 may be run on one or more supercomputers or high-performance computing systems in a data center using GPU-optimized software (e.g., a combination of hardware and software from NVIDIA's DGX system). In at least one embodiment, the hardware 1322 may include any number of GPUs, which may be called to perform parallel processing of data as described herein. In at least one embodiment, the cloud platform may further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computing tasks. In at least one embodiment, the cloud platform (e.g., NVIDIA's NGC) may run using (one or more) AI / deep learning supercomputers and / or GPU-optimized software (e.g., provided on NVIDIA's DGX system) as a hardware abstraction and scaling platform.In at least one embodiment, the cloud platform may integrate an application container clustering system or orchestration system (e.g., Kubernetes) for multiple GPUs to enable seamless scaling and load balancing.

[0140] Figure 14 is a system diagram for an exemplary system 1400 for generating and implementing an imaging introduction pipeline, according to at least one embodiment. In at least one embodiment, system 1400 may be used to implement process 1300 of Figure 13, and / or other processes including advanced processing and inference pipelines. In at least one embodiment, system 1400 may include a training system 1304 and an introduction system 1306. In at least one embodiment, the training system 1304 and the introduction system 1306 may be implemented using software 1318, services 1320, and / or hardware 1322 as described herein.

[0141] In at least one embodiment, System 1400 (e.g., training system 1304 and / or deployment system 1306) may be implemented in a cloud computing environment (e.g., using cloud 1426). In at least one embodiment, System 1400 may be implemented locally with respect to a healthcare service facility, or as a combination of both cloud computing resources and local computing resources. In at least one embodiment, access to APIs in cloud 1426 may be restricted to authorized users through established security measures or protocols. In at least one embodiment, the security protocol may include web tokens, which may be signed by authentication services (e.g., AuthN, AuthZ, Gluecon, etc.) and may have appropriate authorizations. In at least one embodiment, APIs of virtual devices (as described herein), or other instantiations of System 1400, may be restricted to a set of public IPs that are inspected or authorized for interaction.

[0142] In at least one embodiment, various components of System 1400 may communicate with one another using any of various different network types, including, but not limited to, local area networks (LANs) and / or wide area networks (WANs), via wired and / or wireless communication protocols. In at least one embodiment, communication between a facility and components of System 1400 (e.g., for sending inference requests, for receiving the results of inference requests) may be conducted via (one or more) data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0143] In at least one embodiment, the training system 1304 may execute a training pipeline 1404 similar to that described herein with respect to Figure 13. In at least one embodiment, if one or more machine learning models are to be used in the introduction pipeline 1410 by the introduction system 1306, the training pipeline 1404 may be used to train or retrain one or more (e.g., pre-trained) models, and / or to implement one or more of the pre-trained models 1406 (e.g., without the need for retraining or updating). In at least one embodiment, one or more output models 1316 may be produced as a result of the training pipeline 1404. In at least one embodiment, the training pipeline 1404 may include any number of processing steps, including, but not limited to, image data (or other input data) conversion or adaptation. In at least one embodiment, different training pipelines 1404 may be used for different machine learning models used by the introduction system 1306. In at least one embodiment, a training pipeline 1404 similar to the first example described with respect to Figure 13 may be used for a first machine learning model, a training pipeline 1404 similar to the second example described with respect to Figure 13 may be used for a second machine learning model, and a training pipeline 1404 similar to the third example described with respect to Figure 13 may be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1304 may be used, depending on what is required for each respective machine learning model. In at least one embodiment, one or more of the machine learning models may already be trained and ready for deployment, and therefore the machine learning models may not undergo any processing by the training system 1304 and may be implemented by the deployment system 1306.

[0144] In at least one embodiment, the (one or more) output models 1316 and / or the (one or more) pre-trained models 1406 may include any type of machine learning model depending on the implementation or embodiment. In at least one embodiment, and not limited to, the machine learning models used by System 1400 may include (one or more) machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recurrents, perceptrons, Long / Short Term Memory (LSTMs), Hopfield, Boltzmann, deep beliefs, deconvolutions, adversarial generatives, liquid state machines, etc.), and / or other types of machine learning models.

[0145] In at least one embodiment, the training pipeline 1404 may include AI-assisted annotation, as will be described in more detail herein with respect to at least Figure 15B. In at least one embodiment, labeled data 1312 (e.g., conventional annotation) may be generated by any number of techniques. In at least one embodiment, labels or other annotations may be generated in a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground truth, and / or, in some examples, by handwriting. In at least one embodiment, ground truth data may be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated (e.g., a labeller or annotation expert defines the location of the labels), and / or a combination thereof. In at least one embodiment, for each instance of the imaging data 1308 (or other data types used by the machine learning model), there may be corresponding ground truth data generated by the training system 1304. In at least one embodiment, AI-assisted annotation may be performed as part of the introduction pipeline 1410, either in addition to or instead of AI-assisted annotation included in the training pipeline 1404. In at least one embodiment, system 1400 may include a multilayer platform, which may include a software layer (e.g., software 1318) of a diagnostic application (or other application type) capable of performing one or more medical imaging and diagnostic functions. In at least one embodiment, system 1400 may be communicably coupled (e.g., via an encrypted link) to a PACS server network of one or more facilities.In at least one embodiment, the system 1400 may be configured to access and reference data from a PACS server in order to perform operations such as training a machine learning model, deploying a machine learning model, image processing, inference, and / or other operations.

[0146] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or authenticated API through which an application or container may be invoked (e.g., called) from (one or more) external environments (e.g., facility 1302). In at least one embodiment, the application may then call or execute one or more services 1320 to perform compute, AI, or visualization tasks associated with each application, and the software 1318 and / or services 1320 may leverage the hardware 1322 to perform the processing tasks in an effective and efficient manner.

[0147] In at least one embodiment, the introduction system 1306 may execute an introduction pipeline 1410. In at least one embodiment, the introduction pipeline 1410 may include any number of applications that may be applied sequentially, discontinuously, or otherwise to imaging data (and / or other data types) generated by imaging devices, sequencing devices, genomics devices, etc., including AI-assisted annotation as described above. In at least one embodiment, as described herein, the introduction pipeline 1410 for individual devices may be referred to as a virtual instrument for the device (e.g., a virtual ultrasound instrument, a virtual CT scanning instrument, a virtual sequencing instrument, etc.). In at least one embodiment, there may be two or more introduction pipelines 1410 for a single device, depending on the information desired from the data generated by the device. In at least one embodiment, there may be a first introduction pipeline 1410 if anomaly detection is desired from the MRI machine, and there may be a second introduction pipeline 1410 if image enhancement is desired from the output of the MRI machine.

[0148] In at least one embodiment, the image generation application may include processing tasks that involve the use of a machine learning model. In at least one embodiment, the user may wish to use their own machine learning model or select a machine learning model from the model registry 1324. In at least one embodiment, the user may implement their own machine learning model or select a machine learning model to include in the application in order to perform the processing tasks. In at least one embodiment, the application may be selectable and customizable, and by defining the construction of the application, the deployment and implementation of the application for a particular user is presented as a more seamless user experience. In at least one embodiment, by leveraging other features of the system 1400, such as services 1320 and hardware 1322, the deployment pipeline 1410 may become even more user-friendly, provide easier integration, and produce more accurate, efficient, and timely results.

[0149] In at least one embodiment, the deployment system 1306 may include a user interface 1414 (e.g., a graphical user interface, a web interface, etc.) which may be used to select applications for inclusion in one or more deployment pipelines 1410, to deploy applications, to modify or change applications or their parameters or construction, to use and interact with one or more deployment pipelines 1410 during setup and / or deployment, and / or to interact with the deployment system 1306 in other ways. In at least one embodiment, although not shown with respect to the training system 1304, the user interface 1414 (or a different user interface) may be used to select models for use in the deployment system 1306, to select models for training or retraining in the training system 1304, and / or to interact with the training system 1304 in other ways.

[0150] In at least one embodiment, a pipeline manager 1412 may be used in addition to the application orchestration system 1428 to manage interactions between applications or containers in (one or more) deployment pipelines 1410 and services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 may be configured to facilitate application-to-application interactions, applications-to-service interactions, and / or applications or services-to-hardware interactions. In at least one embodiment, it is shown to be included in software 1318, but this is not intended to limit it, and in some examples (for example, shown in Figure 12cc), the pipeline manager 1412 may be included in services 1320. In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, DOCKER, etc.) may include a container orchestration system, which may group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications (e.g., reconstruction applications, segmentation applications, etc.) from one or more deployment pipelines 1410 with individual containers, each application can run in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0151] In at least one embodiment, each application and / or container (or its image) may be developed, modified, and deployed individually (for example, a first user or developer may develop, modify, and deploy a first application, and a second user or developer may develop, modify, and deploy a second application independently of the first user or developer), which may allow for focusing and paying attention to the tasks of a single application and / or container without being interrupted by the tasks of other applications or containers. In at least one embodiment, communication and coordination between different containers or applications may be assisted by the pipeline manager 1412 and the application orchestration system 1428. In at least one embodiment, the application orchestration system 1428 and / or the pipeline manager 1412 may facilitate communication between each of the applications or containers and the sharing of resources between them, as long as the expected inputs and / or outputs of each container or application are known by the system (for example, based on the construction of the application or container). In at least one embodiment, one or more applications or containers in the deployment pipeline 1410 may share the same services and resources, so the application orchestration system 1428 may orchestrate, load balance, and decide on sharing services or resources among the various applications or containers. In at least one embodiment, a scheduler may be used to track the resource requirements of the applications or containers, the current or planned usage of these resources, and the availability of the resources. In at least one embodiment, the scheduler may therefore allocate resources to different applications and distribute resources among applications, taking into account the system requirements and availability.In some examples, the scheduler (and / or other components of the application orchestration system 1428) may determine the availability and distribution of resources based on constraints imposed on the system (e.g., user constraints), such as quality of service (QoS) and the urgency of the need for data output (e.g., to determine whether real-time processing or delayed processing should be performed).

[0152] In at least one embodiment, the services 1320 utilized and shared by applications or containers in the deployment system 1306 may include compute services 1416, AI services 1418, visualization services 1420, and / or other service types. In at least one embodiment, an application may call (e.g., execute) one or more of the services 1320 to perform processing operations for the application. In at least one embodiment, compute service 1416 may be utilized by an application to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more compute services 1416 may be utilized to process data through one or more applications and / or to perform parallel processing (e.g., using a parallel computing platform 1430) for substantially simultaneous processing of one or more tasks of a single application. In at least one embodiment, the parallel computing platform 1430 (e.g., NVIDIA's CUDA) may enable general-purpose computing on GPUs (GPGPU: GPGPU) on a GPU (e.g., GPU 1422). In at least one embodiment, the software layer of the parallel computing platform 1430 may provide access to a virtual instruction set and parallel computing elements of the GPU for the execution of a compute kernel. In at least one embodiment, the parallel computing platform 1430 may include memory, which in some embodiments may be shared among multiple containers and / or among different processing tasks within a single container.In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within containers to use the same data from a shared segment of memory on the parallel computing platform 1430 (for example, when multiple different stages of an application or multiple applications are processing the same information). In at least one embodiment, the same data in the same location in memory may be used for any number of processing tasks (for example, at the same time, at different times, etc.) rather than making copies of the data and moving the data to different locations in memory (e.g., read / write operations). In at least one embodiment, when data is used and new data is generated as a result of processing, this information about the new location of the data may be stored and shared among various applications. In at least one embodiment, the location of the data and the location of the updated or modified data may be part of the definition of how the payload is understood within the container.

[0153] In at least one embodiment, the AI ​​service 1418 may be leveraged to perform inference services for running one or more machine learning models associated with an application (e.g., assigned the role of performing one or more processing tasks of the application). In at least one embodiment, the AI ​​service 1418 may leverage the AI ​​system 1424 to run one or more machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, an application in one or more introductory pipelines 1410 may perform inference on imaging data using one or more output models 1316 from the training system 1304 and / or other models of the application. In at least one embodiment, two or more examples of inference using the application orchestration system 1428 (e.g., a scheduler) may be available. In at least one embodiment, the first category may include high-priority / low-latency routes that can achieve a higher service level agreement, such as for performing reasoning regarding urgent requests in emergencies or for radiologists during diagnosis. In at least one embodiment, the second category may include standard-priority routes that may be used for requests that may not be urgent or where analysis may be performed later. In at least one embodiment, the application orchestration system 1428 may distribute resources (e.g., services 1320 and / or hardware 1322) based on priority routes for different reasoning tasks of the AI ​​service 1418.

[0154] In at least one embodiment, shared storage may be attached to the AI ​​service 1418 within system 1400. In at least one embodiment, shared storage may act as a cache (or other storage device type) and may be used to process inference requests from an application. In at least one embodiment, when an inference request is submitted, the request may be received by a set of API instances of deployment system 1306, and one or more instances may be selected to process the request (e.g., for best fit, for load balancing, etc.). In at least one embodiment, to process the request, the request may be placed in a database, a machine learning model may be located from the model registry 1324 if it is not already in the cache, and a verification step may ensure that a suitable machine learning model is loaded into the cache (e.g., shared storage) and / or a copy of the model can be stored in the cache. In at least one embodiment, if the application is not yet running or there are not enough instances of the application, a scheduler (e.g., of pipeline manager 1412) may be used to start the application referenced in the request. In at least one embodiment, an inference server may be started if one has not yet been started to run the model. Any number of inference servers may be started for each model. In at least one embodiment, in a clustered pull model, the model may be cached whenever load balancing is advantageous. In at least one embodiment, the inference server may be statically loaded onto the corresponding distributed server.

[0155] In at least one embodiment, inference may be performed using an inference server running in a container. In at least one embodiment, an instance of the inference server may be associated with a model (optionally, multiple versions of the model). In at least one embodiment, when a request to perform inference against a model is received and no instance of the inference server exists, a new instance may be loaded. In at least one embodiment, when the inference server is started, a model may be passed to the inference server, thereby allowing the same container to be used to serve different models, as long as the inference server is running as a different instance.

[0156] In at least one embodiment, while the application is running, an inference request for a given application may be received, a container (e.g., hosting an instance of the inference server) may be loaded (if not already loaded), and a start procedure may be called. In at least one embodiment, preprocessing logic in the container may load, decode, and / or perform any additional preprocessing on incoming data (e.g., using one or more CPUs and / or one or more GPUs). In at least one embodiment, once the data is prepared for inference, the container may perform inference on the data as needed. In at least one embodiment, this may involve a single inference call for a single image (e.g., an X-ray of a hand) or may require inference on hundreds of images (e.g., a CT scan of a chest). In at least one embodiment, the application may summarize results before completion, which may include, but are not limited to, generating a single confidence score, pixel-level segmentation, voxel-level segmentation, visualizations, or text to summarize the findings. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have real-time priority (TAT < 1 minute), while others may have low priority (e.g., TAT < 10 minutes). In at least one embodiment, model execution time may be measured from the requesting agency or entity and may include partner network traversal time and execution for inference services.

[0157] In at least one embodiment, the transfer of requests between service 1320 and the inference application may be hidden behind a software development kit (SDK), and robust transport may be provided through a queue. In at least one embodiment, for each application / tenant ID combination, requests are queued via an API, and the SDK retrieves the requests from the queue and delivers them to the application. In at least one embodiment, a name for the queue may be provided in the environment in which the SDK picks up requests. In at least one embodiment, asynchronous communication through a queue may be useful because the communication may allow any instance of the application to pick up the work when the work becomes available. The results may be returned through the queue to ensure that no data is lost. In at least one embodiment, the queue may also provide the ability to segment work, so that the highest-priority work may proceed to a queue where most instances of the application are connected, while the lowest-priority work may proceed to a queue where a single instance is connected, and tasks are processed in the order they are received. In at least one embodiment, the application may run on a GPU-accelerated instance generated in cloud 1426, and the inference service may perform inference on the GPU.

[0158] In at least one embodiment, the visualization service 1420 may be utilized to generate visualizations for viewing the output of an application and / or (one or more) introductory pipelines 1410. In at least one embodiment, the visualization service 1420 may utilize the GPU 1422 to generate visualizations. In at least one embodiment, rendering effects such as ray tracing may be implemented by the visualization service 1420 to generate higher quality visualizations. In at least one embodiment, visualizations may include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slicing, virtual reality display, augmented reality display, etc. In at least one embodiment, a virtualized environment may be used to generate a virtual interactive display or environment (e.g., a virtual environment) for interaction by a user of the system (e.g., a doctor, nurse, radiologist, etc.). In at least one embodiment, the visualization service 1420 may include an internal visualizer, cinematics, and / or other rendering or image processing capabilities or functionalities (e.g., ray tracing, rasterization, internal optics, etc.).

[0159] In at least one embodiment, hardware 1322 may include GPU 1422, AI system 1424, cloud 1426, and / or any other hardware used to run training system 1304 and / or deployment system 1306. In at least one embodiment, GPU 1422 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs, any number of GPUs may be used to perform processing tasks for any of the compute service 1416, AI service 1418, visualization service 1420, other services, and / or features or functionalities of software 1318. For example, with respect to AI service 1418, GPU 1422 may be used to perform preprocessing on imaging data (or other data types used by the machine learning model), postprocessing on the output of the machine learning model, and / or to perform inference (e.g., to run the machine learning model). In at least one embodiment, the cloud 1426, the AI ​​system 1424, and / or other components of system 1400 may use the GPU 1422. In at least one embodiment, the cloud 1426 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, the AI ​​system 1424 may use a GPU, and the cloud 1426, or at least a portion assigned the role of deep learning or inference, may run using one or more AI systems 1424. Thus, although hardware 1322 is shown as a separate component, this is not intended to be limiting, and any component of hardware 1322 may be combined with or leveraged by any other component of hardware 1322.

[0160] In at least one embodiment, the AI ​​system 1424 may include a dedicated computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, the AI ​​system 1424 (e.g., NVIDIA's DGX) may include GPU-optimized software (e.g., a software stack), which may run using multiple GPUs 1422 in addition to the CPU, RAM, storage, and / or other components, features, or functionalities. In at least one embodiment, one or more AI systems 1424 may be implemented in a cloud 1426 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1400.

[0161] In at least one embodiment, the cloud 1426 may include a GPU acceleration infrastructure (e.g., NVIDIA's NGC) which may provide a GPU-optimized platform for performing processing tasks of system 1400. In at least one embodiment, the cloud 1426 may include (one or more) AI systems 1424 (e.g., as a hardware abstraction and scaling platform) for performing one or more of the AI-based tasks of system 1400. In at least one embodiment, the cloud 1426 may integrate with an application orchestration system 1428, leveraging multiple GPUs to enable seamless scaling and load balancing between applications and services 1320. In at least one embodiment, the cloud 1426 may be tasked with performing at least some of the services 1320 of system 1400, including compute service 1416, AI service 1418, and / or visualization service 1420, as described herein. In at least one embodiment, Cloud 1426 may perform large and small batch inference (e.g., running NVIDIA's TENSOR RT), provide accelerated parallel computing APIs and platforms 1430 (e.g., NVIDIA's CUDA), run application orchestration systems 1428 (e.g., KUBERNETES), provide graphics rendering APIs and platforms (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality cinematics), and / or provide other functionality for system 1400.

[0162] Figure 15A shows a data flow diagram for a process 1500 for training, retraining, or updating a machine learning model, according to at least one embodiment. In at least one embodiment, process 1500 may be performed using system 1400 of Figure 14 as a non-limiting example. In at least one embodiment, process 1500 may leverage the services 1320 and / or hardware 1322 of system 1400 as described herein. In at least one embodiment, the improved model 1512 produced by process 1500 may be executed by deployment system 1306 for one or more containerized applications in deployment pipeline 1410.

[0163] In at least one embodiment, model training 1314 may include retraining or updating the initial model 1504 (e.g., a pre-trained model) using new training data (e.g., customer dataset 1506, and / or new input data such as new ground truth data associated with the input data). In at least one embodiment, in order to retrain or update the initial model 1504, one or more output or loss layers of the initial model 1504 may be reset or deleted and / or replaced with one or more updated or new output or loss layers. In at least one embodiment, the initial model 1504 may have parameters (e.g., weights and / or biases) that were previously fine-tuned and remain from the previous training, and therefore training or retraining 1314 may not take as long or require as much processing as training the model from scratch. In at least one embodiment, during model training 1314, by having one or more reset or replaced output or loss layers of the initial model 1504, the parameters may be updated and readjusted for the new dataset based on the loss calculation associated with the accuracy of one or more output or loss layers when generating predictions with respect to the new customer dataset 1506 (e.g., image data 1308 in Figure 13).

[0164] In at least one embodiment, the pre-trained model 1406 may be stored in a data store or registry (for example, the model registry 1324 in Figure 13). In at least one embodiment, the pre-trained model 1406 may be trained at least partially at one or more facilities other than the facility where process 1500 is performed. In at least one embodiment, in order to protect the privacy and rights of patients, subjects, or customers at different facilities, the pre-trained model 1406 may be trained on-site using on-site generated customer or patient data. In at least one embodiment, the pre-trained model 1406 may be trained using the cloud 1426 and / or other hardware 1322, but privacy-protected sensitive patient data may not be transferred to any component of the cloud 1426 (or other off-site hardware), may not be used by such components, or may be inaccessible to such components. In at least one embodiment, if the pre-trained model 1406 is trained using patient data from two or more facilities, the pre-trained model 1406 may be trained individually for each facility and then trained with respect to patient or customer data from another facility. In at least one embodiment, customer or patient data from any number of facilities may be used to train the pre-trained model 1406 on-premises and / or off-premises, such as in a data center or other cloud computing infrastructure, if the customer or patient data is released from privacy concerns (e.g., by waiver, for experimental use, etc.) or if the customer or patient data is included in a public data set.

[0165] In at least one embodiment, when selecting an application for use in the deployment pipeline 1410, the user may also select a machine learning model to be used for that particular application. In at least one embodiment, the user may not have a model for use and therefore may select a pre-trained model 1406 for use with the application. In at least one embodiment, the pre-trained model 1406 may not be optimized to produce accurate results with respect to the user's facility's customer dataset 1506 (for example, based on patient diversity, demographics, type of medical imaging device used, etc.). In at least one embodiment, before deploying the pre-trained model 1406 into the deployment pipeline 1410 for use with one or more applications, the pre-trained model 1406 may be updated, retrained, and / or fine-tuned for use at each respective facility.

[0166] In at least one embodiment, the user may select a pre-trained model 1406 to be updated, retrained, and / or fine-tuned, which may be referred to as the initial model 1504 for the training system 1304 within process 1500. In at least one embodiment, a customer dataset 1506 (e.g., imaging data, genomics data, sequencing data, or other data types generated by devices in a facility) may be used to perform model training 1314 (which may include, but is not limited to, transfer learning) with respect to the initial model 1504 to generate an improved model 1512. In at least one embodiment, ground truth data corresponding to the customer dataset 1506 may be generated by the training system 1304. In at least one embodiment, ground truth data may be generated at least partially by clinicians, scientists, physicians, or practitioners in a facility (e.g., as labeled clinic data 1312 in Figure 13).

[0167] In at least one embodiment, AI-assisted annotation 1310 may be used in several examples to generate ground truth data. In at least one embodiment, AI-assisted annotation 1310 (implemented, for example, using the AI-assisted annotation SDK) may leverage a machine learning model (e.g., a neural network) to generate suggestive or predictive ground truth data about a customer dataset. In at least one embodiment, user 1510 may use the annotation tool within a user interface (graphical user interface (GUI)) on computing device 1508.

[0168] In at least one embodiment, user 1510 may interact with a GUI via computing device 1508 to edit or fine-tune (automatic) annotations. In at least one embodiment, polygon editing features may be used to move polygon vertices to more precise or fine-tuned locations.

[0169] In at least one embodiment, if the customer dataset 1506 has relevant ground truth data, the ground truth data (e.g., from AI-assisted annotation, manual labeling, etc.) may be used during model training 1314 to generate an improved model 1512. In at least one embodiment, the customer dataset 1506 may be applied to the initial model 1504 any number of times, and the ground truth data may be used to update the parameters of the initial model 1504 for the improved model 1512 until an acceptable level of accuracy is achieved. In at least one embodiment, once the improved model 1512 is generated, the improved model 1512 may be deployed in one or more deployment pipelines 1410 at the facility to perform one or more processing tasks on medical imaging data.

[0170] In at least one embodiment, the improved model 1512 may be uploaded to a pre-trained model 1406 in a model registry 1324 to be selected by another facility. In at least one embodiment, the process may be completed in any number of facilities, thereby allowing the improved model 1512 to be further improved any number of times with respect to new datasets to generate a more universal model.

[0171] Figure 15B is an illustrative diagram of a client-server architecture 1532 for extending an annotation tool with a pre-trained annotation model, according to at least one embodiment. In at least one embodiment, an AI-assisted annotation tool 1536 may be instantiated based on the client-server architecture 1532. In at least one embodiment, the annotation tool 1536 in an imaging application may assist a radiologist in identifying, for example, organs and anomalies. In at least one embodiment, the imaging application may include, as a non-limiting example, a software tool that helps a user 1510 identify several extreme points on a particular organ of interest in a raw image 1534 (for example, in a 3D MRI or CT scan) and receives automatically annotated results for all 2D slices of the particular organ. In at least one embodiment, the results may be stored in a data store as training data 1538 and used as ground truth data for training (for example, but not limited to). In at least one embodiment, when computing device 1508 sends out extreme points for AI-assisted annotation 1310, a deep learning model, for example, may receive this data as input and return inference results for segmented organs or anomalies. In at least one embodiment, a pre-instantiated annotation tool, such as AI-assisted annotation tool 1536B in Figure 15B, may be extended by making an API call (e.g., API call 1544) to a server, such as annotation support server 1540, which may contain a set of pre-trained models 1542 stored, for example, in an annotation model registry. In at least one embodiment, the annotation model registry may store pre-trained models 1542 (e.g., machine learning models such as deep learning models) that have been pre-trained to perform AI-assisted annotation for specific organs or anomalies. These models may be further updated by using a training pipeline 1404.In at least one embodiment, the pre-installed annotation tool may improve over time as new labeled clinic data 1312 are added.

[0172] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0173] Autonomous vehicles Figure 16A shows an example of an autonomous vehicle 1600 according to at least one embodiment. In at least one embodiment, the autonomous vehicle 1600 (alternatively referred to herein as “vehicle 1600”) may be a passenger vehicle, including, but not limited to, a car, truck, bus, and / or another type of vehicle that accommodates one or more passengers. In at least one embodiment, vehicle 1a00 may be a semi-tractor-trailer truck used for transporting cargo. In at least one embodiment, vehicle 1a00 may be an aircraft, a robotic vehicle, or another type of vehicle.

[0174] Autonomous vehicles can be described in terms of automation levels as defined by the National Highway Traffic Safety Administration ("NHTSA"), a division of the U.S. Department of Transportation, and Non-Patent Document 1 of the Society of Automotive Engineers ("SAE"). In one or more embodiments, the vehicle 1600 may be capable of functionality at one or more of the autonomous driving levels from Level 1 to Level 5. For example, in at least one embodiment, the vehicle 1600 may be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment.

[0175] In at least one embodiment, the vehicle 1600 may include components such as a chassis, a vehicle body, wheels (e.g., two, four, six, eight, eighteen, etc.), tires, axles, and other components of the vehicle. In at least one embodiment, the vehicle 1600 may include a propulsion system 1650, such as an internal combustion engine, a hybrid power plant, a fully electric engine, and / or another propulsion system type. In at least one embodiment, the propulsion system 1650 may be connected to the drivetrain of the vehicle 1600, and the drivetrain may include a transmission to enable the propulsion of the vehicle 1600, but is not limited. In at least one embodiment, the propulsion system 1650 may be controlled in response to receiving a signal from one or more throttles / accelerators 1652.

[0176] In at least one embodiment, a steering system 1654, which may include a steering wheel, is used to steer the vehicle 1600 (for example, along a desired path or route) when the propulsion system 1650 is operating (for example, when the vehicle is moving). In at least one embodiment, the steering system 1654 may receive signals from (one or more) steering actuators 1656. The steering wheel may be optional for fully automated (level 5) functionality. In at least one embodiment, a brake sensor system 1646 may be used to actuate the vehicle brakes in response to receiving signals from (one or more) brake actuators 1648 and / or brake sensors.

[0177] In at least one embodiment, but not limited to, one or more controllers 1636 which may include one or more system-on-chip ("SoC") (not shown in Figure 16A) and / or one or more graphics processing units ("GPU") provide signals (for example, representing commands) to one or more components and / or systems of the vehicle 1600. For example, in at least one embodiment, one or more controllers 1636 may send signals to operate the vehicle brakes via one or more brake actuators 1648, signals to operate the steering system 1654 via one or more steering actuators 1656, and / or signals to operate the propulsion system 1650 via one or more throttle / accelerators 1652. The (one or more) controllers 1636 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving the vehicle 1600. In at least one embodiment, the (one or more) controllers 1636 may include a first controller 1636 for autonomous driving functions, a second controller 1636 for functional safety functions, a third controller 1636 for artificial intelligence functions (e.g., computer vision), a fourth controller 1636 for infotainment functions, a fifth controller 1636 for emergency redundancy, and / or other controllers. In at least one embodiment, a single controller 1636 may address two or more of the above functions, two or more controllers 1636 may address a single function, and / or any combination thereof.

[0178] In at least one embodiment, one or more controllers 1636 provide signals for controlling one or more components and / or systems of the vehicle 1600 in response to sensor data (e.g., sensor inputs) received from one or more sensors. In at least one embodiment, the sensor data may include, but are not limited to, one or more global navigation satellite system ("GNSS") sensors 1658 (e.g., one or more global positioning system sensors), one or more radar sensors 1660, one or more ultrasonic sensors 1662, one or more lithium-ion sensors 1664, or one or more inertial measurement units ("IMUs"). (unit) Sensors 1666 (e.g., (one or more) accelerometers, (one or more) gyroscopes, (one or more) magnetic compasses, (one or more) magnetometers, etc.), (one or more) microphones 1696, (one or more) stereo cameras 1668, (one or more) wide-angle cameras 1670 (e.g., fisheye cameras), (one or more) infrared cameras 1672, (one or more) ambient cameras 1674 (e.g., 360-degree cameras) The signals may be received from, for example, a long-range camera (not shown in Figure 16A), one or more medium-range cameras (not shown in Figure 16A), one or more speed sensors 1644 (for measuring the speed of the vehicle 1600), one or more vibration sensors 1642, one or more steering sensors 1640, one or more brake sensors (for example, as part of a brake sensor system 1646), and / or other sensor types.

[0179] In at least one embodiment, one or more of the controllers 1636 may receive inputs (represented, for example, by input data) from the instrument cluster 1632 of the vehicle 1600 and provide outputs (represented, for example, by output data, display data, etc.) via the human-machine interface ("HMI") display 1634, an audible annunciator, a loudspeaker, and / or other components of the vehicle 1600. In at least one embodiment, the output may include information such as vehicle speed, speed, time, map data (e.g., a high-resolution map (not shown in Figure 16A)), location data (e.g., the location of vehicle 1600 on a map), direction, location of other vehicles (e.g., occupying grid), and information about objects and the status of objects sensed by (one or more) controllers 1636. For example, in at least one embodiment, the HMI display 1634 may display information about the presence of one or more objects (e.g., road signs, warning signs, changes in traffic lights, etc.) and / or information about driving operations that the vehicle has performed, is performing, or will perform (e.g., currently changing lanes, exiting at exit 34B 3.22 km (2 miles) ahead, etc.).

[0180] In at least one embodiment, the vehicle 1600 further includes a network interface 1624, which may use (one or more) wireless antennas 1626 and / or (one or more) modems to communicate over one or more networks. For example, in at least one embodiment, the network interface 1624 may enable communication over Long-Term Evolution ("LTE"), Wideband Code Division Multiple Access ("WCDMA®"), Universal Mobile Telecommunications System ("UMTS"), Global System for Mobile Communication ("GSM"), IMT-CDMA Multi-Carrier ("CDMA2000"), and the like. Furthermore, in at least one embodiment, one or more wireless antennas 1626 may enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth Low Energy ("LE"), Zwave, ZigBee, and / or one or more low-power wide-area networks ("LPWAN") such as LoRaWAN, SigFox, etc.

[0181] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, the inference and / or training logic 715 may be used in the system of Figure 16A for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0182] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0183] Figure 16B shows an example of camera locations and fields of view for the autonomous vehicle 1600 of Figure 16A, according to at least one embodiment. In at least one embodiment, the cameras and their respective fields of view are exemplary embodiments and are not limiting. For example, in at least one embodiment, additional and / or alternative cameras may be included, and / or the cameras may be located in different locations on the vehicle 1600.

[0184] In at least one embodiment, the camera type for a camera may include, but is not limited to, a digital camera that can be adapted for use with components and / or systems of the vehicle 1600. In at least one embodiment, one or more of the (one or more) cameras may operate at Automotive Safety Integrity Level ("ASIL") B and / or another ASIL. In at least one embodiment, the camera type may be capable of any image capture rate, depending on the embodiment, such as 60 frames per second (fps), 120 fps, 240 fps, etc. In at least one embodiment, the camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In at least one embodiment, the color filter array may include a red, clear, clear, clear ("RCCC") color filter array, a red, clear, clear, blue ("RCCB") color filter array, a red, blue, green, clear ("RBGC") color filter array, a Foveon X3 color filter array, a Bayer sensor ("RGGB") color filter array, a monochrome sensor color filter array, and / or other types of color filter arrays. In at least one embodiment, a clear pixel camera may be used to increase light sensitivity, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays.

[0185] In at least one embodiment, one or more of the cameras may be used to implement advanced driver assistance system ("ADAS") functions (for example, as part of a redundant or fail-safe design). For example, in at least one embodiment, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. In at least one embodiment, one or more of the cameras (for example, all of the cameras) may simultaneously record and provide image data (for example, video).

[0186] In at least one embodiment, one or more of the cameras may be mounted in a mounting assembly, such as a custom-designed (3D printed) assembly, to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard to the windshield) that could interfere with the camera's image data capture ability. Referring to a door mirror mounting assembly, in at least one embodiment, the door mirror assembly may be custom 3D printed so that the camera mounting plate matches the shape of the door mirror. In at least one embodiment, one or more cameras may be integrated into the door mirror. For side-view cameras, one or more cameras may also be integrated in at least one embodiment within the four pillars at each corner of the cabin.

[0187] In at least one embodiment, a camera having a field of view including a portion of the environment in front of the vehicle 1600 (e.g., a front camera) may be used for a perimeter view to help identify the path and obstacles ahead and to assist in providing information essential for generating an occupied grid and / or determining a preferred vehicle path, with the help of one or more of the controllers 1636 and / or control SoCs. In at least one embodiment, the front camera may be used to implement many of the same ADAS functions as LIDAR, including, but not limited to, emergency braking, pedestrian detection, and collision avoidance. In at least one embodiment, the front camera may also be used for ADAS functions and systems, including, but not limited to, lane departure warning ("LDW"), autonomous cruise control ("ACC"), and / or traffic sign recognition.

[0188] In at least one embodiment, various cameras, including a monocular camera platform including, for example, a CMOS ("complementary metal oxide semiconductor") color imager, may be used in a front configuration. In at least one embodiment, a wide-angle camera 1670 may be used to perceive objects entering the view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-angle camera 1670 is shown in Figure 16B, in other embodiments, there may be any number of wide-angle cameras 1670 (including zero) on the vehicle 1600. In at least one embodiment, any number of long-range cameras 1698 (e.g., long-view stereo camera pairs) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. In at least one embodiment, the long-range cameras 1698 may also be used for object detection and classification, as well as basic object tracking.

[0189] In at least one embodiment, any number of stereo cameras 1668 may also be included in the front configuration. In at least one embodiment, one or more of the (one or more) stereo cameras 1668 may include an integrated control unit with a scalable processing unit, which may provide a programmable logic ("FPGA") and a multicore microprocessor having an integrated Controller Area Network ("CAN") or Ethernet interface on a single chip. In at least one embodiment, such a unit may be used to generate a 3D map of the environment of the vehicle 1600, including distance estimation for all points in the image. In at least one embodiment, one or more of the (one or more) stereo cameras 1668 may include, but are not limited to, one or more compact stereo vision sensors, which may include, but are not limited to, two camera lenses (one on the left and one on the right) and an image processing chip capable of measuring the distance from the vehicle 1600 to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. In at least one embodiment, other types of (one or more) stereo cameras 1668 may be used in addition to or as alternatives to those described herein.

[0190] In at least one embodiment, a camera having a field of view that includes a portion of the environment to the sides of the vehicle 1600 (e.g., a side-view camera) may be used for the surrounding view and provide information used to create and update the occupy grid and generate side collision warnings. For example, in at least one embodiment, one or more surrounding cameras 1674 (e.g., four surrounding cameras 1674 shown in Figure 16B) may be positioned on the vehicle 1600. In at least one embodiment, one or more surrounding cameras 1674 may include, but are not limited to, any number and combination of wide-angle cameras 1670, one or more fisheye cameras, one or more 360-degree cameras, etc. For example, in at least one embodiment, four fisheye cameras may be positioned in front of, behind, and to the sides of the vehicle 1600. In at least one embodiment, the vehicle 1600 may use three surrounding cameras 1674 (e.g., left, right, and rear) and may utilize one or more other cameras (e.g., a front camera) as a fourth surrounding view camera.

[0191] In at least one embodiment, a camera having a field of view including a portion of the environment behind the vehicle 1600 (e.g., a rear-view camera) may be used for parking assistance, surrounding view, rear collision warning, and creation and updating of the occupancy grid. In at least one embodiment, a wide variety of cameras may be used, including, but not limited to, cameras that are also suitable as (one or more) front cameras as described herein (e.g., long-range camera 1698 and / or (one or more) medium-range cameras 1676, (one or more) stereo cameras 1668, (one or more) infrared cameras 1672, etc.).

[0192] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 715 are provided below. In at least one embodiment, the inference and / or training logic 715 may be used in the system of Figure 16B for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0193] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0194] Figure 16C is a block diagram illustrating an exemplary system architecture for the autonomous vehicle 1600 of Figure 16A, according to at least one embodiment. In at least one embodiment, each of the components, features, and systems of the vehicle 1600 in Figure 16C is shown as being connected via a bus 1602. In at least one embodiment, the bus 1602 may include, but is not limited to, a CAN data interface (alternatively referred to herein as the “CAN bus”). In at least one embodiment, the CAN bus may be an internal network of the vehicle 1600 used to assist in the control of various features and functionalities of the vehicle 1600, such as brake activation, acceleration, brake control, steering, and windshield wipers. In at least one embodiment, the bus 1602 may be configured to have tens or even hundreds of nodes, each having its own unique identifier (e.g., a CAN ID). In at least one embodiment, bus 1602 may be read to find the steering angle, ground speed, engine revolutions per minute ("RPM"), button position, and / or other vehicle status indicators. In at least one embodiment, bus 1602 may be an ASIL B compliant CAN bus.

[0195] In at least one embodiment, FlexRay and / or Ethernet may be used in addition to or as an alternative to CAN. In at least one embodiment, there may be any number of buses 1602, which may include, but are not limited to, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using different protocols. In at least one embodiment, two or more buses 1602 may be used to perform different functions and / or for redundancy. For example, a first bus 1602 may be used for collision avoidance functionality and a second bus 1602 may be used for operation control. In at least one embodiment, each bus 1602 may communicate with any of the components of the vehicle 1600, and two or more buses 1602 may communicate with the same component. In at least one embodiment, each of any number of system-on-a-chip ("SoC") 1604, each of (one or more) controllers 1636, and / or each computer in the vehicle may have access to the same input data (e.g., inputs from sensors in the vehicle 1600) and may be connected to a common bus, such as a CAN bus.

[0196] In at least one embodiment, the vehicle 1600 may include one or more controllers 1636, such as those described herein with respect to Figure 16A. The (one or more) controllers 1636 may be used for a variety of functions. In at least one embodiment, the (one or more) controllers 1636 may be coupled to any of the various other components and systems of the vehicle 1600 and may be used for the control of the vehicle 1600, artificial intelligence of the vehicle 1600, infotainment for the vehicle 1600, and so on.

[0197] In at least one embodiment, the vehicle 1600 may include any number of SoCs 1604. Each of the SoCs 1604 may include, but are not limited to, a central processing unit ("CPU") 1606, a graphics processing unit ("GPU") 1608, one or more processors 1610, one or more caches 1612, one or more accelerators 1614, one or more data stores 1616, and / or other components and features not shown. In at least one embodiment, one or more SoCs 1604 may be used to control the vehicle 1600 on various platforms and systems. For example, in at least one embodiment, one or more SoCs 1604 may be combined in a system (for example, a system in a vehicle 1600) that has a high-definition ("HD") map 1622 that can receive map refreshes and / or updates from one or more servers (not shown in Figure 16C) via a network interface 1624.

[0198] In at least one embodiment, one or more CPUs 1606 may include a CPU cluster or CPU complex (alternatively referred to herein as “CCPLEX”). In at least one embodiment, one or more CPUs 1606 may include multiple cores and / or Level 2 ("L2") caches. For example, in at least one embodiment, one or more CPUs 1606 may include eight cores in a coherent multiprocessor configuration. In at least one embodiment, one or more CPUs 1606 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., 2MB of L2 cache). In at least one embodiment, one or more CPUs 1606 (e.g., CCPLEX) may be configured to support concurrent cluster operation, which allows any combination of clusters of one or more CPUs 1606 to be active at any given time.

[0199] In at least one embodiment, one or more of the (one or more) CPUs 1606 may implement power management capabilities, which include, but are not limited to, one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing an interrupt-wait ("WFI") / event-wait ("WFE") instruction; each core may be power-gated independently; each core cluster may be clock-gated independently when all cores are clock-gated or power-gated; and / or each core cluster may be power-gated independently when all cores are power-gated. In at least one embodiment, one or more CPU1606s may further implement an extended algorithm for managing power states, specifying acceptable power states and expected wake-up times, and the hardware / microcode determines the best power state to enter for the core, cluster, and CCPLEX. In at least one embodiment, the processing core may support a simple power state entry sequence in software where the work is offloaded to the microcode.

[0200] In at least one embodiment, one or more GPU1608 may include an integrated GPU (alternatively referred to herein as an “iGPU”). In at least one embodiment, one or more GPU1608 may be programmable and efficient for parallel workloads. In at least one embodiment, one or more GPU1608 may use an extended tensor instruction set. In at least one embodiment, one or more GPU1608 may include one or more streaming microprocessors, each streaming microprocessor may include a Level 1 ("L1") cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In at least one embodiment, one or more GPU1608 may include at least eight streaming microprocessors. In at least one embodiment, one or more GPU1608s may use one or more compute application programming interfaces (APIs). In at least one embodiment, one or more GPU1608s may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0201] In at least one embodiment, one or more of the (one or more) GPU1608s may be power-optimized for best performance in automotive and embedded use cases. For example, in one embodiment, the (one or more) GPU1608s may be fabricated on Finn field-effect transistors ("FinFETs"). In at least one embodiment, each streaming microprocessor may incorporate several mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In at least one embodiment, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR CORES for deep learning matrix arithmetic, a Level 0 ("L0") instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In at least one embodiment, the streaming microprocessor may include independent parallel integer and floating-point data paths to efficiently execute workloads involving a mixture of computation and addressing calculations. In at least one embodiment, the streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between parallel threads. In at least one embodiment, the streaming microprocessor may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0202] In at least one embodiment, one or more of the (one or more) GPU1608s may include high-bandwidth memory ("HBM") and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of approximately 900 GB / s in some examples. In at least one embodiment, in addition to or as an alternative to HBM memory, synchronous graphics random-access memory ("SGRAM"), such as graphics double data rate type five synchronous random-access memory ("GDDR5").

[0203] In at least one embodiment, one or more GPU1608s may include unified memory technology. In at least one embodiment, address translation service (ATS) support may be used to enable one or more GPU1608s to directly access the page tables of one or more CPU1606s. In at least one embodiment, when the memory management unit (MMU) of one or more GPU1608s encounters a miss, an address translation request may be sent to one or more CPU1606s. In at least one embodiment, in response, one or more CPU1606s may look up virtual-physical mappings for addresses in their page tables and send the translation back to one or more GPU1608s. In at least one embodiment, the unified memory technology enables a single, unified virtual address space for the memory of both (one or more) CPU1606 and (one or more) GPU1608, thereby simplifying the programming of (one or more) GPU1608 and the porting of applications to (one or more) GPU1608.

[0204] In at least one embodiment, one or more GPU1608s may include any number of access counters that can track how often one or more GPU1608s access the memory of other processors. In at least one embodiment, one or more access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently, thereby improving the efficiency of memory ranges shared between processors.

[0205] In at least one embodiment, one or more of the (one or more) SoC1604 may include any number of caches 1612, including those described herein. For example, in at least one embodiment, the (one or more) caches 1612 may include a Level 3 ("L3") cache that is available to both the (one or more) CPUs 1606 and the (one or more) GPUs 1608 (for example, connected to both the (one or more) CPUs 1606 and the (one or more) GPUs 1608). In at least one embodiment, the (one or more) caches 1612 may include a write-back cache that can track the state of the line, for example by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, the L3 cache may include 4 MB or more, depending on the embodiment, but smaller cache sizes may be used.

[0206] In at least one embodiment, one or more of the (one or more) SoC1604 may include one or more accelerators 1614 (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, the (one or more) SoC1604 may include a hardware acceleration cluster which may include optimized hardware accelerators and / or large on-chip memory. In at least one embodiment, large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. In at least one embodiment, the hardware acceleration cluster may complement the (one or more) GPU1608 and be used to offload some of the tasks of the (one or more) GPU1608 (e.g., to free up more cycles of the (one or more) GPU1608 to perform other tasks). In at least one embodiment, one or more accelerators 1614 may be used for a target workload that is stable enough to accept acceleration (e.g., perception, convolutional neural networks ("CNN"), recurrent neural networks ("RNN"), etc.). In at least one embodiment, the CNN may include region-based, i.e., regional convolutional neural networks ("RCNN"), and fast RCNNs (such as those used for object detection), or other types of CNNs.

[0207] In at least one embodiment, one or more accelerators 1614 (e.g., a hardware acceleration cluster) may include one or more deep learning accelerators ("DLAs"). One or more DLAs may include, but are not limited to, one or more Tensor Processing Units ("TPUs"), which may be configured to provide additional, 10 trillion operations per second for deep learning applications and inference. In at least one embodiment, one or more TPUs may be configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and may be accelerators optimized for that purpose. One or more DLAs may be further optimized for specific neural network types and sets of floating-point operations, as well as for inference. In at least one embodiment, a design of one or more DLAs may provide more performance per millisecond than a typical general-purpose GPU, and generally far exceed the performance of a CPU. In at least one embodiment, one or more TPUs may implement several functions, including, for example, a single-instance convolution function supporting INT8, INT16, and FP16 data types for both features and weights, as well as a post-processing function. In at least one embodiment, one or more DLAs may rapidly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including, but not limited to, a CNN for object recognition and detection using data from camera sensors, a CNN for distance estimation using data from camera sensors, a CNN for emergency vehicle detection and identification and detection using data from microphone 1696, a CNN for face recognition and vehicle owner identification using data from camera sensors, and / or a CNN for security and / or safety-related events.

[0208] In at least one embodiment, one or more DLAs may perform any function of one or more GPUs 1608, and for example, by using inference accelerators, the designer may target either one or more DLAs or one or more GPUs 1608 for any function. For example, in at least one embodiment, the designer may concentrate CNN and floating-point arithmetic processing on one or more DLAs and offload other functions to one or more GPUs 1608 and / or one or more other accelerators 1614.

[0209] In at least one embodiment, one or more accelerators 1614 (e.g., a hardware acceleration cluster) may include one or more programmable vision accelerators ("PVAs"), which may be referred to herein as alternatively as computer vision accelerators. In at least one embodiment, one or more PVAs may be designed and configured to accelerate computer vision algorithms for advanced driver-assistance systems ("ADAS") 1638, autonomous driving, augmented reality ("AR") applications, and / or virtual reality ("VR") applications. One or more PVAs may provide a balance between performance and flexibility. For example, in at least one embodiment, each PVA may include, for example, any number of reduced instruction set computer ("RISC") cores, direct memory access ("DMA"), and / or any number of vector processors.

[0210] In at least one embodiment, the RISC core may interact with an image sensor (e.g., an image sensor of any of the cameras described herein), one or more image signal processors, and so on. In at least one embodiment, each of the RISC cores may include any amount of memory. In at least one embodiment, the RISC core may use one of several protocols, depending on the embodiment. In at least one embodiment, the RISC core may run a real-time operating system ("RTOS"). In at least one embodiment, the RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits ("ASICs"), and / or memory devices. For example, in at least one embodiment, the RISC core may include an instruction cache and / or tightly coupled RAM.

[0211] In at least one embodiment, the DMA may enable one or more PVA components to access system memory independently of one or more CPUs 1606. In at least one embodiment, the DMA may support any number of features used to provide optimization to the PVA, including, but not limited to, support for multidimensional addressing and / or circular addressing. In at least one embodiment, the DMA may support up to six or more addressing dimensions, which may include, but not limited to, block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0212] In at least one embodiment, a vector processor is a programmable processor that can be designed to efficiently and flexibly perform programming for computer vision algorithms and can provide signal processing capabilities. In at least one embodiment, the PVA may include a PVA core and two vector processing subsystem partitions. In at least one embodiment, the PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, the vector processing subsystem may act as the primary processing engine of the PVA and may include a vector processing unit ("VPU"), an instruction cache, and / or vector memory (e.g., "VMEM"). In at least one embodiment, the VPU may include a digital signal processor, such as a single instruction, multiple data ("SIMD") or very long instruction word ("VLIW") digital signal processor. In at least one embodiment, a combination of SIMD and VLIW may improve throughput and speed.

[0213] In at least one embodiment, each vector processor may include an instruction cache and be coupled to dedicated memory. As a result, in at least one embodiment, each vector processor may be configured to operate independently of other vector processors. In at least one embodiment, vector processors included in a particular PVA may be configured to employ data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In at least one embodiment, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on a contiguous image or portion of an image. In at least one embodiment, among other things, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In at least one embodiment, (one or more) PVAs may include additional error correction code ("ECC") memory to improve the overall safety of the system.

[0214] In at least one embodiment, one or more accelerators 1614 (e.g., a hardware acceleration cluster) may include a computer vision network on-chip and static random access memory ("SRAM") to provide high-bandwidth, low-latency SRAM for one or more accelerators 1614. In at least one embodiment, the on-chip memory may include, for example, at least 4 MB of SRAM consisting of eight, but not limited to, eight field-configurable memory blocks, which may be accessible by both the PVA and DLA. In at least one embodiment, each pair of memory blocks may include an advanced peripheral bus ("APB") interface, configurable circuit elements, a controller, and a multiplexer. In at least one embodiment, any type of memory may be used. In at least one embodiment, the PVA and DLA may access the memory via a backbone that provides high-speed access to the memory. In at least one embodiment, the backbone may include a computer vision network on a chip that interconnects the PVA and DLA to memory (for example, using an APB).

[0215] In at least one embodiment, the computer vision network on chip may include an interface in which both the PVA and DLA determine to provide ready and enable signals before any control signals / addresses / data are transmitted. In at least one embodiment, the interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. In at least one embodiment, the interface may conform to the International Organization for Standardization (ISO) 26262 or the International Electrotechnical Commission (IEC) 61508 standard, but other standards and protocols may be used.

[0216] In at least one embodiment, one or more of the (one or more) SoC1604s may include a real-time ray tracing hardware accelerator. In at least one embodiment, the real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and extent of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general waveform propagation simulation, comparison with LIDAR data for localization and / or other functions, and / or other uses.

[0217] In at least one embodiment, one or more accelerators 1614 (e.g., a hardware accelerator cluster) have diverse uses for autonomous driving. In at least one embodiment, the PVA may be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of the PVA are well matched for algorithmic domains requiring predictable processing at low power and low latency. In other words, the PVA performs well even on small datasets for semi-dense or dense regular computations that require predictable runtime along with low latency and low power. In at least one embodiment, in an autonomous vehicle such as vehicle 1600, the PVA is designed to run conventional computer vision algorithms because they are efficient in object detection and integer numerical computation.

[0218] For example, according to at least one embodiment of the technology, PVA is used to implement computer stereo vision. In at least one embodiment, a semi-global matching-based algorithm may be used in some examples, but is not limited to this. In at least one embodiment, an application for Level 3–5 autonomous driving uses motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.) on the fly. In at least one embodiment, PVA may implement computer stereo vision functionality for input from two monocular cameras.

[0219] In at least one embodiment, PVA may be used to perform high-density optical flow. For example, in at least one embodiment, PVA may process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR data. In at least one embodiment, PVA may be used for time-of-flight depth processing by processing raw time-of-flight data to provide processed time-of-flight data.

[0220] In at least one embodiment, DLA may be used to power any type of network for improving control and driving safety, including, for example, a neural network that outputs a measure of confidence for each object detection. In at least one embodiment, confidence may be expressed or interpreted as the probability of each detection compared to other detections, or as providing its relative “weight.” In at least one embodiment, confidence allows the system to make further determinations about which detections should be considered true positive detections rather than false positive detections. For example, in at least one embodiment, the system may set a threshold for confidence and consider only detections exceeding the threshold as true positive detections. In embodiments where an automatic emergency braking (“AEB”) system is used, a false positive detection would cause the vehicle to automatically apply the emergency brakes, which is obviously undesirable. In at least one embodiment, a highly reliable detection may be considered a trigger for the AEB. In at least one embodiment, DLA may power a neural network to regress confidence values. In at least one embodiment, the neural network may take as its input at least a subset of parameters, including, among other things, the dimensions of the bounding box, ground plane estimates obtained (e.g., from another subsystem), outputs from one or more IMU sensors 1666 correlated with the orientation of the vehicle 1600, distance, and 3D location estimates of objects obtained from the neural network and / or other sensors (e.g., one or more LiDAR sensors 1664 or one or more RADAR sensors 1660).

[0221] In at least one embodiment, one or more of the (one or more) SoC1604 may include (one or more) datastores 1616 (e.g., memory). In at least one embodiment, the (one or more) datastores 1616 may be on-chip memory of the (one or more) SoC1604, which may store neural networks to be run on the (one or more) GPUs 1608 and / or DLA. In at least one embodiment, the capacity of the (one or more) datastores 1616 may be large enough to store multiple instances of the neural network for redundancy and safety. In at least one embodiment, the (one or more) datastores 1616 may include (one or more) L2 or L3 caches.

[0222] In at least one embodiment, one or more of the (one or more) SoC1604 may include any number of (one or more) processors 1610 (e.g., embedded processors). In at least one embodiment, the (one or more) processors 1610 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions and related security enforcement. In at least one embodiment, the boot and power management processor may be part of the boot sequence of the (one or more) SoC1604 and may provide runtime power management services. In at least one embodiment, the boot power and management processor may provide clock and voltage programming, assistance with system low-power state transitions, management of thermal and temperature sensors of the (one or more) SoC1604, and / or management of the power state of the (one or more) SoC1604. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and (one or more) SoC 1604 may use the ring oscillator to detect the temperatures of (one or more) CPU 1606, (one or more) GPU 1608, and / or (one or more) accelerator 1614. In at least one embodiment, if it is determined that the temperature exceeds a threshold, the boot and power management processor may enter a temperature fault routine, which may put (one or more) SoC 1604 into a low-power state and / or put the vehicle 1600 into chauffeur-to-safe stop mode (e.g., to safely stop the vehicle 1600).

[0223] In at least one embodiment, the (one or more) processors 1610 may further include a set of embedded processors that can function as an audio processing engine. In at least one embodiment, the audio processing engine may be an audio subsystem that enables full hardware support for multi-channel audio via multiple interfaces and for a wide range of flexible audio I / O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0224] In at least one embodiment, the (one or more) processors 1610 may further include an always-on processor engine that can provide the hardware features necessary to support low-power sensor management and startup use cases. In at least one embodiment, the always-on processor engine may include, but is not limited to, a processor core, tightly coupled RAM, supporting peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0225] In at least one embodiment, the (one or more) processor 1610 may further include a safety cluster engine, which may include, but is not limited to, a dedicated processor subsystem for addressing safety management for automotive applications. In at least one embodiment, the safety cluster engine may include, but is not limited to, two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may, in at least one embodiment, operate in lockstep mode and function as a single core with comparison logic for detecting any differences between their operations. In at least one embodiment, the (one or more) processor 1610 may further include a real-time camera engine, which may include, but is not limited to, a dedicated processor subsystem for addressing real-time camera management. In at least one embodiment, the processor 1610 (one or more) may further include a high dynamic range signal processor, which may include, but is not limited to, an image signal processor that is a hardware engine that is part of a camera processing pipeline.

[0226] In at least one embodiment, the (one or more) processor 1610 may include a video image synthesizer, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce a final image for a player window. In at least one embodiment, the video image synthesizer may perform lens distortion correction on the (one or more) wide-angle camera 1670, the (one or more) ambient camera 1674, and / or the (one or more) cabin surveillance camera sensors. In at least one embodiment, the (one or more) cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the (one or more) SoC 1604, which is configured to identify and respond to events within the cabin. In at least one embodiment, the in-cabin system may perform lip-reading to activate cellular services, make phone calls, write emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. In at least one embodiment, some functions are available to the driver when the vehicle is operating in autonomous mode and are unavailable in other cases.

[0227] In at least one embodiment, the video image synthesizer may include extended temporal noise reduction for both spatial and temporal noise reduction. For example, in at least one embodiment, if motion occurs in the video, the noise reduction appropriately weights the spatial information and reduces the weight of the information provided by adjacent frames. In at least one embodiment, if the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer may use information from previous images to reduce noise in the current image.

[0228] In at least one embodiment, the video image synthesizer may also be configured to perform stereo rectification on the input stereo lens frame. In at least one embodiment, the video image synthesizer may be further used for user interface compositing when the operating system desktop is in use, and (one or more) GPU1608s are not required to continuously render new surfaces. In at least one embodiment, when (one or more) GPU1608s are powered on, active, and performing 3D rendering, the video image synthesizer may be used to offload (one or more) GPU1608s to improve performance and responsiveness.

[0229] In at least one embodiment, one or more of the SoC1604s may further include a Mobile Industry Processor Interface ("MIPI") camera serial interface, a high-speed interface, and / or a video input block that can be used for the camera and associated pixel input functions to receive video and input from the camera. In at least one embodiment, one or more of the SoC1604s may further include one or more input / output controllers, which may be controlled by software and may be used to receive I / O signals not committed to a specific role.

[0230] In at least one embodiment, one or more of the (one or more) SoC1604 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio encoders / decoders ("codecs"), power management, and / or other devices. The (one or more) SoC1604 may be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet), data from sensors (e.g., one or more) LiDAR sensors 1664, one or more RADAR sensors 1660, etc., which may be connected via Ethernet), data from bus 1602 (e.g., vehicle speed, steering wheel position, etc.), data from one or more GNSS sensors 1658 (e.g., connected via Ethernet or CAN bus), and the like. In at least one embodiment, one or more of the (one or more) SoC1604 may further include a dedicated high-performance mass storage controller, which may include its own DMA engine and may be used to free up the (one or more) CPU1606 from routine data management tasks.

[0231] In at least one embodiment, one or more SoC1604s may be an end-to-end platform with a flexible architecture spanning automation levels 3–5, thereby providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and a platform for a flexible, reliable driving software stack, along with deep learning tools. In at least one embodiment, one or more SoC1604s may be faster, more reliable, and more energy-efficient and space-efficient than conventional systems. For example, in at least one embodiment, one or more accelerators 1614, when combined with one or more CPUs 1606, one or more GPUs 1608, and one or more data stores 1616, may provide a fast and efficient platform for Level 3–5 autonomous vehicles.

[0232] In at least one embodiment, a computer vision algorithm may run on a CPU, and this algorithm may be configured using a high-level programming language such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, in at least one embodiment, the CPU often fails to meet the performance requirements of many computer vision applications, such as requirements related to execution time and power consumption. In at least one embodiment, many CPUs are unable to execute complex object detection algorithms in real time, as used in in-vehicle ADAS applications and in actual Level 3-5 autonomous vehicles.

[0233] The embodiments described herein enable multiple neural networks to be implemented simultaneously and / or sequentially, allowing the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, in at least one embodiment, a DLA or a CNN running on a separate GPU (e.g., one or more GPU1620s) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. In at least one embodiment, the DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of the signs, which can then pass that semantic understanding to a route planning module running on a CPU complex.

[0234] In at least one embodiment, multiple neural networks may be operating simultaneously with respect to Level 3, 4, or 5 driving. For example, in at least one embodiment, a warning sign consisting of an electric light and the text "Caution: Flashing light indicates icy condition" may be interpreted independently or collectively by several neural networks. In at least one embodiment, the sign itself may be identified as a traffic sign by a first introduced neural network (e.g., a trained neural network), and the text "Flashing light indicates icy condition" may be interpreted by a second introduced neural network, which informs the vehicle's route planning software (preferably running on the CPU complex) that an icy condition is present when a flashing light is detected. In at least one embodiment, the flashing light may be identified by running a third introduced neural network over multiple frames, which informs the vehicle's route planning software of the presence (or absence) of the flashing light. In at least one embodiment, all three neural networks may run simultaneously within the DLA and / or on one or more GPU1608s, etc.

[0235] In at least one embodiment, a CNN for facial recognition and vehicle owner identification may use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 1600. In at least one embodiment, an always-on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and in security mode to disable the vehicle when the owner leaves the vehicle. In this way, (one or more) SoC1604 provides security against theft and / or vehicle hijacking.

[0236] In at least one embodiment, a CNN for emergency vehicle detection and identification may use data from microphone 1696 to detect and identify emergency vehicle sirens. In at least one embodiment, one or more SoCs 1604 use a CNN to classify environmental and urban sounds, as well as visual data. In at least one embodiment, a CNN running on a DLA is trained to identify the relative speed at which an emergency vehicle is approaching (for example, by using the Doppler effect). In at least one embodiment, a CNN may also be trained to identify emergency vehicles specific to the area in which the vehicle is operating, as identified by one or more GNSS sensors 1658. In at least one embodiment, when operating in Europe, the CNN attempts to detect European sirens, and when in the United States, the CNN attempts to identify only North American sirens. In at least one embodiment, when an emergency vehicle is detected, a control program may be used to execute an emergency vehicle safety routine, slow down the vehicle, pull over to the side of the road, stop the vehicle, and / or idle the vehicle in conjunction with one or more ultrasonic sensors 1662 until (one or more) emergency vehicles have passed.

[0237] In at least one embodiment, the vehicle 1600 may include one or more CPUs 1618 (e.g., one or more individual CPUs, or one or more dCPUs), and the one or more CPUs 1618 may be coupled to one or more SoCs 1604 via a high-speed interconnect (e.g., PCIe). In at least one embodiment, the one or more CPUs 1618 may include, for example, an x86 processor. The one or more CPUs 1618 may be used to perform any of a variety of functions, including, for example, mediating potentially inconsistent results between ADAS sensors and one or more SoCs 1604, and / or monitoring the status and health of one or more controllers 1636 and / or the infotainment system ("Infotainment SoC") 1630 on the chip.

[0238] In at least one embodiment, the vehicle 1600 may include (one or more) GPUs 1620 (e.g., (one or more) individual GPUs, or (one or more) dGPUs), and the (one or more) GPUs 1620 may be coupled to the (one or more) SoCs 1604 via a high-speed interconnect (e.g., NVIDIA NVLINK). In at least one embodiment, the (one or more) GPUs 1620 may provide additional artificial intelligence functionality, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based at least in part on input from the vehicle 1600's sensors (e.g., sensor data).

[0239] In at least one embodiment, vehicle 1600 may further include a network interface 1624, which may include, but is not limited to, one or more wireless antennas 1626 (e.g., one or more wireless antennas 1626 for different communication protocols, such as a cellular antenna or a Bluetooth antenna). In at least one embodiment, the network interface 1624 may be used to enable wireless connectivity over the Internet to the cloud (e.g., one or more servers and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). In at least one embodiment, a direct link may be established between vehicle 160 and other vehicles for communication with other vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). In at least one embodiment, the direct link may be provided using an inter-vehicle communication link. The vehicle-to-vehicle communication link may provide vehicle 1600 with information about nearby vehicles (for example, vehicles in front of, to the side of, and / or behind vehicle 1600). In at least one embodiment, the above functionality may be part of the cooperative adaptive driving control functionality of vehicle 1600.

[0240] In at least one embodiment, the network interface 1624 may include an SoC that provides modulation and demodulation functionality, enabling one or more controllers 1636 to communicate over a wireless network. In at least one embodiment, the network interface 1624 may include a radio frequency front end for baseband-to-radio frequency up-conversion and radio frequency-to-baseband down-conversion. In at least one embodiment, frequency conversion may be carried out in any technically feasible manner. For example, frequency conversion may be carried out through a well-known process and / or using a superheterodyne process. In at least one embodiment, the radio frequency front end functionality may be provided by a separate chip. In at least one embodiment, the network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0241] In at least one embodiment, the vehicle 1600 may further include one or more data stores 1628, which may include, but are not limited to, off-chip (e.g., not on one or more SoCs 1604) storage. In at least one embodiment, the data stores 1628 may include, but are not limited to, one or more storage elements, which include RAM, SRAM, dynamic random-access memory ("DRAM"), video random-access memory ("VRAM"), flash, hard disks, and / or other components and / or devices capable of storing at least one bit of data.

[0242] In at least one embodiment, the vehicle 1600 may further include (one or more) GNSS sensors 1658 (e.g., GPS and / or auxiliary GPS sensors) to assist mapping, perception, occupy grid generation, and / or route planning functions. In at least one embodiment, any number of GNSS sensors 1658 may be used, including, for example, a GPS using a USB connector with an Ethernet-to-serial (e.g., RS-232) bridge.

[0243] In at least one embodiment, the vehicle 1600 may further include one or more RADAR sensors 1660. The one or more RADAR sensors 1660 may be used by the vehicle 1600 for long-range vehicle detection, even in darkness and / or severe weather conditions. In at least one embodiment, the functional safety level of the RADAR may be ASIL B. In some examples, the one or more RADAR sensors 1660 may use CAN and / or bus 1602 for control (e.g., to transmit data generated by the one or more RADAR sensors 1660) and for accessing object tracking data, along with access to Ethernet for accessing raw data. In at least one embodiment, a wide variety of RADAR sensor types may be used. For example, but not limited to, the one or more RADAR sensors 1660 may be preferred for forward, rear, and side RADAR use. In at least one embodiment, one or more of the (one or more) RADAR sensors 1660 are (one or more) pulsed Doppler RADAR sensors.

[0244] In at least one embodiment, the (one or more) RADAR sensors 1660 may include different configurations, such as narrow-field long-range, wide-field short-range, and short-range lateral coverage. In at least one embodiment, the long-range RADAR may be used for adaptive driving control functionality. In at least one embodiment, the long-range RADAR system may provide a wide field of view achieved by two or more independent scans, such as within a 250m range. In at least one embodiment, the (one or more) RADAR sensors 1660 may help distinguish between static and moving objects and may be used by the ADAS system 1638 for emergency braking assistance and forward collision warning. The (one or more) sensors 1660 included in the long-range RADAR system may include, but are not limited to, multiple (e.g., six or more) fixed RADAR antennas, as well as monostatic and multimodal RADARs with high-speed CAN and FlexRay interfaces. In at least one embodiment, if there are six antennas, the four central antennas may create a focused beam pattern designed to record the area around vehicle 1600 at a faster speed with minimal interference from traffic in adjacent lanes. In at least one embodiment, the other two antennas may expand the field of view, which may allow for the rapid detection of vehicles entering or leaving the lane of vehicle 1600.

[0245] In at least one embodiment, the medium-range RADAR system may, as an example, include a range of up to 160 m (forward) or 80 m (rear) and a field of view of up to 42 degrees (forward) or 150 degrees (rear). In at least one embodiment, the short-range RADAR system may include, but not limited to, any number of RADAR sensors 1660 designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, in at least one embodiment, the RADAR sensor system may create two beams that constantly monitor the blind spots behind and beside the vehicle. In at least one embodiment, the short-range RADAR system may be used in an ADAS system 1638 for blind spot detection and / or lane change assistance.

[0246] In at least one embodiment, the vehicle 1600 may further include one or more ultrasonic sensors 1662. One or more ultrasonic sensors 1662, which can be positioned in front of, behind, and / or to the side of the vehicle 1600, may be used for parking assistance and / or to create and update the occupancy grid. In at least one embodiment, a wide variety of one or more ultrasonic sensors 1662 may be used, and different ultrasonic sensors 1662 may be used for different detection ranges (e.g., 2.5m, 4m). In at least one embodiment, one or more ultrasonic sensors 1662 may operate at functional safety level ASIL B.

[0247] In at least one embodiment, the vehicle 1600 may include one or more LiDAR sensors 1664. These LiDAR sensors 1664 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, the LiDAR sensors 1664 may have a functional safety level of ASIL B. In at least one embodiment, the vehicle 1600 may include multiple LiDAR sensors 1664 (e.g., two, four, six, etc.), and these LiDAR sensors 1664 may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0248] In at least one embodiment, one or more LiDAR sensors 1664 may be capable of providing a list of objects and their distances over a 360-degree field of view. In at least one embodiment, one or more commercially available LiDAR sensors 1664 may have an advertised range of approximately 100m, for example, with an accuracy of 2cm to 3cm and support for a 100Mbps Ethernet connection. In at least one embodiment, one or more non-protruding LiDAR sensors 1664 may be used. In such embodiments, one or more LiDAR sensors 1664 may be implemented as small devices that can be incorporated into the front, rear, side, and / or corners of a vehicle 1600. In at least one embodiment, one or more LiDAR sensors 1664 may, in such embodiments, provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees over a range of 200m, even for low-reflectivity objects. In at least one embodiment, one or more front-mounted LiDAR sensors 1664 may be configured for a horizontal field of view between 45 and 135 degrees.

[0249] In at least one embodiment, LiDAR technology such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a transmission source to illuminate the area around the vehicle 1600 up to approximately 200m. In at least one embodiment, the flash LiDAR unit includes, but is not limited to, a receptor which records the transit time of the laser pulse and the reflected light on each pixel, corresponding to the range from the vehicle 1600 to the object. In at least one embodiment, the flash LiDAR may enable the generation of highly accurate and distortion-free ambient images with each laser flash. In at least one embodiment, four flash LiDAR sensors may be introduced, one on each side of the vehicle 1600. In at least one embodiment, the 3D flash LiDAR system includes, but is not limited to, a solid-state 3D staring array LiDAR camera (e.g., a non-scanning LiDAR device) with no moving parts other than a fan. In at least one embodiment, one or more flash LiDAR devices may use Class I (eye-safe) laser pulses of 5 nanoseconds per frame and capture reflected laser light in the form of a 3D-range point cloud and position-synchronized (co-registered) intensity data.

[0250] In at least one embodiment, the vehicle may further include one or more IMU sensors 1666. In at least one embodiment, the one or more IMU sensors 1666 may be located in the center of the rear axle of the vehicle 1600. In at least one embodiment, the one or more IMU sensors 1666 may include, for example, one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In at least one embodiment, such as in a 6-axis application, the one or more IMU sensors 1666 may include, for example, an accelerometer and a gyroscope. In at least one embodiment, such as in a 9-axis application, the one or more IMU sensors 1666 may include, for example, an accelerometer, a gyroscope, and a magnetometer.

[0251] In at least one embodiment, the (one or more) IMU sensors 1666 may be implemented as a small, high-performance GPS-Aided Inertial Navigation System ("GPS / INS") that combines micro-electro-mechanical systems ("MEMS") inertial sensors, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. In at least one embodiment, the (one or more) IMU sensors 1666 may enable a vehicle 1600 to estimate its bearing without requiring input from magnetic sensors by directly observing changes in velocity and correlating them from GPS to the (one or more) IMU sensors 1666. In at least one embodiment, the (one or more) IMU sensors 1666 and the (one or more) GNSS sensors 1658 may be combined in a single integrated unit.

[0252] In at least one embodiment, the vehicle 1600 may include one or more microphones 1696 placed inside and / or around the vehicle 1600. In at least one embodiment, one or more microphones 1696 may, among other things, be used for emergency vehicle detection and identification.

[0253] In at least one embodiment, the vehicle 1600 may further include any number of camera types, including (one or more) stereo cameras 1668, (one or more) wide-angle cameras 1670, (one or more) infrared cameras 1672, (one or more) ambient cameras 1674, (one or more) long-range cameras 1698, (one or more) medium-range cameras 1676, and / or other camera types. In at least one embodiment, cameras may be used to capture image data around the entire periphery of the vehicle 1600. In at least one embodiment, the type of camera used depends on the vehicle 1600. In at least one embodiment, any combination of camera types may be used to provide the required coverage around the vehicle 1600. In at least one embodiment, the number of cameras may vary depending on the embodiment. For example, in at least one embodiment, the vehicle 1600 may include six cameras, seven cameras, ten cameras, twelve cameras, or any other number of cameras. The camera may, in one example, but not limited to, support Gigabit Multimedia Serial Link ("GMSL") and / or Gigabit Ethernet. In at least one embodiment, each of the (one or more) cameras has been described in more detail previously herein with respect to Figures 16A and 16B.

[0254] In at least one embodiment, the vehicle 1600 may further include one or more vibration sensors 1642. In at least one embodiment, one or more vibration sensors 1642 may measure vibrations of components of the vehicle 1600, such as one or more axles. For example, in at least one embodiment, a change in vibration may indicate a change in the road surface. In at least one embodiment, when two or more vibration sensors 1642 are used, the difference in vibration may be used to determine the amount of friction or slip of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0255] In at least one embodiment, the vehicle 1600 may include an ADAS system 1638. The ADAS system 1638 may include a SoC in some examples, but is not limited to. In at least one embodiment, the ADAS system 1638 may include, but not limited to, any number and combination of autonomous / adaptive / automatic cruise control ("ACC") systems, cooperative adaptive cruise control ("CACC") systems, forward crash warning ("FCW") systems, automatic emergency braking ("AEB") systems, lane departure warning ("LDW") systems, lane keep assist ("LKA") systems, blind spot warning ("BSW") systems, rear cross-traffic warning ("RCTW") systems, collision warning ("CW") systems, lane centering ("LC") systems, and / or other systems, features, and / or functionalities.

[0256] In at least one embodiment, the ACC system may use (one or more) RADAR sensors 1660, (one or more) LIDAR sensors 1664, and / or any number of cameras. In at least one embodiment, the ACC system may include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, the longitudinal ACC system monitors and controls the distance of vehicle 1600 to the vehicle immediately in front and automatically adjusts the speed of vehicle 1600 to maintain a safe distance from the vehicle ahead. In at least one embodiment, the lateral ACC system implements distance maintenance and advises vehicle 1600 to change lanes when necessary. In at least one embodiment, the lateral ACC is relevant to other ADAS applications such as LC and CW.

[0257] In at least one embodiment, the CACC system uses information from other vehicles, which may be received from other vehicles via a network interface 1624 and / or (one or more) wireless antennas 1626, either wirelessly or indirectly via a network connection (e.g., via the Internet). In at least one embodiment, a direct link may be provided by a vehicle-to-vehicle ("V2V") communication link, and an indirect link may be provided by an infrastructure-to-vehicle ("I2V") communication link. Generally, the V2V communication concept provides information about the immediately preceding vehicle (e.g., a vehicle in the same lane immediately before vehicle 1600), and the I2V communication concept provides information about traffic further ahead. In at least one embodiment, the CACC system may include either or both I2V and V2V information sources. In at least one embodiment, if information about the vehicle in front of vehicle 1600 is available, the CACC system may become more reliable, which could improve the smoothness of traffic flow and reduce congestion on the road.

[0258] In at least one embodiment, the FCW system is designed to alert the driver about hazardous materials, thereby enabling the driver to take corrective action. In at least one embodiment, the FCW system uses a front camera and / or (one or more) radar sensors 1660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration component. In at least one embodiment, the FCW system may provide warnings in the form of sound, visual warnings, vibration, and / or quick brake pulses.

[0259] In at least one embodiment, the AEB system may detect an impending forward collision with another vehicle or other object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. In at least one embodiment, the AEB system may use one or more front cameras and / or one or more RADAR sensors 1660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. In at least one embodiment, when the AEB system detects a hazard, the AEB system typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system may automatically apply the brakes to prevent the anticipated collision or at least mitigate its impact. In at least one embodiment, the AEB system may include techniques such as dynamic brake support and / or pre-crash braking.

[0260] In at least one embodiment, the LDW system provides visual, auditory, and / or tactile warnings, such as vibration of the steering wheel or seat, to alert the driver when the vehicle 1600 crosses a lane marker. In at least one embodiment, the LDW system does not activate when the driver indicates intentional lane departure by activating the turn signal. In at least one embodiment, the LDW system may use a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration component. In at least one embodiment, the LKA system is a variation of the LDW system. The LKA system provides steering input or brake control to correct the vehicle 1600 if it begins to move out of its lane.

[0261] In at least one embodiment, the BSW system detects vehicles in the vehicle's blind spot and warns the driver about those vehicles. In at least one embodiment, the BSW system may provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is not safe. In at least one embodiment, the BSW system may provide additional warnings when the driver uses the turn signal. In at least one embodiment, the BSW system may use one or more rear-facing cameras and / or one or more RADAR sensors 1660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0262] In at least one embodiment, the RCTW system may provide visual, auditory, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle 1600 is reversing. In at least one embodiment, the RCTW system includes an AEB system to ensure that the vehicle brakes are applied to avoid a crash. In at least one embodiment, the RCTW system may use one or more rear-facing radar sensors 1660 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0263] In at least one embodiment, conventional ADAS systems may be prone to producing false positive results, which can be annoying and distracting to the driver, but this is usually not a major issue as conventional ADAS systems alert the driver and allow the driver to determine whether safety conditions truly exist and act accordingly. In at least one embodiment, the vehicle 1600 itself determines, in the event of conflicting results, whether to follow the result from a primary computer (e.g., a first controller 1636) or a secondary computer (e.g., a second controller 1636). For example, in at least one embodiment, the ADAS system 1638 may be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. In at least one embodiment, the backup computer rationality monitor may run redundant and varied software on hardware components to detect failures in perceptual and dynamic driving tasks. In at least one embodiment, the output from the ADAS system 1638 may be provided to a supervisory MCU. In at least one embodiment, if the output from the primary computer and the output from the secondary computer are inconsistent, the supervising MCU determines how to reconcile the inconsistency to ensure safe operation.

[0264] In at least one embodiment, the primary computer may be configured to provide the supervising MCU with a reliability score indicating the reliability of the primary computer in the selected outcome. In at least one embodiment, if the reliability score exceeds a threshold, the supervising MCU may follow the instructions of the primary computer, regardless of whether the secondary computer provides contradictory or inconsistent results. In at least one embodiment, if the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., contradictory), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0265] In at least one embodiment, the supervising MCU may be configured to operate one or more neural networks trained and configured to determine, at least in part, the conditions under which a secondary computer provides a false alarm, based on the outputs from the primary and secondary computers. In at least one embodiment, one or more neural networks in the supervising MCU may learn when the output of the secondary computer can be trusted and when it cannot. For example, in at least one embodiment, when the secondary computer is a radar-based FCW system, one or more neural networks in the supervising MCU may learn when the FCW system identifies a metallic object that is not actually a hazard, such as a drain grate or manhole cover, which triggers an alarm. In at least one embodiment, when the secondary computer is a camera-based LDW system, the neural network in the supervising MCU may learn to disable the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In at least one embodiment, the supervisor MCU may include at least one DLA or GPU suitable for running (one or more) neural networks together with associated memory. In at least one embodiment, the supervisor MCU may comprise and / or be included as a component of (one or more) SoC1604.

[0266] In at least one embodiment, the ADAS system 1638 may include a secondary computer that implements ADAS functionality using conventional computer vision rules. In at least one embodiment, the secondary computer may use conventional computer vision rules (if-then), and the presence of (one or more) neural networks in the supervising MCU may improve reliability, safety, and performance. For example, in at least one embodiment, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly against failures caused by software (or software-hardware interface) functionality. For example, in at least one embodiment, if there is a software bug or error in the software running on the primary computer, and non-identical software code running on the secondary computer provides the same overall result, the supervising MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer has not caused a critical error.

[0267] In at least one embodiment, the output of the ADAS system 1638 may be fed to a perception block and / or a dynamic driving task block of the primary computer. For example, in at least one embodiment, if the ADAS system 1638 indicates a forward crash warning due to an object immediately preceding, the perception block may use this information when identifying the object. In at least one embodiment, the secondary computer may have its own trained, and therefore false-positive-reducing, neural network, as described herein.

[0268] In at least one embodiment, the vehicle 1600 may further include an infotainment SoC 1630 (for example, an in-vehicle infotainment system (IVI)). Although shown and described as an SoC, the infotainment system 1630 may not be an SoC in at least one embodiment and may include, but not limited to, two or more separate components. In at least one embodiment, the infotainment SoC 1630 may include, but is not limited to, a combination of hardware and software which may be used to provide the vehicle 1600 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., a navigation system, rear parking assist, wireless data system, vehicle-related information such as fuel level, total mileage, brake fuel level, oil level, door open / closed, air filter information, etc.). For example, the infotainment SoC 1630 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control, hands-free voice control, heads-up display ("HUD"), HMI display 1634, telematics devices, control panels (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. In at least one embodiment, the infotainment SoC 1630 may be further used to provide information (for example, visual and / or auditory) to the vehicle's (one or more) users, such as information from the ADAS system 1638, autonomous driving information such as planned vehicle operation and trajectory, ambient information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0269] In at least one embodiment, the infotainment SoC 1630 may include any amount and type of GPU functionality. In at least one embodiment, the infotainment SoC 1630 may communicate with other devices, systems, and / or components of the vehicle 1600 via a bus 1602 (e.g., a CAN bus, Ethernet, etc.). In at least one embodiment, the infotainment SoC 1630 may be coupled to a supervisory MCU so that if one or more primary controllers 1636 (e.g., the primary and / or backup computers of the vehicle 1600) fail, the GPU of the infotainment system may perform some self-driving functions. In at least one embodiment, the infotainment SoC 1630 may put the vehicle 1600 into a driver-safe stop mode as described herein.

[0270] In at least one embodiment, the vehicle 1600 may further include an instrument cluster 1632 (e.g., a digital dashboard, electronic instrument cluster, digital instrument panel, etc.). In at least one embodiment, the instrument cluster 1632 may include, but is not limited to, a controller and / or supercomputer (e.g., a separate controller or supercomputer). In at least one embodiment, the instrument cluster 1632 may include, but is not limited to, a set of instrumentation such as a speedometer, fuel level, oil pressure, tachometer, odometer, direction indicator, shift lever position indicator, (one or more) seat belt warning lights, (one or more) parking brake warning lights, (one or more) engine fault lights, auxiliary restraint system (e.g., airbag) information, light control, safety system control, navigation information, etc. In some examples, the information may be displayed and / or shared between the infotainment SoC 1630 and the instrument cluster 1632. In at least one embodiment, the instrument cluster 1632 may be included as part of the infotainment SoC 1630, and vice versa.

[0271] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details relating to the inference and / or training logic 715 are provided below. In at least one embodiment, the inference and / or training logic 715 may be used in the system of Figure 16C for inference or prediction operations, at least in part, based on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0272] Such components can be used to generate synthetic data that mimics failure scenarios in the network training process, which can help improve network performance while limiting the amount of synthetic data to avoid overfitting.

[0273] Figure 16D is a diagram of a system 1676 for communication between one or more cloud-based servers and the autonomous vehicle 1600 of Figure 16A, according to at least one embodiment. In at least one embodiment, system 1676 may include, but is not limited to, one or more servers 1678, one or more networks 1690, and any number and types of vehicles, including vehicle 1600. In at least one embodiment, server 1678 may include, but is not limited to, a plurality of GPUs 1684(A) to 1684(H) (collectively referred to herein as GPU 1684), PCIe switches 1682(A) to 1682(D) (collectively referred to herein as PCIe switches 1682), and / or CPUs 1680(A) to 1680(B) (collectively referred to herein as CPU 1680). The GPU 1684, CPU 1680, and PCIe switch 1682 may be interconnected by a high-speed interconnect, such as, for example, an NVLink interface 1688 and / or PCIe connection 1686 developed by NVIDIA. In at least one embodiment, the GPU 1684 is connected via an NVLink and / or NVSwitch SoC, and the GPU 1684 and PCIe switch 1682 are connected via a PCIe interconnect. In at least one embodiment, eight GPUs 1684, two CPUs 1680, and four PCIe switches 1682 are shown, but this is not limited to them. In at least one embodiment, each of (one or more) servers 1678 may include, but not limited to, any number of GPUs 1684, CPUs 1680, and / or PCIe switches 1682 in any combination. For example, in at least one embodiment, one or more servers 1678 may each include 8, 16, 32, and / or more GPUs 1684.

[0274] In at least one embodiment, one or more servers 1678 may receive from a vehicle via one or more networks 1690 image data representing images of unexpected or altered road conditions, such as recently started road construction. In at least one embodiment, one or more servers 1678 may transmit map information 1694, including a neural network 1692, an updated neural network 1692, and / or information about traffic and road conditions, to a vehicle via one or more networks 1690. In at least one embodiment, updates to the map information 1694 may include updates to the HD map 1622, including information about construction sites, potholes, detours, floods, and / or other obstacles, but are not limited to. In at least one embodiment, the neural network 1692, the updated neural network 1692, and / or the map information 1694 may arise from new training and / or experience represented in data received from any number of vehicles in the environment, and / or from training performed in a data center (for example, using one or more servers 1678 and / or other servers).

[0275] In at least one embodiment, one or more servers 1678 may be used to train a machine learning model (e.g., a neural network) based at least in part on training data. In at least one embodiment, the training data may be generated by a vehicle and / or generated in a simulation (e.g., using a game engine). In at least one embodiment, any amount of training data is tagged and / or undergoes other preprocessing (e.g., if the associated neural network benefits from supervised learning). In at least one embodiment, any amount of training data is not tagged and / or not preprocessed (e.g., if the associated neural network does not require supervised learning). In at least one embodiment, once the machine learning model is trained, the machine learning model may be used by the vehicle (e.g., transmitted to the vehicle via one or more networks 1690), and / or the machine learning model may be used by one or more servers 1678 to remotely monitor the vehicle.

[0276] In at least one embodiment, one or more servers 1678 may receive data from a vehicle and apply the data to an up-to-date real-time neural network for real-time intelligent inference. In at least one embodiment, one or more servers 1678 may include a deep learning supercomputer and / or a dedicated AI computer powered by one or more GPUs 1684, such as DGX and DGX Station machines developed by NVIDIA. However, in at least one embodiment, one or more servers 1678 may include a deep learning infrastructure using a data center powered by a CPU.

[0277] In at least one embodiment, the deep learning infrastructure of one or more servers 1678 may be capable of high-speed real-time inference and may use that capability to assess and verify the health of the processor, software, and / or associated hardware in the vehicle 1600. For example, in at least one embodiment, the deep learning infrastructure may receive periodic updates from the vehicle 1600, such as a series of images and / or objects located in that series of images (e.g., via computer vision and / or other machine learning object classification techniques). In at least one embodiment, the deep learning infrastructure may operate its own neural network to identify objects and compare them to objects identified by the vehicle 1600. If the results do not match and the deep learning infrastructure concludes that the AI ​​in the vehicle 1600 is malfunctioning, the one or more servers 1678 may send a signal to the vehicle 1600 instructing the vehicle 1600's fail-safe computer to take control, notify passengers, and complete a safe parking operation.

[0278] In at least one embodiment, the server 1678 may include one or more GPUs 1684 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT3). In at least one embodiment, a combination of a server powered by a GPU and inference acceleration may enable real-time response. In at least one embodiment, such as when performance is not critical, a server powered by a CPU, FPGA, and other processors may be used for inference. In at least one embodiment, inference and / or training logic 715 is used to carry out one or more embodiments. Further details regarding the inference and / or training logic 715 are provided elsewhere in this specification.

[0279] Other variations are within the scope of the present disclosure. Thus, while the disclosed techniques are capable of various modifications and alternative constructions, some of their exemplary embodiments are shown in the drawings and described in detail above. However, there is no intention to limit the present disclosure to the specific one or more disclosed forms, and on the contrary, it should be understood that it is intended to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure, as defined in the appended claims.

[0280] In the context describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar demonstrative pronouns, should be interpreted as encompassing both singular and plural, and not as definitions of terms, unless otherwise stated herein or clearly refuted by the context. The terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to,”) unless otherwise stated. The term “connected,” when unmodified and referring to a physical connection, should be interpreted as being partially or completely contained, attached, or joined to one another, even if there is something intervening. The detailing of ranges of values ​​herein is merely intended to serve as a concise way of individually referring to each distinct value that falls within a range, unless otherwise stated herein and unless each distinct value is incorporated into the specification as if it were individually detailed herein. The use of the terms “set” (for example, “set of items”) or “subset” should be interpreted as a non-empty set comprising one or more members, unless otherwise stated or denied by the context. Furthermore, unless otherwise stated or denied by the context, the term “subset” of a corresponding set does not necessarily refer to a strict subset of the corresponding set, and a subset and a corresponding set can be equivalent.

[0281] Combinations such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in contexts generally used to indicate that an item, term, etc., can be either A, B, or C, or any non-empty subset of the set of A, B, and C, unless otherwise specifically stated or explicitly denied by the context. For example, in a descriptive example of a set having three members, the combinations “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such combinations do not imply as a whole that some embodiments require the presence of each of A, B, and C. Furthermore, unless otherwise stated or negated by the context, the term "plurality" refers to a state of being multiple (for example, "a plurality of items" refers to multiple items). Plurality refers to at least two items, but can be more when explicitly or contextually indicated as such. Moreover, unless otherwise stated or otherwise evident from the context, the phrase "based on" means "at least partially based on," and does not mean "based solely on."

[0282] The operation of the processes described herein may be carried out in any preferred order unless otherwise stated herein or explicitly refuted by the context. In at least one embodiment, a process such as the processes described herein (or variations and / or combinations thereof) is carried out under the control of one or more computer systems consisting of executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or by a combination thereof. In at least one embodiment, the code is stored in a computer-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-temporary computer-readable storage medium that excludes temporary signals (e.g., transient electrical or electromagnetic transmissions that propagate) but includes non-temporary data storage circuit elements (e.g., buffers, caches, and queues) in transceivers for temporary signals. In at least one embodiment, code (e.g., executable code or source code) is stored in a set of one or more non-temporary computer-readable storage media which, when executed by one or more processors of a computer system (i.e., as a result of execution), causes the computer system to perform the operations described herein (or has other memory for storing executable instructions). In at least one embodiment, the set of non-temporary computer-readable storage media comprises a plurality of non-temporary computer-readable storage media which one or more of the individual non-temporary storage media of the plurality of non-temporary computer-readable storage media do not contain all of the code, but the plurality of non-temporary computer-readable storage media collectively contain all of the code.In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors, for example, a non-temporary computer-readable storage medium stores the instructions, the main central processing unit ("CPU") executes some of the instructions, and the graphics processing unit ("GPU") executes others. In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0283] Accordingly, in at least one embodiment, the computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system consists of applicable hardware and / or software that enables the performance of the operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system comprising multiple devices operating in different ways so that the distributed computer system performs the operations described herein and the single device does not perform all of the operations.

[0284] Any use of any examples or illustrative language provided herein (e.g., "such as") is intended solely to further illustrate the embodiments of this disclosure and, unless otherwise asserted, does not limit the scope of this disclosure. Nothing in this specification should be construed as indicating any unclaimed element as essential to the practice of this disclosure.

[0285] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as if they were included herein in their entirety, provided that each reference is clearly indicated by individual reference.

[0286] In the specification and claims, the terms “joined” and “connected” may be used together with their derivatives. It should be understood that these terms may not be intended to be synonymous with one another. Rather, in certain instances, “connected” or “joined” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with one another. “Joined” may also mean that two or more elements are not in direct contact with one another, but still interact or communicate with one another.

[0287] Unless otherwise specifically stated, throughout this specification, terms such as “processing,” “computing,” “calculating,” or “determining” should be understood to refer to actions and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data, represented as electronic or other physical quantities, in the registers and / or memory of the computing system, into other data, similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display device of the computing system.

[0288] Similarly, the term “processor” may refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. In non-limiting examples, “processor” may be a CPU or a GPU. A “computing platform” may comprise one or more processors. As used herein, “software” processes may include software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Each process may also refer to multiple processes for executing instructions serially or in parallel, continuously or intermittently. The terms “system” and “method” are used interchangeably herein only if one or more methods can be embodied by a system, and a method can be considered a system.

[0289] This specification may refer to acquiring, obtaining, receiving, or inputting analog or digital data into subsystems, computer systems, or computer-implemented machines. Acquiring, obtaining, receiving, or inputting analog and digital data can be implemented in various ways, such as by receiving data as parameters to function calls or calls to application programming interfaces. In some implementations, the process of acquiring, obtaining, receiving, or inputting analog or digital data can be implemented by transferring data via serial or parallel interfaces. In other implementations, the process of acquiring, obtaining, receiving, or inputting analog or digital data can be implemented by transferring data via a computer network from a providing entity to a receiving entity. It may also refer to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transferring data as input or output parameters to function calls, application programming interfaces, or parameters to inter-process communication mechanisms.

[0290] While the above description outlines exemplary implementations of the techniques described, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibility are defined above for illustrative purposes, various functions and responsibilities may be distributed and divided in different ways depending on the context. Furthermore, while the subject matter is described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the attached claims is not necessarily limited to the described specific features or actions. Rather, the specific features and actions are disclosed as exemplary forms that implement the claims.

Claims

1. A step of identifying a first set of images containing multiple objects of multiple classes, The steps include providing a first set of images as input to a first machine learning model trained to detect the presence of one or more objects of at least one of the plurality of classes shown in the given input images, and to predict at least mask data associated with one or more of the detected objects; A step of determining object data associated with each of a first set of images from one or more first outputs of the first machine learning model, wherein the object data for each of the images in the first set of images includes mask data associated with each object detected in each of the images. A step of training a second machine learning model to detect objects of a target class in a second set of images, wherein the second machine learning model is trained using at least one subset of the first set of images and a target output for the at least one subset of the first set of images, the target output includes the mask data associated with each object detected in the at least one subset of the first set of images and an indication of whether the class associated with each object detected in the at least one subset of the first set of images corresponds to the target class, The steps include identifying one or more heads of the second machine learning model that correspond to predicting mask data for a given input image when training the second machine learning model using the at least one subset of the first set of images and the target output, The steps include updating the second machine learning model to remove the one or more identified heads, and Methods that include...

2. The method according to claim 1, wherein the first machine learning model is further trained to predict, for each of the one or more detected objects, a particular class among the plurality of classes associated with each detected object.

3. The step of generating the target output further includes, The step of determining whether the specific class associated with each of the detected objects corresponds to the target class. The method according to claim 2, including the method described in claim 2.

4. Steps to identify ground truth data associated with each of the objects shown in the image, using bounding box indications associated with the image. The method according to claim 1, further comprising:

5. The method according to claim 4, wherein at least one of the one or more bounding boxes is provided by at least one of the platform's approved bounding box authority entities or users.

6. The method according to claim 1, wherein the second machine learning model is a multi-head machine learning model.

7. The steps include providing a third set of images as input to the second machine learning model, The steps include obtaining one or more second outputs of the second machine learning model, A step of determining additional object data associated with each of the third set of images based on the one or more second outputs, wherein the additional object data for each image in the second set of images includes an indication of a region of each image containing an object detected in each image, and a class associated with the detected object. The method according to claim 6, further comprising:

8. The step of sending the updated second machine learning model over the network to at least one of an edge device or an endpoint device. The method according to claim 6, further comprising:

9. Memory devices and, A processing device coupled to the memory device and A system comprising, wherein the processing device is A step of generating training data for a machine learning model, wherein the step of generating the training data is A step of generating a training input that includes an image of an object, and A step of generating a target output for the training input, wherein the target output includes a bounding box associated with the indicated object, mask data associated with the indicated object, and a class indication associated with the indicated object. Steps including, (i) providing the training data for training the machine learning model with respect to a set of training inputs including the generated training inputs, and (ii) a set of target outputs including the generated target outputs. A step of identifying one or more heads of the trained machine learning model that correspond to predicting mask data for a given input image, The steps include updating the trained machine learning model to remove the one or more identified heads, and A system that performs operations including those mentioned above.

10. The aforementioned operation is, The steps include providing a set of images as input to the updated trained machine learning model, The steps include obtaining one or more outputs of the updated trained machine learning model, A step of determining object data associated with each of the set of images from one or more outputs, wherein the object data for each of the images in a second set of images includes an indication of a region of each image containing an object detected in each image, and a class associated with the detected object. The system according to claim 9, further comprising:

11. The aforementioned operation, Steps to deploy the updated trained machine learning model using at least one of an edge device or an endpoint device. The system according to claim 9, further comprising:

12. The step of generating the target output for the training input is: A step of providing the image showing the object as input to an additional machine learning model, wherein the additional machine learning model is trained to detect the presence of one or more objects shown in a given input image and to predict at least mask data associated with one or more of the detected objects. A step of determining object data related to the image from the output of one or more of the additional machine learning models, wherein the object data for the image includes mask data related to the indicated object. The system according to claim 9, including the system described in claim 9.

13. The system according to claim 12, wherein the additional machine learning model is further trained to predict, for each of the one or more detected objects, the class associated with each detected object, and the object data for the image further includes the designation of the class associated with the indicated object.

14. The step of generating the target output for the training input is: A step of obtaining ground truth data related to the aforementioned image, wherein the ground truth data includes the bounding box related to the indicated object. The system according to claim 9, including the system described in claim 9.

15. The system according to claim 14, wherein the ground truth data is obtained from a database containing indications for one or more bounding boxes relating to an object shown in a set of images, the images are included in the set of images, and the one or more bounding boxes are provided by an authorized bounding box authority entity or user of the platform.

16. A non-temporary computer-readable storage medium containing instructions, wherein when the instructions are executed by a processing device, the processing device has: A step of providing a current set of images as input to a first machine learning model, wherein the first machine learning model is trained to detect objects of a target class in a given set of images using (i) a training input comprising a set of training images, and (ii) a target output for the training input, wherein the target output includes, for each training image in the set of training images, ground truth data relating to each object shown in each training image, wherein the ground truth data indicates a region of each training image containing each object, mask data relating to each object shown in each training image, wherein the mask data is obtained based on one or more outputs of a second machine learning model, and an indication of whether the class relating to each object shown in each training image corresponds to the target class. The steps include obtaining one or more outputs of the first machine learning model, A step of determining object data associated with each of the set of current images based on the one or more outputs of the first machine learning model, wherein the object data for each of the current images in the set of current images includes an indication of a region of the current image containing an object detected in the respective current image, and an indication of whether the detected object corresponds to the target class. The steps include: training the first machine learning model using the training input and the target output, identifying one or more heads of the first machine learning model that correspond to predicting mask data for a given input image; The steps include updating the first machine learning model to remove the one or more identified heads, and A non-temporary computer-readable storage medium that enables the execution of operations including the following.

17. The non-temporary computer-readable storage medium according to claim 16, wherein the object data further includes mask data relating to the object detected in each of the current images.

18. The step of determining object data associated with each of the aforementioned set of images is: A step of extracting one or more sets of object data from the one or more outputs of the first machine learning model, wherein each of the one or more sets of object data is related to a confidence level, wherein the object data corresponds to an object detected in the respective current image. A step of determining whether the level of reliability associated with each set of object data satisfies the level of the reliability criterion. A non-temporary computer-readable storage medium according to claim 16, including the following:

19. A step of providing the set of training images as input to the second machine learning model, wherein the second machine learning model is trained to detect one or more objects of at least one of a plurality of classes shown in the given input image, and to predict at least mask data associated with each of the one or more detected objects. A step of determining object data associated with each of the set of training images from one or more outputs of the second machine learning model, wherein the object data for each of the training images in the set of training images includes mask data associated with each object detected in each of the images. The non-temporary computer-readable storage medium according to claim 16, further comprising the step of training the first machine learning model by the above.

20. The non-temporary computer-readable storage medium according to claim 16, wherein the ground truth data is obtained using a database containing instructions for one or more bounding boxes associated with the set of training images, and each of the one or more bounding boxes is provided by at least one of the platform's approved bounding box authority entities or users.

Images