Artificial intelligence system for multi-standard sample detection

Abstract

In various examples, the present disclosure provides systems and techniques for training and deployment of multi-standard sample detection systems. The disclosed techniques include processing an image of a sample using a backbone neural network to generate sample features representing the sample, and obtaining reference features representing reference images. The disclosed techniques also include generating differentiated features representing differences between the sample features and the reference features, and predicting a characteristic of the sample based on aggregating the differentiated features.

Description

Technical Field

[0001] At least one embodiment relates to processing resources for performing and facilitating artificial intelligence (AI). For example, according to various novel techniques described herein, at least one embodiment relates to operations encountered when training and using machine learning models to achieve efficient data processing and detection. Background Technology

[0002] Machine learning is frequently used in image processing, such as identifying objects depicted in images. Object recognition is used in medical imaging, scientific research, autonomous driving systems, robotic automation, security applications, law enforcement practices, and many other fields. Machine learning involves training a computational system using training images and other training data to identify patterns in images that may aid in object recognition. Training can be supervised or unsupervised. Machine learning models can use various computational algorithms, such as decision tree algorithms (or other rule-based algorithms), artificial neural networks, etc. During the inference phase, new images are fed into the trained machine learning model, which then utilizes the patterns and features identified during training to identify various target objects of interest (e.g., vehicles in a road image). Attached Figure Description

[0003] Figure 1 This is a block diagram of an exemplary computer system capable of training and deploying a high-precision multi-standard sample detection (MGSI) system according to at least one embodiment.

[0004] Figure 2 It is an exemplary computing device that may support the operation of an MGSI system according to at least one embodiment.

[0005] Figure 3A The illustrations, according to at least one embodiment, depict exemplary operation of an MGSI system for performing high-precision inference on test data.

[0006] Figure 3B The illustration shows a configuration that can be deployed for execution according to at least one embodiment. Figure 3A An exemplary architecture of the backbone network operating in the middle.

[0007] Figure 4 The illustration shows training according to at least one embodiment. Figure 3A Exemplary operation of the MGSI system.

[0008] Figure 5 This is a flowchart of an exemplary method for deploying an MGSI system for inference on sample data according to at least one embodiment.

[0009] Figure 6 This is a flowchart of an exemplary method for training an MGSI system according to at least one embodiment;

[0010] Figure 7A The inference and / or training logic according to at least one embodiment is illustrated;

[0011] Figure 7B The inference and / or training logic according to at least one embodiment is illustrated;

[0012] Figure 8 The training and deployment of a neural network according to at least one embodiment are illustrated;

[0013] Figure 9 This is an example data flow diagram of an advanced computing pipeline according to at least one embodiment;

[0014] Figure 10 This is a system diagram of an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment. Detailed Implementation

[0015] Computer vision AI systems are used in applications such as manufacturing quality control. Production lines or manufacturing processes produce products such as wafers, dies, integrated circuits, and printed circuit boards. Various products (also referred to as samples in this document) can be subjected to appropriate inspections, such as optical inspection, scanning electron microscopy, and X-ray inspection, to assess product quality and determine whether the products meet the specifications required by the manufacturing process. Inspections can be performed on each product, on periodically selected products, or on randomly selected products. Some inspection techniques may involve using trained AI models to process images and generate outputs, such as binary predictions of whether an image depicts a product that meets manufacturing objectives.

[0016] In standard sample detection, the image of a sample is compared to an image of a "standard sample" of a high-quality product. The quality of the sample is inferred from the similarity between the sample image and the standard sample image. This inference can be performed by an AI model that processes the sample image and the standard sample image separately using a backbone neural network and a classifier network. The backbone neural network generates features (feature vectors, embedding vectors) representing each image, while the classifier network compares the similarity between the generated features and predicts whether the sample meets the quality standard. While standard sample detection can successfully identify high-quality samples, it often produces a large number of false negatives. It may label samples that meet the quality standard but differ from the standard sample image, leading the model to incorrectly classify them as low-quality samples.

[0017] The aspects and embodiments of this disclosure address these and other technical challenges of standard sample testing techniques by providing a high-precision multi-standard sample inspection (MGSI) system and techniques. In some embodiments, a set of multiple (e.g., N) standard samples representing an acceptable range (or distribution) of deviations in a product for a manufacturing process or other technical process can be obtained. Such a set can indicate a range of tolerances acceptable for the intended use of the manufactured product. Samples whose images are similar to some or all of the standard samples in the set can represent products that meet quality standards. Images of the sample being tested (test sample) and N standard sample images can be processed by N+1 parallel instances of the backbone network (or by using a single backbone network instance applied N+1 times serially), and N+1 corresponding feature sets can be generated, such as vectors in a reduced-dimensional space with a dimension D sufficient to capture the visual appearance of the image of interest. In some embodiments, such as when inspecting a large stream of test samples by comparing them to a fixed set of standard samples, backbone processing of the N standard sample images is not required during field testing. Instead, the features of the N standard samples can be pre-computed and stored in the memory of the MGSI system, and a single instance of the backbone network can process the images of the test samples. The features generated by N+1 can then be processed (in parallel or serially) by N instances of the differential network to generate N differential features representing the differences between the features of the test samples and the corresponding features of the standard samples.

[0018] Next, the N differential features can be processed by an aggregator that combines the differential features into aggregate features, collectively representing the test sample and the N standard samples. In some embodiments, the aggregator may compute the average of all input differential features. In some embodiments, the aggregator may use one or more neural network layers (e.g., fully connected layers or linear layers) to concatenate and process the input differential features. In some embodiments, the aggregator may include one or more Transformer blocks that perform attention processing on the differential features, e.g., using various differential features as attention queries and other various differential features as keys and values. The output of the aggregator—the aggregate features—can be processed by a final classification head that outputs a prediction of the sample. This prediction may include a probability P (e.g., generated by the sigmoid layer of the classification head) that the sample has acceptable quality (and a corresponding probability 1-P for unacceptable quality). In some embodiments, the classification head may generate multiple probabilities P. i(e.g., generated by a Softmax layer), indicating that the test samples conform to a corresponding number of test categories, such as critical size, purity, chemical composition, etc. In some embodiments, the disclosed MGSI system can be trained end-to-end, with the backbone network, differentiation network, aggregator (if implemented as a neural network), and classification network trained together, for example using a suitable loss function and concurrent error backpropagation through the various layers of the network.

[0019] The advantages of the disclosed system and technology include, but are not limited to, more accurate detection of samples with compliant (or conversely, non-compliant) quality, and a significant reduction in false negative results. Furthermore, the disclosed technology eliminates the need for labor-intensive screening of a single standard sample, and avoids extensive subsequent training of the detector model to learn the various acceptable biases of that standard sample; instead, it enables the use of a set of suboptimal samples obtained in normal manufacturing processes and screened by process engineers with minimal effort. Although various concepts are illustrated with reference to the image examples throughout this disclosure, substantially the same or similar concepts can be used to analyze a wide variety of other types of data, such as audio data, sensor data (e.g., temperature data, pressure data, optical reflectance data, chemical composition data, etc.), traffic data, security data, public and private security data, financial data, and a variety of other types of data encountered in many practical applications.

[0020] System Architecture

[0021] Figure 1 This is a block diagram of an exemplary computer system 100 capable of training and deploying a high-precision multi-standard sample detection (MGSI) system according to at least some embodiments. Figure 1 As shown, the computing system 100 may include a computing device 102, a data storage device 150, and a training server 160 connected to a network 140. The network 140 may be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wireless network, a personal area network (PAN), a combination thereof, and / or another network type.

[0022] The computing device 102 may be a desktop computer, laptop computer, smartphone, tablet computer, server, or any suitable computing device capable of performing the techniques described herein. The computing device 102 may be configured to receive sample 101 via any suitable input / output interface. In some embodiments, sample 101 may include one or more images. In the following description, for the sake of brevity, "sample" refers both to the physical object being examined and to any suitable data (e.g., images, sensor data, audio data, etc.) depicting, describing, and / or characterizing that physical object. In some embodiments, images or other data may be generated by one or more devices connected to the computing device 102. For example, devices capable of generating sample 101 may include photographic equipment, scanners, video cameras, optical inspection equipment (including photodetectors, reflectometers, elliptic apparatuses, etc.), scanning electron microscopes, tunneling electron microscopes, atomic force microscopes, X-ray electron spectroscopy devices, autonomous vehicle sensors (e.g., lidar, radar, long-range and mid-range cameras, infrared cameras, etc.), etc. In some embodiments, sample 101 may include images of wafers (substrates), one or more dies, integrated circuits, printed circuit boards, and / or any other products and / or objects. Sample 101 can be in any digital (e.g., pixel-based or vector-based) format, including but not limited to JPEG, GIF, PNG, BMP, TIFF, CIB, DIMAP, NITF, etc. Sample 101 can be stored (e.g., along with other samples or data) in data storage device 150. Furthermore, according to some embodiments disclosed herein, data storage device 150 can store training samples 152 for training one or more neural networks (or other models) in a multi-standard sample detection system. Computing device 102 can directly access data storage device 150 or (as shown) indirectly via network 140.

[0023] Data storage device 150 may be a persistent storage device capable of storing images and metadata of the stored images. Data storage device 150 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage devices, disks, tape or hard disk drives, network attached storage (NAS), storage area network (SAN), etc. Although data storage device 150 is shown as separate from computing device 102, in at least one embodiment, the data storage device may be part of computing device 102. In at least some embodiments, data storage device 150 may be a network-attached file server, while in other embodiments, data storage device 150 may be other types of persistent storage devices, such as object-oriented databases, relational databases, etc., which may be hosted by a server machine or one or more different machines coupled to computing device 102 via network 140.

[0024] The computing device 102 can receive sample 101 and perform any suitable sample preprocessing 106, including cropping, sharpening, blurring, noise or other artifact removal, compression, resampling, normalization, upsampling or other operations, or any combination thereof.

[0025] The computing device 102 may include a memory 104 communicatively coupled to one or more processing devices, such as one or more graphics processing units (GPUs) 110, one or more central processing units (CPUs) 130, one or more parallel processing units (PPUs) or accelerators (such as deep learning accelerators), data processing units (DPUs), etc. The memory 104 may store one or more models of the MGSI system 120, including various models implementing single-sample processing 122 (e.g., backbone networks, differential networks, etc.) and aggregated sample processing 124 (e.g., aggregators, classification heads, etc.). The MGSI system 120 may be executed by the GPU 110, CPU 130, PPU, DPU, etc. The MGSI system 120 may use sample 101 (or training sample 152) as input to predict one or more characteristics of sample 101, such as overall sample quality, various key dimensions, chemical purity, chemical composition, etc. In at least one embodiment, the MGSI system 120 may operate in conjunction with a manufacturing system (e.g., machine, production line, process, etc.). Based on the predicted sample characteristics, the manufacturing system can be stopped, adjusted, optimized, or otherwise modified to improve sample quality or ensure that the samples produced by the manufacturing system conform to any applicable specifications of the implemented manufacturing process.

[0026] Various networks and / or components of the MGSI system 120 can be trained by the training server 160. In at least one embodiment, the training server 160 may be part of the computing device 102. In other embodiments, the training server 160 may be directly or indirectly communicatively coupled to the computing device 102 via network 140. The training server 160 may be (or include) a rack server, router computer, desktop computer, laptop computer, smartphone, tablet computer, server, media center, and / or any suitable computing device or combination thereof capable of performing the techniques described herein. The training server 160 may include a training engine 162. In at least one embodiment, the training engine 162 may initiate and train one or more machine learning models (e.g., the neural network of the MGSI system 120).

[0027] Training (updating) of the MGSI system 120 can be performed using training data stored in data storage device 150. The training data may include training samples 152, reference samples 154, and annotations 156, such as ground truth evaluations of training samples 152. Reference samples 154 may be (or include) standard samples indicating acceptable variations in samples being detected. Annotations 156 may indicate whether training sample 152 is sufficiently similar to one or more reference samples 154 (and thus represents acceptable product variation) or substantially different from any reference sample 154 (and thus unacceptably deviates from the target specification). In some embodiments, various networks and models of the MGSI system 120 may be trained together, e.g., end-to-end training, with errors in the final prediction backpropagated through multiple networks (e.g., backbone networks, differentiation networks, aggregator networks, classification heads, etc.).

[0028] In at least one embodiment, various networks and models of the MGSI system 120 can be implemented as deep learning neural networks with multi-level linear or non-linear operations. For example, each or some of the networks and models in the MGSI system 120 may include convolutional neural networks, recurrent neural networks (RNNs), fully connected neural networks, attention-based neural networks, Transformer neural networks, visual Transformer neural networks, etc. In at least one embodiment, each or some of the networks and models in the MGSI system 120 may include multiple neurons, each receiving input from other neurons or external sources and producing an output by applying an activation function to a (trainable) weighted sum of the input and bias values. In at least one embodiment, each or some of the networks and models in the MGSI system 120 may include multiple neurons arranged in layers, including an input layer, one or more hidden layers, and an output layer. Neurons in adjacent layers can be connected by weighted edges. Initially, some initial (e.g., random) values ​​can be assigned to the edge weights.

[0029] Training of the MGSI system 120 can be performed by a training engine 162 by selecting a set of training samples 152 and corresponding sets of two or more reference samples 154. The training samples 152 and reference samples 154 can be processed by various networks of the MGSI system 120, and the MGSI system is trained to generate training predictions 164 for the training samples 152. In some embodiments, the training prediction 164 can be a single binary prediction, such as "sample meets" or "sample does not meet". In some embodiments, the training prediction 164 can include a set of binary predictions, such as "layout meets / does not meet", "size meets / does not meet", "chemical composition meets / does not meet", "purity meets / does not meet", etc. In some embodiments, the training prediction 164 can include a set of non-binary predictions, such as "overall quality: 0, 1, ... or M", "layout accuracy: 0, 1, ... or N", etc. In some embodiments, training prediction 164 may include continuous probabilities, such as those defined in the interval [0,1], representing the probability that training sample 152 has acceptable quality (and / or other similar characteristics of the sample). Training prediction 164 may be compared with label 156 (the ground truth, the desired target output) using a suitable loss function 166, and the difference between training prediction 164 and label 156 may be backpropagated through the network of MGSI system 120 to change various network parameters (e.g., the weights and biases of individual neurons) to make training prediction 164 closer to label 156. This adjustment may be repeated until the output error of a given training sample 152 meets a predetermined condition (e.g., drops below a predetermined value). Subsequently, different training samples 152 may be selected, new outputs generated, and a series of new adjustments implemented until the individual neural networks are trained to acceptable accuracy or the maximum accuracy achievable given the network architecture and complexity (e.g., the number of neurons / layers in the model).

[0030] This type of learning technique can train the MGSI system 120 to recognize patterns in training samples 152 based on the desired target output. Subsequently, additional training samples / labels can be used to verify the predictive utility of the recognized patterns, and then used in the inference phase when processing new (unencountered) samples in the future.

[0031] Figure 2 This is an exemplary computing device 200 according to at least one embodiment, which can support the operation of a high-precision multi-standard sample detection system. In some embodiments, the computing device 200 may be (or include) Figure 1The computing device 102, the computing device of the training server 160, and / or any other suitable computing device performing the operations of this disclosure. The operation of the MGSI system 120 and / or various modules operating in conjunction with the MGSI system 120, such as single-sample processing 122, aggregated-sample processing 124, and / or other software / firmware instantiated on the computing device 200, can be executed using one or more GPUs 110, CPUs 130, PPUs, DPUs, etc. In at least one embodiment, the GPU 110 includes multiple cores 210. A single core 210 can execute multiple threads 212. A single core 210 can run multiple threads 212 simultaneously (e.g., in parallel). In at least one embodiment, threads 212 can access registers 213. Registers 213 can be thread-specific registers, with access to the register limited to the corresponding thread. Furthermore, one or more (e.g., all) threads of core 210 can access shared registers 214. In at least one embodiment, a single core 210 may include a scheduler 215 for allocating computational tasks and processes among different threads 212 of the core. The dispatch unit 216 can use the correct dedicated register 213 and shared register 214 to implement the scheduled task on the appropriate thread. The computing device 200 may include an input / output component 217 to exchange information with one or more users or developers.

[0032] In at least one embodiment, GPU 110 may have a (high-speed) cache 218, which multiple cores 210 may share access to. Furthermore, computing device 200 may include GPU memory 219, in which GPU 110 may store intermediate and / or final results (outputs) of various computations performed by GPU 110. After completing a specific task, GPU 110 (or CPU 130) may move the output to (main) memory 104. In at least one embodiment, CPU 130 may execute processes involving serial computation tasks, while GPU 110 may execute tasks suitable for parallel processing (e.g., multiplying the input of a neural node by weights and adding a bias).

[0033] Figure 3A The illustration depicts exemplary operation 300 of a multi-standard sample detection system for high-precision inference on test data, according to at least one embodiment. In at least one embodiment, operation 300 may be performed by... Figure 1 The MGSI system 120 of the computing device 102 in the middle or Figure 2 The computing device 200 in the middle executes. For example... Figure 3A As shown, operation 300 can be performed on sample data 301, which may include images of samples (e.g., Figure 1The data can be: sample data 301, audio data, data collected by one or more physical sensors and / or one or more chemical sensors (including any applicable time-series data), and / or data collected by any other measuring device, and / or combinations thereof. Sample data 301 can represent the characteristics of any product or process, including but not limited to manufactured products (e.g., semiconductor devices, such as chips with transistors and interconnect circuits). Sample data 301 can represent the characteristics of a final product or any intermediate product that still requires at least some additional processing stages. For example, intermediate products can include bare wafers, wafers with one or more thin films deposited, wafers that have undergone plasma etching, atomic layer deposition, mask deposition, beam epitaxy, and / or any other manufacturing operation.

[0034] In one example, sample data 301 may include two-dimensional I... k (x, y) or three-dimensional I k (x, y, z), for multiple pixels or voxels identified by coordinates x, y and x, y, z, having one (e.g., black and white) or multiple (e.g., color) intensity values ​​I. k Intensity can be measured in any suitable unit, for example, within a finite range from 0 to 1 (0 to 100, or any other finite range), where I = 0 corresponds to dark (e.g., black) pixels / voxels and I = 1 corresponds to bright (e.g., white) pixels / voxels. In at least one embodiment, intensity values ​​can be obtained for different time instances (or points in time) (e.g., a time series of images, such as video frames). In another example, sample data 301 may include a time series of audio recordings or audio frames or an audio spectrogram.

[0035] In some embodiments, sample data 301 may undergo data preprocessing 306, which may include data cropping, data denoising (e.g., image sharpening, blurring or other artifact removal), data compression, data resampling, data normalization, data upsampling, data batching, or performing any other suitable operation on sample data 301, and / or any combination thereof.

[0036] Sample data 301 can be used in conjunction with N sets of reference data 302-1, 302-2, ..., 302-N, which may include standard samples of a specific product or process associated with sample data 301. Reference data 302-n may be of the same type as sample data 301 (e.g., an image). Reference data 302-n may represent the distribution of acceptable deviations from sample data 301, for example, depicting different products identified (e.g., by a process engineer) in a manufacturing process as having acceptable quality. The set of reference data 302-1, 302-2, ..., 302-N may indicate the acceptable tolerance range for a manufactured product for its intended use without explicitly listing the limits of various features, characteristics, and / or dimensions of the product. Sample data 301 similar to one or more of the reference data 302-n may represent a product with acceptable quality.

[0037] Sample data 301 (which may be preprocessed and digitized as appropriate) can be processed by an instance of the backbone network, such as backbone network 310. In some embodiments, backbone network 310 may be (or include) a convolutional neural network (CNN), a transformer neural network, a conformer (a combination of CNN and transformer) neural network, and / or some other suitable neural network. Backbone network 310 can transform sample data 301 into sample features 320, also referred to as feature vectors or embeddings. Features (feature vectors, embeddings) should be understood as any suitable numerical representation of input data (e.g., sample data 301) via a vector (set) of any number of D components, which may have integer or floating-point values. Features (e.g., sample features 320) can be viewed as vectors or points in a D-dimensional feature space, where the dimension D of the feature space is smaller than the size of the input data, and are defined as part of the architecture of the model (e.g., backbone network 310). During training, the model learns to associate similar input datasets with similar features represented by closely adjacent points in the feature space, and dissimilar input datasets with points that are further apart in the feature space.

[0038] like Figure 3AAs shown, additional backbone networks 311, 312, ..., 31N can process corresponding reference data 302-1, 302-2, ..., 302-N and generate reference features 321, 322, ..., 32N. In some embodiments, N+1 backbone networks 310, 311, 312, ..., 31N can process input data in parallel. In some embodiments, backbone networks 310, 311, 312, ..., 31N can include the same neural network operation performed by a separate processing thread of a processing device (e.g., GPU) or multiple processing devices, for example, using a shared memory device or different memory devices. In some embodiments, a single instance of the backbone network can serially perform the operations shown by N+1 backbone networks 310, 311, 312, ..., 31N. In some embodiments, N reference features 321, 322, ..., 32N can be pre-computed and stored in system memory (e.g., ...). Figure 1 (Memory 104 of the computing device 102). In such embodiments, a single backbone network 310 can process sample data 301 and generate sample features 320, while reference features 321, 322, ..., 32N can be retrieved from system memory.

[0039] The N+1 generated features can be processed by N instances of the differential network. More specifically, differential network 331 can process sample feature 320 and reference feature 321 generated by backbone network 311. Similarly, differential networks 332, ..., 33N can process a copy of sample feature 320 and reference features 322, ..., 32N (generated by corresponding backbone networks 312, ..., 31N, respectively). Differential networks 331, 332, ..., 33N can generate N differential features 341, 342, ..., 34N. Differential feature 34n can represent (encode) the difference between the corresponding reference feature 32n and sample feature 320. In some embodiments, differential networks 331, 332, ..., 33N can be identical and may include one or more learned neuron layers (e.g., fully connected layers or linear layers). In some embodiments, differential features 341, 342, ..., 34N may have the same dimension D as the sample / reference features. In some embodiments, the dimensions of the differential features may differ. In some embodiments, N instances of the differential network 33n may be used in parallel (e.g., for faster processing). In some embodiments, a single instance of the differential network may process different input pairs sequentially, one after another (e.g., for less resource-intensive processing).

[0040] N differential features 341, 342, ..., 34N can be processed by an aggregator 350, which combines the differential features into aggregate features 360, collectively representing sample data 301 and N reference data 302-1, ..., 302-N. In some embodiments, the aggregator 350 can be a hard-coded function, such as the average of the N differential features 341, 342, ..., 34N. In some embodiments, the aggregator 350 can use one or more neural network layers (e.g., fully connected layers or linear layers) to concatenate and process the N differential features 341, 342, ..., 34N. In some embodiments, the aggregator 350 may include one or more Transformer modules to perform attention processing on the differential features. For example, in one round of processing, differential feature 341 can be used as an attention query, and other differential features 342, ..., 34N can be used as keys and values. (More specifically, various query-key scalar products can be calculated and used to determine the corresponding weights, with said weights, combining the corresponding values ​​into the hidden state of a particular query.) In another round of processing, differential feature 342 can be used as an attention query, and other differential features 341, 342, ..., 34N can be used as keys and values, etc. (In some embodiments, when a given differential feature 34n is used as a query, the same differential feature 34n can also be used as one of the key-value pairs.)

[0041] The aggregated features 360 generated by aggregator 350 can be processed by classification head 370, which outputs a prediction 380 for the sample. Prediction 380 may include a probability P (or log probability), which can be generated by the final sigmoid layer of classification head 370, representing that the sample has acceptable quality. The value 1-P can represent the probability that the sample quality is unacceptable. In some embodiments, classification head 370 may generate multiple probabilities P. i For example, the probabilities generated by the final Softmax layer of the classification head 370. Different probabilities P i It can indicate whether a sample meets the corresponding number of test categories, such as overall quality (i=1), critical size (i=2), purity (i=3), chemical composition (i=1), etc.

[0042] Figure 3B An exemplary architecture of a backbone network 310 according to at least one embodiment is illustrated, which can be used to perform... Figure 3A Operation 300 in the middle. Figure 3BA portion of the backbone network 310 is depicted, comprising processing stages 390, 392, 394, and 396 with progressively decreasing resolution (the field of view gradually expands as the number of pixels / voxels representing intermediate features increases). Arrows schematically indicate the corresponding processing stages. Each processing stage may apply one or more layers of neurons and may generate intermediate features, which are aggregated (e.g., concatenated, averaged, weighted averaged, etc.) to generate sample features 320. For example, processing stage 390 may generate intermediate feature 391, processing stage 392 may generate intermediate feature 393, processing stage 394 may generate intermediate feature 395, processing stage 396 may generate intermediate feature 397, and so on. Intermediate features 391, 393, 395, and 397 (besides being used as inputs for subsequent processing stages) may be aggregated into sample features 320. The number of processing stages in which intermediate features are aggregated into sample features 320, such as one, two, four, etc., can be determined experimentally, for example, during the training of the backbone network. Although Figure 3B The operation of the backbone network is illustrated through the example of the backbone network 310 (processing sample data 301 and generating sample features 320). However, when the backbone networks 311, ..., 31N process the reference data 302-1, ..., 302-N to generate the corresponding reference features 321, ..., 32N, the same or similar processing and feature aggregation methods can be used.

[0043] Figure 4 The illustration shows training according to at least one embodiment. Figure 3A An exemplary operation 400 of a multi-standard sample detection system is described. In at least one embodiment, operation 400 may respond to instructions from training engine 162. Figure 1 (As also shown in the image), it is used to train the MGSI system. Specifically, the training engine 162 can select a set of training sample data 401 and multiple training reference data 402-n. Figure 4 As shown, operation 400 can be performed on training sample data 401, the type of which (e.g., image, audio data, sensor data, etc.) can be related to the MGSI system. Figure 3A The sample data 301 processed during the inference operation shown is of the same type. In some embodiments, the training sample data 401 may undergo similar data preprocessing 306 as the sample data 301 during the inference operation, including data cropping, denoising, sharpening, artifact removal, compression, normalization, upsampling, or downsampling.

[0044] Training sample data 401 can be used in conjunction with N sets of training reference data 402-1, 402-2, ..., 402-N, such as standard samples of various products or processes associated with training sample data 401. In some embodiments, any, some, or all of the training reference data 402-1, 402-2, ..., 402-N may (but must) be the same as (or similar to) the reference data 302-1, 302-2, ..., 302-N used during inference operations. In some embodiments, the training reference data 402-1, 402-2, ..., 402-N may be different from the reference data 302-1, 302-2, ..., 302-N used during inference operations. In some embodiments, the acceptable preprocessing 306 for training sample data 401 may be different from the preprocessing of training references 402-1, 402-2, ..., 402-N.

[0045] Training sample data 401 and training reference data 402-n (each data being appropriately preprocessed and digitized, where applicable) can be processed by corresponding instances of the backbone networks 310, 311, ..., 31N (in parallel, serial, or some combination of both) to generate training sample features 420 and training reference features 421, ..., 42N. These generated features can be processed by N instances of the differentiation networks 331, ..., 33N to generate N training differential features 441, 442, ..., 44N, which digitally encode the differences between the corresponding training reference features 42n and the training sample features 420. The N training differential features 441, 442, ..., 44N can be processed by the aggregator 350 (a hard-coded algorithm or a trainable neural network) to generate training aggregate features 460, which collectively represent N instances of the training sample data 401 and the training reference data 402-1, ..., 402-N.

[0046] The classification head 370 can process the training aggregated features 460 generated by the aggregator 350 and output a training prediction 164 of the training sample data 401. In some embodiments, the training prediction 164 can be a single binary prediction, such as "qualified sample" or "unqualified sample". In some embodiments, the training prediction 164 can include a continuous probability P (e.g., defined on the interval [0,1]) or a log probability logP (e.g., defined on the interval (-∞, +∞)) to predict whether the sample quality meets the standards. In some embodiments, the training prediction 164 can include a set of multiple binary predictions, such as "layout qualified / unqualified", "size qualified / unqualified", "chemical composition qualified / unqualified", "purity qualified / unqualified", etc. In some embodiments, the training prediction 164 can include multiple continuous probabilities {P} i} or log probability {logP iThe set of} represents the probability that the training samples depicted in the training sample data 401 conform to the above (or any other number) test categories.

[0047] In some embodiments, the trained prediction 164 can be compared with (ground) labels 156 using a suitable loss function 166. Labels 156 can include any evaluation of the training sample data 401, such as an evaluation that could be performed by a process or product engineer, including classification of the training samples in one or more categories. The loss function 166 can be (or include) a binary cross-entropy loss function, a Kullback-Leibler loss function, and / or some other suitable loss function. The loss function 166 can use the trained prediction 164 and labels 156 as inputs (parameters) and output a loss value 470 that quantifies the difference between the inputs. Figure 4 As schematically shown by the dashed arrow, the loss value 470 can be backpropagated through one or more networks of the MGSI system. Various network parameters of these networks (e.g., the weights and biases of individual neurons) are modified using various techniques such as backpropagation and stochastic gradient descent to align (closer to) the training prediction 164 with the annotation 156. This adjustment can be repeated until the output error of a given training sample data 401 meets a predetermined condition (e.g., drops below a predetermined value). Subsequently, different training sample data 401 can be selected, new training predictions 164 can be generated for the training sample data 401, and a new series of adjustments can be implemented until the individual neural networks are trained to an acceptable accuracy or until the network reaches the accuracy limited by its architecture. In some embodiments, multiple networks can be trained end-to-end, for example, the backbone network, differential network, aggregator (if implemented as a neural network), differential networks 331-33N, aggregator (if implemented as a neural network), and / or classification head 370 can be trained together, for example, through concurrent error backpropagation across various layers of the network.

[0048] According to some embodiments of this disclosure Figure 5 and Figure 6 Examples of methods 500 and 600 for training and deploying a multi-standard sample detection system are illustrated. In one embodiment, Figure 5 and / or Figure 6The illustrated methods can be used for quality inspection of manufactured products. Methods 500 and / or 600 can be executed by one or more processing units (e.g., CPU and / or GPU), which may include (or communicate with) one or more memory devices. In at least one embodiment, methods 500 and / or 600 can be executed by multiple processing threads (e.g., CPU threads and / or GPU threads), each thread executing one or more independent functions, routines, subroutines, or operations of the method. In at least one embodiment, the processing threads implementing methods 500 and / or 600 can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing methods 500 and / or 600 can execute asynchronously with each other. Figure 5 and / or Figure 6 Compared to the order shown, the individual operations of method 500 and / or method 600 may be performed in a different order. Figure 5 and / or Figure 6 Some operations can be performed simultaneously with other operations. In at least one embodiment, Figure 5 and / or Figure 6 One or more of the operations shown may not always be performed.

[0049] According to at least one embodiment, Figure 5 This is a flowchart of an exemplary method 500 for deploying a multi-standard sample detection system for inference of sample data. Method 500 may be provided by... Figure 1 Computing device 102 or Figure 2 The method is executed by one or more processing units in the computing device 200. At block 510, the processing unit executing method 500 may use a backbone neural network (e.g., Figure 3A The backbone network 310 in the process processes the images of the samples (e.g., Figure 3A Sample data 301 in the sample is used to generate sample features representing the sample (e.g., Figure 3A (Sample features 320). In some embodiments, the backbone neural network may include a convolutional neural network, a transformer neural network, a visual transformer neural network, and / or other suitable neural networks.

[0050] At box 520, method 500 may continue to obtain a plurality of reference features, each of which (e.g., Figure 3A Reference features 321, ..., 32N in the reference images represent corresponding reference images among a plurality of reference images. In some embodiments, obtaining the corresponding reference features among the plurality of reference features may include the operations shown by the annotation boxes 522 and / or 524. More specifically, at box 522, method 500 may include using a backbone neural network (e.g., Figure 3AThe method 500 may process the corresponding reference image from a backbone network (one of the backbone networks 311, ..., 31N) in the memory device. At block 524, method 500 may include processing the reference image from a memory device (e.g., ...). Figure 1 The memory 104 of the computing device 102 or Figure 1 The data storage device 150 retrieves multiple reference features (e.g., pre-computed) generated previously by processing multiple reference images using a backbone neural network.

[0051] At box 530, method 500 can continue to generate multiple differential features (e.g., Figure 3A The plurality of differential features (341, ..., 34N) represent a difference between a sample feature and a corresponding reference feature among a plurality of reference features. In some embodiments, generating the plurality of differential features may include the operation of bounding box 532, including using a differential neural network (e.g., Figure 3A The differential network (one of the differential networks 331, ..., 33N) processes sample features (e.g., sample feature 320) and corresponding reference features (e.g., Figure 3A The input is one of the reference features 321, ..., 32N.

[0052] At box 540, method 500 may include aggregating multiple differential features into a single aggregated feature (e.g., Figure 3A (Assembly feature 360 ​​in the diagram). In some embodiments, aggregating multiple reference features may include the operations shown in boxes 542 and / or 544. More specifically, at box 542, aggregating multiple reference features may include averaging multiple differential features to obtain the aggregated feature. In some embodiments, at box 544, aggregating multiple reference features may include processing multiple differential features by an aggregation neural network. In some embodiments, the aggregation neural network may include one or more fully connected neuron layers, one or more neuron attention blocks, and / or other suitable combinations of neurons.

[0053] At box 550, method 500 may continue to predict one or more characteristics of the sample based on aggregated features. In some embodiments, predicting one or more characteristics of the sample may include the operations shown in labeled boxes 552 and / or 554. More specifically, at box 552, the operation of method 500 may include processing the aggregated features using a classification head neural network (e.g., classification head 370) to generate a probability that the sample conforms to the sample manufacturing process specification. In some embodiments, predicting one or more characteristics of the sample may include, at box 554, processing the aggregated features using a classification head neural network to generate multiple probabilities, each of which represents the likelihood that the sample conforms to a corresponding test category among multiple test categories. One or more characteristics of the sample may be integrated into or otherwise incorporated into the manufacturing process. For example, these characteristics may be used to adjust one or more operations of the manufacturing system to ensure that the samples produced by the manufacturing system conform to applicable specifications.

[0054] Figure 6 This is an exemplary flowchart of a multi-standard sample detection system training method 600 according to at least one embodiment. Method 600 can be provided by... Figure 1 One or more processing units of the training server 160 shown execute this method. In block 610, the processing unit executing method 600 can process the training images of the training samples using a backbone neural network (e.g., Figure 4 The training sample data 401 in the dataset is used to generate training sample features representing the training sample (e.g., ...). Figure 4 (420) Training sample features.

[0055] At box 620, method 600 can continue to use the backbone neural network to process multiple training reference images (e.g., Figure 4 The training reference data (402-1, ..., 402-N) are used to generate multiple corresponding training reference features (e.g., Figure 4 Training reference features 421, ..., 42N).

[0056] At box 630, method 600 may include generating multiple trained differential features (e.g., Figure 4 The training differential features (441, ..., 44N) represent training sample features (e.g., training differential features 441, ..., 44N). Figure 4 The training sample features 420 in the training reference features and the corresponding training reference features (e.g., in the training reference features) among multiple training reference features. Figure 4 The corresponding differences between the training reference features 421, ..., 42N in the dataset. In some embodiments, as shown in box 632, generating multiple training differential features may include using a differential neural network (e.g., Figure 4 (A differential network among the differential networks 331, ..., 33N).

[0057] At box 640, the operation of method 600 may include aggregating multiple training differential features to obtain training aggregated features (e.g., Figure 4 (Training aggregated features 460). In some embodiments, as shown in box 642, aggregating multiple training differential features may include using an aggregation neural network (e.g., Figure 4 Aggregator 350 in the middle).

[0058] At box 650, method 600 may include using a classification head neural network (e.g., Figure 4 The classification head 370 processes training aggregate features to predict one or more training features of training samples.

[0059] At box 660, method 600 may continue based on one or more training features of the training samples and the ground truth of the training samples (e.g., such as...). Figure 4 The comparison (shown by the dashed arrow in the diagram) modifies the parameters of at least one of the backbone neural network or the classification head neural network. In some embodiments, as shown in step 670, method 600 may further include modifying one or more parameters of at least one of the differential neural network or the aggregation neural network.

[0060] The systems and methods described herein can be used for a variety of purposes, including but not limited to: machine control (e.g., robots, vehicles, construction machinery, warehouse vehicles / equipment, autonomous, semi-autonomous and / or other types of machines), machine movement, machine driving, synthetic data generation, model training (e.g., using real data, augmented data and / or synthetic data, such as synthetic data generated by simulation platforms or systems, synthetic data generation techniques (e.g., but not limited to the techniques described herein)), perception, analysis operations, factory operations, generation and / or presentation of augmented reality (AR), virtual reality (VR), mixed reality (MR), etc., robot operation, medical operations, security monitoring (e.g., In smart city embodiments, autonomous or semi-autonomous machine applications, deep learning, environment simulation, object or role simulation and / or digital twins, data center processing, generative AI operations, conversational AI operations, operations involving visual language models, large language models, multimodal language models, optical transport simulation (e.g., ray tracing, path tracing, etc.), distributed or collaborative content creation of 3D assets (e.g., using common scene descriptor (USD) data (such as OpenUSD) and / or other data types), cloud computing, generative artificial intelligence (e.g., using one or more diffusion models, Transformer models, etc.) and / or any other suitable applications.

[0061] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), in-vehicle infotainment systems for autonomous or semi-autonomous machines, systems based on robots or robotic platforms, aviation systems, medical systems, marine systems, smart area monitoring systems, systems performing deep learning operations, systems performing simulation operations (e.g., driving or vehicle simulation, robot simulation, smart city or surveillance simulation, etc.), systems performing digital twin operations (e.g., combined with collaborative content creation platforms or systems, such as, but not limited to, NVIDIA's OMNIVERSE and / or other platforms, systems or services using USD or OpenUSD data types), edge device-based systems, systems integrating one or more virtual machines (VMs), and systems performing synthetic data generation operations. Systems that utilize one or more neural rendering fields (such as those using one or more neural rendering fields (NERF), Gaussian splashing, diffusion models, Transformer models, etc.), systems that are at least partially implemented in a data center, systems that perform conversational AI operations, systems that implement one or more language models (such as one or more large language models (LLM), one or more small language models (SLM), one or more visual language models (VLM), one or more multimodal language models, etc.), systems that perform optical transmission simulations, systems for creating collaborative content for 3D assets (such as those using generic scene descriptor (USD) data (such as OpenUSD), computer-aided design (CAD) data, 2D and / or 3D graphics or design data and / or other data types), systems that are at least partially implemented using cloud computing resources, and / or other types of systems.

[0062] Reasoning and training logic

[0063] Figure 7A Inference and / or training logic 715 is shown for performing inference and / or training operations associated with one or more embodiments.

[0064] In at least one embodiment, inference and / or training logic 715 may include, but is not limited to, code and / or data storage 701 for storing forward and / or output weights and / or input / output data, and / or other parameters for configuring neurons or layers of a neural network trained for and / or used for inference in one or more embodiments. In at least one embodiment, training logic 715 may include or be coupled to code and / or data storage 701 storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic, which includes integer and / or floating-point units (collectively referred to as arithmetic logic units (ALUs) or simple circuits). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, code and / or data storage 701 stores weight parameters and / or input / output data of each layer of a neural network trained or used in one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inference using one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 701 may be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0065] In at least one embodiment, any portion of the code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 701 may be a cache memory, dynamic random-addressable memory (“DRAM”), static random-addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 701 is internal or external to the processor, for example, or composed of DRAM, SRAM, flash memory, or some other storage type, may depend on the available on-chip or off-chip storage space, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0066] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 705 for storing backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, during training and / or inference using one or more embodiments, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of a neural network trained or used in one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 715 may include or be coupled to stored graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic, which includes integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).

[0067] In at least one embodiment, code (such as graph code) causes the architecture of the neural network corresponding to that code to load weights or other parameter information into the processor ALU. In at least one embodiment, any portion of the code and / or data storage 705 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice between the code and / or data storage 705 being internal or external to the processor, for example, whether it consists of DRAM, SRAM, flash memory, or some other type of storage, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the data batch size used in the inference and / or training of the neural network, or some combination of these factors.

[0068] In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be separate storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be combined storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data storage 701 and code and / or data storage 705 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0069] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 710 (including integer and / or floating-point units) for performing logical and / or mathematical operations at least in part based on or instructed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from layers or neurons within a neural network) stored in activation storage 720, which are functions of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, activation is activated in response to execution instructions or other code, linear algebraic and / or matrix-based mathematical generation performed by ALU 710, and the activation is stored in activation storage 720. The weight values ​​stored in code and / or data storage 705 and / or data storage 701 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters. Any or all of these can be stored in code and / or data storage 705 or code and / or data storage 701 or another on-chip or off-chip memory.

[0070] In at least one embodiment, one or more processors or other hardware logic devices or circuits include one or more ALUs 710, while in another embodiment, one or more ALUs 710 may be located outside the processor or other hardware logic device or the circuitry that uses them (e.g., a coprocessor). In at least one embodiment, one or more ALUs 710 may be included within an execution unit of a processor, or otherwise included in a group of ALUs accessible by the execution unit of the processor, which may be within the same processor or distributed among different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may share a processor or other hardware logic device or circuitry, while in another embodiment, they may be located in different processors or other hardware logic devices or circuitry, or in some combination of the same and different processors or other hardware logic devices or circuitry. In at least one embodiment, any portion of activation storage 720 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be retrieved and / or processed using the processor’s fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0071] In at least one embodiment, the active memory 720 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 720 may be wholly or partially located inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory 720 is internal to or external to the processor may depend on the available on-chip or off-chip storage, the latency requirements for training and / or inference functions, the batch size of data used in inference and / or training the neural network, or some combination of these factors. For example, it may include DRAM, SRAM, flash memory, or some other memory type.

[0072] In at least one embodiment, Figure 7A The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7A The inference and / or training logic 715 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware such as field programmable gate array (“FPGA”)

[0073] Figure 7B Inference and / or training logic 715 according to at least one embodiment is illustrated. In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, hardware logic, wherein computational resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM The inference processing unit (IPU) or from Intel. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7BThe inference and / or training logic 715 shown can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., field-programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 7B In at least one embodiment shown, each of code and / or data storage 701 and code and / or data storage 705 is associated with dedicated computing resources (e.g., computing hardware 702 and computing hardware 706), respectively. In at least one embodiment, each of computing hardware 702 and computing hardware 706 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) only on the information stored in code and / or data storage 701 and code and / or data storage 705, respectively, and the results of the function execution are stored in activation storage 720.

[0074] In at least one embodiment, each of the code and / or data storage 701 and 705 and the corresponding computing hardware 702 and 706 corresponds to a different layer of the neural network, such that activations obtained from one storage / computation pair 701 / 702 of the code and / or data storage 701 and computing hardware 702 provide input as input to the next storage / computation pair 705 / 706 of the code and / or data storage 705 and computing hardware 706, in order to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / computation pair 701 / 702 and 705 / 706 may correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic 715 after or in parallel with the storage / computation pairs 701 / 702 and 705 / 706.

[0075] Neural network training and deployment

[0076] Figure 8Training and deployment of a deep neural network according to at least one embodiment are illustrated. In at least one embodiment, an untrained neural network 806 is trained using a training dataset 802. In at least one embodiment, the training framework 804 is the PyTorch framework, while in other embodiments, the training framework 804 is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training frameworks. In at least one embodiment, the training framework 804 trains the untrained neural network 806 and enables it to be trained using the processing resources described herein to generate a trained neural network 808. In at least one embodiment, the weights may be randomly selected or pre-trained using a deep belief network. In at least one embodiment, training may be performed in a supervised, partially supervised, or unsupervised manner.

[0077] In at least one embodiment, supervised learning is used to train an untrained neural network 806, wherein training dataset 802 includes inputs paired with desired outputs for input, or wherein training dataset 802 includes inputs with known outputs and neural network 806 is manually graded output. In at least one embodiment, the untrained neural network 806 is trained in a supervised manner, and inputs from training dataset 802 are processed, and the resulting output is compared with a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through the untrained neural network 806. In at least one embodiment, training framework 804 adjusts the weights controlling the untrained neural network 806. In at least one embodiment, training framework 804 includes tools for monitoring the degree to which the untrained neural network 806 converges to a model (e.g., a trained neural network 808) adapted to generate the correct answer (e.g., result 814) based on input data (e.g., a new dataset 812). In at least one embodiment, training framework 804 repeatedly trains the untrained neural network 806 while adjusting the weights to improve the output of the untrained neural network 806 using a loss function and tuning algorithms (e.g., stochastic gradient descent). In at least one embodiment, the training framework 804 trains the untrained neural network 806 until the untrained neural network 806 reaches the desired accuracy. In at least one embodiment, the trained neural network 808 can then be deployed to perform any number of machine learning operations.

[0078] In at least one embodiment, unsupervised learning is used to train an untrained neural network 806, which attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised learning training dataset 802 will include input data without any associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 806 can learn groupings within the training dataset 802 and can determine how each input relates to the untrained dataset 802. In at least one embodiment, unsupervised training can be used to generate a self-organizing graph in the trained neural network 808, which is capable of performing operations useful for reducing the dimensionality of the new dataset 812. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows the identification of data points in the new dataset 812 that deviate from the normal patterns of the new dataset 812.

[0079] In at least one embodiment, semi-supervised learning can be used, a technique in which the training dataset 802 comprises a mixture of labeled and unlabeled data. In at least one embodiment, the training framework 804 can be used to perform incremental learning, for example, through transfer learning techniques. In at least one embodiment, incremental learning enables the trained neural network 808 to adapt to a new dataset 812 without forgetting the knowledge injected into the trained neural network 808 during initial training.

[0080] Reference Figure 9 , Figure 9 This is an example data flow diagram of process 900 for generating and deploying a processing and inference pipeline according to at least one embodiment. In at least one embodiment, process 900 may be deployed to perform game name recognition analysis and inference on user feedback data at one or more facilities 902, such as a data center.

[0081] In at least one embodiment, process 900 may be executed within training system 904 and / or deployment system 906. In at least one embodiment, training system 904 may be used to train, deploy, and implement machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use with deployment system 906. In at least one embodiment, deployment system 906 may be configured to offload processing and computing resources in a distributed computing environment to reduce infrastructure requirements of facility 902. In at least one embodiment, deployment system 906 may provide a pipeline platform for selecting, customizing, and implementing virtual instruments for use with computing devices at facility 902. In at least one embodiment, virtual instruments may include software-defined applications for performing one or more processing operations on feedback data. In at least one embodiment, one or more applications in the pipeline may use or invoke services of deployment system 906 (e.g., inference, visualization, computation, AI, etc.) during application execution.

[0082] In at least one embodiment, some applications used in the advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at facility 902 using feedback data 908 stored at facility 902 (e.g., imaging data) or feedback data 908 from another or more facilities, or a combination thereof. In at least one embodiment, training system 904 may be used to provide applications, services, and / or other resources to generate a deployable machine learning model for the operation of deployment system 906.

[0083] In at least one embodiment, the model registry 924 may be supported by an object storage system that supports version control and object metadata. In at least one embodiment, it may be available from within a cloud platform via, for example, cloud storage (e.g., Figure 10 The system uses a Cloud 1026-compatible Application Programming Interface (API) to access object storage. In at least one embodiment, machine learning models within the model registry 924 can be uploaded, listed, modified, or deleted by the developer or partner of the system interacting with the API. In at least one embodiment, the API can provide access to methods that allow users with appropriate credentials to associate models with applications, enabling the models to be executed as part of the containerized instantiation of the application.

[0084] In at least one embodiment, training pipeline 1004 ( Figure 10This can include situations where facility 902 is training its own machine learning model or has an existing machine learning model that needs optimization or updating. In at least one embodiment, feedback data 908 can be received from various channels (such as forums, web forms, etc.). In at least one embodiment, once feedback data 908 is received, AI-assisted annotation 910 can be used to help generate annotations corresponding to the feedback data 908 for use as ground truth data for the machine learning model. In at least one embodiment, AI-assisted annotation 910 can include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that can be trained to generate annotations corresponding to certain types of feedback data 908 (e.g., from certain devices), and / or certain types of anomalies in the feedback data 908. In at least one embodiment, AI-assisted annotation 910 can then be used directly, or it can be adjusted or fine-tuned using annotation tools to generate ground truth data. In at least one embodiment, in some examples, labeled data 912 can be used as ground truth data for training the machine learning model. In at least one embodiment, AI-assisted annotations 910, labeled data 912, or a combination thereof can be used as ground truth data for training a machine learning model (e.g., via model training 914). In at least one embodiment, the trained machine learning model can be referred to as output model 916 and can be used by deployment system 906, as described herein.

[0085] In at least one embodiment, training pipeline 1004 ( Figure 10This can include situations where facility 902 requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 906, but facility 902 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for this purpose). In at least one embodiment, an existing machine learning model can be selected from model registry 924. In at least one embodiment, model registry 924 can include machine learning models trained to perform various inference tasks on imaging data. In at least one embodiment, the machine learning model in model registry 924 can be trained on imaging data from a different facility (e.g., a remote facility) instead of facility 902. In at least one embodiment, the machine learning model may have already been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a specific location (e.g., which may be in the form of feedback data 908), training can be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of the imaging data off-site (e.g., in compliance with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model has been trained or partially trained at one location, a machine learning model can be added to a model registry 924. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be used in the model registry 924. In at least one embodiment, a machine learning model (referred to as output model 916) can then be selected from the model registry 924 and used in a deployment system 906 to perform one or more processing tasks for one or more applications of the deployment system.

[0086] In at least one embodiment, training pipeline 1004 ( Figure 10This can be used in scenarios including facility 902, which requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 906, but facility 902 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for this purpose). In at least one embodiment, the machine learning model selected from model registry 924 may not be fine-tuned or optimized for the feedback data 908 generated at facility 902 due to population differences, genetic variations, robustness, anomalous diversity of the training data, and / or other problems with the training data used to train the machine learning model. In at least one embodiment, AI-assisted annotation 910 can be used to help generate annotations corresponding to the feedback data 908 for use as ground truth data for retraining or updating the machine learning model. In at least one embodiment, labeled data 912 can be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 914. In at least one embodiment, model training 914 (e.g., AI-assisted annotation 910, labeled data 912, or a combination thereof) can be used as ground truth data for retraining or updating the machine learning model.

[0087] In at least one embodiment, the deployment system 906 may include software 918, service 920, hardware 922, and / or other components, features, and functions. In at least one embodiment, the deployment system 906 may include a software "stack" such that software 918 can be built on top of service 920 and can be used to perform some or all of the processing tasks, and service 920 and software 918 can be built on top of hardware 922 and can be used to perform processing, storage, and / or other computing tasks of the deployment system 906.

[0088] In at least one embodiment, software 918 may include any number of different containers, each of which may perform an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in an advanced processing and inference pipeline. In at least one embodiment, for each type of computing device, there may be any number of containers that may perform data processing tasks on feedback data 908 (or other data types, such as those described herein). In at least one embodiment, in addition to the containers that receive and configure imaging data for use by each container and / or for use by facility 902 after processing through the pipeline, the advanced processing and inference pipeline may also be defined based on the selection of different containers desired or required for processing feedback data 908 (e.g., to convert output back to a usable data type for storage and display in facility 902). In at least one embodiment, a combination of containers within software 918 (e.g., which constitute a pipeline) may be referred to as a virtual instrument (as described in more detail herein), and the virtual instrument may utilize service 920 and hardware 922 to perform some or all of the processing tasks of the application instantiated in the container.

[0089] In at least one embodiment, data may be preprocessed as part of a data processing pipeline to prepare it for processing by one or more applications. In at least one embodiment, post-processing may be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare output data for the next application and / or to prepare output data for user transmission and / or use (e.g., as a response to an inference request). In at least one embodiment, the inference task may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include the output model 916 of the training system 904.

[0090] In at least one embodiment, the tasks of the data processing pipeline can be encapsulated in one or more containers, each container representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications can be published to a private (e.g., limited access) area of ​​a container registry (described in more detail herein), and trained or deployed models can be stored in a model registry 924 and associated with one or more applications. In at least one embodiment, an image of the application (e.g., a container image) can be used in the container registry, and once a user selects an image from the container registry for deployment in the pipeline, that image can be used to generate containers for instantiation of the application for use by the user's system.

[0091] In at least one embodiment, the developer can develop, publish, and store an application (e.g., as a container) for performing processing and / or inference on the provided data. In at least one embodiment, a software development kit (SDK) associated with the system can be used to perform development, publication, and / or storage (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application can be tested locally using the SDK (e.g., at a first facility, on data from a first facility), the SDK serving as a system (e.g.,...). Figure 10 System 1000 may support at least some services 920. In at least one embodiment, once verified by system 1000 (e.g., for accuracy, etc.), the application becomes available in the container registry for users (e.g., hospitals, clinics, laboratories, healthcare providers, etc.) to select and / or implement one or more processing tasks on data at the user's facility (e.g., a second facility).

[0092] In at least one embodiment, the developer can then share the application or container over a network for the system (e.g., Figure 10 The system 900 allows for user access and use. In at least one embodiment, a completed and validated application or container may be stored in a container registry, and the associated machine learning model may be stored in a model registry 924. In at least one embodiment, a requesting entity (which provides an inference or image processing request) may browse the container registry and / or model registry 924 for applications, containers, datasets, machine learning models, etc., select the desired combination of elements to include in the data processing pipeline, and submit a processing request. In at least one embodiment, the request may include input data necessary to execute the request, and / or may include a selection of the application and / or machine learning model to be executed when the request is processed. In at least one embodiment, the request may then be passed to one or more components of the deployment system 906 (e.g., the cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 906 may include referencing elements (e.g., applications, containers, models, etc.) selected from the container registry and / or model registry 924. In at least one embodiment, once results are generated through the pipeline, the results may be returned to the user for reference (e.g., for viewing in a suite of viewing applications executed on a local machine, local workstation, or terminal).

[0093] In at least one embodiment, service 920 may be utilized to assist in processing or executing applications or containers in the pipeline. In at least one embodiment, service 920 may include computing services, collaborative content creation services, simulation services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, service 920 may provide functionality common to one or more applications in software 918, thus abstracting functionality into services that can be invoked or utilized by applications. In at least one embodiment, the functionality provided by service 920 can operate dynamically and more efficiently, while also allowing applications to process data in parallel (e.g., using...). Figure 10 The parallel computing platform 1030 in the system can be scaled well. In at least one embodiment, it is not required that each application providing the same functionality as the shared service 920 must have a corresponding instance of the service 920, but rather that the service 920 can be shared between and among various applications. In at least one embodiment, as a non-limiting example, the service may include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included, which can provide the ability to train and / or retrain machine learning models.

[0094] In at least one embodiment, where service 920 includes an AI service (e.g., an inference service), as part of application execution, one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) can be executed by invoking (e.g., as an API call) the inference service (e.g., an inference server) to execute one or more machine learning models or their processing. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, the application can invoke the inference service to execute the machine learning models for performing one or more processing operations associated with the segmentation task. In at least one embodiment, the software 918 implementing the advanced processing and inference pipeline can be pipelined because each application can invoke the same inference service to execute one or more inference tasks.

[0095] In at least one embodiment, hardware 922 may include a GPU, CPU, graphics card, AI / deep learning system (e.g., an AI supercomputer, such as NVIDIA's DGX). TM(Supercomputer system), cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 922 may be used to provide efficient, specially built support for software 918 and services 920 in deployment system 906. In at least one embodiment, GPU processing may be used to perform local processing (e.g., at facility 902) within the AI / deep learning system, in the cloud system, and / or other processing components of deployment system 906 to improve the efficiency, accuracy, and performance of game name recognition.

[0096] In at least one embodiment, as a non-limiting example, regarding deep learning, machine learning and / or high-performance computing, simulation and visual computing, software 918 and / or service 920 may be optimized for GPU processing. In at least one embodiment, at least some of the computing environment in which the deployment system 906 and / or training system 904 are located may have GPU-optimized software (e.g., NVIDIA DGX). TM The system's hardware and software combination is executed in a data center, one or more supercomputers, or high-performance computing systems. In at least one embodiment, as described herein, hardware 922 may include any number of GPUs that can be invoked to perform data processing in parallel. In at least one embodiment, the cloud platform may also include GPU-optimized execution for deep learning tasks, GPU processing for machine learning tasks, or other computing tasks. In at least one embodiment, AI / deep learning supercomputers and / or GPU-optimized software (e.g., such as NVIDIA's DGX) may be used. TM The system provides a hardware abstraction and scaling platform to execute cloud platforms (e.g., NVIDIA's NGC). TM In at least one embodiment, the cloud platform can integrate application container cluster systems or coordination systems (e.g., KUBERNETES) across multiple GPUs to achieve seamless scaling and load balancing.

[0097] Figure 10 This is a system diagram of an example system 1000 for generating and deploying a deployment pipeline according to at least one embodiment. In at least one embodiment, system 1000 can be used to implement Figure 9 The process 900 and / or other processes include advanced processing and inference pipelines. In at least one embodiment, system 1000 may include a training system 904 and a deployment system 906. In at least one embodiment, the training system 904 and deployment system 906 may be implemented using software 918, services 920, and / or hardware 922, as described herein.

[0098] In at least one embodiment, system 1000 (e.g., training system 904 and / or deployment system 906) may be implemented in a cloud computing environment (e.g., using cloud 1026). In at least one embodiment, system 1000 may be implemented locally (regarding facilities) or as a combination of cloud computing resources and local computing resources. In at least one embodiment, access to the API in cloud 1026 may be restricted to authorized users by establishing security measures or protocols. In at least one embodiment, the security protocol may include a network token, which may be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and may carry appropriate authorization. In at least one embodiment, the API of the virtual instrument (described herein) or other instances of system 1000 may be restricted to a set of public Internet Service Providers (ISPs) that have been audited or authorized for interaction.

[0099] In at least one embodiment, the various components of system 1000 may communicate with each other and among themselves using any of a variety of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communication between facilities and components of system 1000 (e.g., for sending inference requests, for receiving the results of inference requests, etc.) may be transmitted via one or more data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0100] In at least one embodiment, similar to the description herein. Figure 9 As described, training system 904 can execute training pipeline 1004. In at least one embodiment, where deployment system 906 uses one or more machine learning models in deployment pipeline 1010, training pipeline 1004 can be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1006 (e.g., without retraining or updating). In at least one embodiment, as a result of training pipeline 1004, output model 916 can be generated. In at least one embodiment, training pipeline 1004 can include any number of processing steps, AI-assisted annotation 910, tagging or annotating feedback data 908 to generate labeled data 912, selecting a model from a model registry, model training 914, training, retraining, or updating the model, and / or other processing steps. In at least one embodiment, different training pipelines 1004 can be used for different machine learning models used by deployment system 906. In at least one embodiment, similar to the approach regarding... Figure 9 The training pipeline 1004 described in the first example can be used for the first machine learning model, similar to the one described above. Figure 9 The training pipeline 1004 described in the second example can be used for a second machine learning model, similar to the one described above. Figure 9 The training pipeline 1004 of the third example described can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 904 can be used according to the requirements of each respective machine learning model. In at least one embodiment, one or more machine learning models may have already been trained and are ready for deployment, so the training system 904 may not perform any processing on the machine learning models, and the machine learning models may be implemented by the deployment system 906.

[0101] In at least one embodiment, one or more output models 916 and / or pre-trained models 1006 may comprise any type of machine learning model. In at least one embodiment, and not limited thereto, the machine learning model used by system 1000 may include models using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recursion, perceptrons, long short-term memory (LSTM), Bi-LSTM, Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0102] In at least one embodiment, training pipeline 1004 may include AI-assisted annotation. In at least one embodiment, labeled data 912 (e.g., conventional annotation) can be generated using any number of techniques. In at least one embodiment, in some examples, labels or other annotations can be generated by drawing programs (e.g., annotation programs), computer-aided design (CAD) programs, tagging programs, another type of application suitable for generating ground truth annotations or labels, and / or can be hand-drawn. In at least one embodiment, ground truth data can be synthetically generated (e.g., generated from computer models or renderings), realistically generated (e.g., designed and generated from real-world data), automatically generated by machines (e.g., extracting features from data using feature analysis and learning, and then generating labels), manually annotated (e.g., taggers or annotation experts, defining the placement of labels), and / or combinations thereof. In at least one embodiment, for each instance of feedback data 908 (or other data types used by the machine learning model), corresponding ground truth data generated by training system 904 may exist. In at least one embodiment, AI-assisted annotation may be performed as part of deployment pipeline 1010; supplementing or replacing AI-assisted annotation included in training pipeline 1004. In at least one embodiment, system 1000 may include a multi-layer platform, which may include a software layer (e.g., software 918) of a diagnostic application (or other application type) that can perform one or more medical imaging and diagnostic functions.

[0103] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or certified API that can invoke (e.g., call) an application or container from an external environment (e.g., facility 902). In at least one embodiment, the application may then invoke or execute one or more services 920 to perform computational, AI, or visualization tasks associated with their respective applications, and the software 918 and / or service 920 may utilize hardware 922 to perform processing tasks efficiently and effectively.

[0104] In at least one embodiment, deployment system 906 may execute deployment pipeline 1010. In at least one embodiment, deployment pipeline 1010 may include any number of applications, which may be sequential, non-sequential, or otherwise applied to feedback data (and / or other data types) – including AI-assisted annotations, as described above. In at least one embodiment, as described herein, deployment pipeline 1010 for an individual device may be referred to as a virtual instrument for the device. In at least one embodiment, for a single device, there may be more than one deployment pipeline 1010, depending on the desired information from the data generated from the device.

[0105] In at least one embodiment, the applications that can be used to deploy pipeline 1010 may include any application that can perform processing tasks on feedback data or other data from the device. In at least one embodiment, since various applications can share common image operations, in some embodiments, data augmentation libraries (e.g., as one of services 920) may be used to accelerate these operations. In at least one embodiment, to avoid bottlenecks in traditional processing methods that rely on CPU processing, parallel computing platform 1030 may be used for GPU acceleration of these processing tasks.

[0106] In at least one embodiment, deployment system 906 may include user interface 1014 (e.g., graphical user interface, web interface, etc.) which may be used to select applications to be included in deployment pipeline 1010, deploy applications, modify or change applications or their parameters or configurations, use and interact with deployment pipeline 1010 during setup and / or deployment, and / or otherwise interact with deployment system 906. In at least one embodiment, although not shown with respect to training system 904, UI 1014 (or a different user interface) may be used to select models to be used in deployment system 906, to select models to be trained or retrained in training system 904, and / or to otherwise interact with training system 904. In at least one embodiment, training system 904 and deployment system 906 may include DICOM adapters 1002A and 1002B.

[0107] In at least one embodiment, in addition to the application coordination system 1028, a pipeline manager 1012 may be used to manage interactions between applications or containers deployed through the pipeline 1010 and services 920 and / or hardware 922. In at least one embodiment, the pipeline manager 1012 may be configured to facilitate interactions from application to application, from application to service 920, and / or from application or service to hardware 922. In at least one embodiment, although shown as included in software 918, this is not intended to be limiting, and in some examples, the pipeline manager 1012 may be included in service 920. In at least one embodiment, the application coordination system 1028 (e.g., Kubernetes, DOCKER, etc.) may include a container coordination system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline 1010 (e.g., rebuilding applications, splitting applications, etc.) with individual containers, each application can execute in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0108] In at least one embodiment, each application and / or container (or its image) can be developed, modified, and deployed independently (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer). This allows focus on the tasks of a single application and / or container without being hindered by the tasks of other applications or containers. In at least one embodiment, the pipeline manager 1012 and the application coordination system 1028 can facilitate communication and collaboration between different containers or applications. In at least one embodiment, the application coordination system 1028 and / or the pipeline manager 1012 can facilitate communication and resource sharing between and within each application or container, provided that the expected inputs and / or outputs of each container or application are known to the system (e.g., based on the application or container's architecture). In at least one embodiment, since one or more applications or containers in the deployment pipeline 1010 can share the same services and resources, the application coordination system 1028 can coordinate, load balance, and determine the sharing of services or resources between and within the various applications or containers. In at least one embodiment, the scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. Therefore, in at least one embodiment, taking into account the system's needs and availability, the scheduler can allocate resources to different applications and distribute resources between and among applications. In some examples, the scheduler (and / or other components of the application coordination system 1028) can determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as Quality of Service (QoS), the urgency of data output (e.g., to determine whether to perform real-time processing or delayed processing), etc.

[0109] In at least one embodiment, service 920, utilized and shared by applications or containers in deployment system 906, may include computing service 1016, collaborative content creation service 1017, AI service 1018, simulation service 1019, visualization service 1020, and / or other service types. In at least one embodiment, an application may invoke (e.g., execute) one or more services 920 to perform processing operations for the application. In at least one embodiment, an application may utilize computing service 1016 to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more computing services 1016 may be utilized to perform parallel processing (e.g., using parallel computing platform 1030) to process data substantially simultaneously through one or more applications and / or one or more tasks of a single application. In at least one embodiment, parallel computing platform 1030 (e.g., NVIDIA's CUDA) may implement general-purpose computing on a GPU (GPGPU) (e.g., GPU 1022). In at least one embodiment, the software layer of parallel computing platform 1030 may provide access to the GPU's virtual instruction set and parallel computing elements to execute computing kernels. In at least one embodiment, the parallel computing platform 1030 may include memory, and in some embodiments, memory may be shared between and within multiple containers, and / or between and within different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within containers to use the same data from a shared memory segment of the parallel computing platform 1030 (e.g., where multiple different stages of one or more applications are processing the same information). In at least one embodiment, instead of copying data and moving it to different locations in memory (e.g., read / write operations), the same data in the same memory location can be used for any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, this information about the new location of the data can be stored and shared between applications because the resulting data from processing is used to generate new data. In at least one embodiment, the location of the data, and the location of the updated or modified data, may be part of the definition of how the payload in the container is understood.

[0110] In at least one embodiment, AI service 1018 may be used to perform an inference service for executing a machine learning model associated with the application (e.g., a task to perform one or more processing tasks of the application). In at least one embodiment, AI service 1018 may utilize AI system 1024 to execute a machine learning model (e.g., a neural network such as a CNN) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, the application deploying pipeline 1010 may use one or more output models 916 from training system 904 and / or other models from the application to perform inference on imaging data (e.g., DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.). In at least one embodiment, two or more examples of inference using application coordination system 1028 (e.g., a scheduler) may be available. In at least one embodiment, the first category may include a high-priority / low-latency path that can implement a higher service level protocol, such as for performing inference on urgent requests in emergency situations or for radiologists during diagnostic procedures. In at least one embodiment, the second category may include standard priority paths, which can be used for requests that may not be urgent or for situations where analysis can be performed at a later time. In at least one embodiment, the application coordination system 1028 may allocate resources (e.g., services 920 and / or hardware 922) based on priority paths for different inference tasks of the AI ​​service 1018.

[0111] In at least one embodiment, shared memory may be installed into AI service 1018 in system 1000. In at least one embodiment, shared memory may operate as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a set of API instances of deployment system 906 may receive the request and may select one or more instances (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request may be fed into a database, and if not already in the cache, a machine learning model may be located from model registry 924. A verification step may ensure that an appropriate machine learning model is loaded into the cache (e.g., shared memory), and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not already running or there are not enough instances of the application, a scheduler (e.g., the scheduler of pipeline manager 1012) may be used to start the application referenced in the request. In at least one embodiment, if an inference server has not yet been started to execute the model, an inference server may be started. In at least one embodiment, any number of inference servers may be started for each model. In at least one embodiment, in a pull model that clusters inference servers, the model can be cached whenever load balancing is favorable. In at least one embodiment, the inference servers can be statically loaded into the corresponding distributed servers.

[0112] In at least one embodiment, an inference server running in a container can be used to perform inference. In at least one embodiment, an instance of the inference server can be associated with a model (and optionally multiple versions of the model). In at least one embodiment, if an instance of the inference server does not exist when a request to perform inference on the model is received, a new instance can be loaded. In at least one embodiment, when the inference server is started, a model can be passed to the inference server, allowing the same container to be used to serve different models, as long as the inference server runs as different instances.

[0113] In at least one embodiment, during application execution, an inference request for a given application can be received and a container (e.g., an instance of a hosted inference server) can be loaded (if not already loaded), and a launcher can be invoked. In at least one embodiment, preprocessing logic within the container can (e.g., using a CPU and / or GPU) load, decode, and / or perform any additional preprocessing on the incoming data. In at least one embodiment, once the data is ready for inference, the container can infer the data as needed. In at least one embodiment, this can include a single inference call for an image (e.g., a hand X-ray) or can request inference for hundreds of images (e.g., a chest CT scan). In at least one embodiment, the application can summarize the results before completion, which may include, but is not limited to, a single confidence score, pixel-level segmentation, voxel-level segmentation, generating visualizations, or generating text to summarize the results. In at least one embodiment, different priorities can be assigned to different models or applications. For example, some models may have a real-time (turnaround time less than 1 minute) priority, while other models may have a lower priority (e.g., turnaround time less than 10 minutes). In at least one embodiment, model execution time can be measured from the requesting agency or entity, and may include cooperative network traversal time and inference service execution time.

[0114] In at least one embodiment, the transfer of requests between service 920 and the inference application can be hidden behind a software development kit (SDK) and robust transfer can be provided via queues. In at least one embodiment, requests are placed in queues via an API for individual application / tenant ID combinations, and the SDK pulls requests from the queues and provides them to the application. In at least one embodiment, the name of the queue can be provided in the environment from which the SDK picks up the request. In at least one embodiment, asynchronous communication via queues may be useful because it allows any instance of the application to pick up work as soon as it becomes available. In at least one embodiment, results can be sent back via queues to ensure no data loss. In at least one embodiment, queues can also provide the ability to partition work, as the highest priority work can go into a queue connected to a majority of instances of the application, while the lowest priority work can go into a queue connected to a single instance that processes tasks in the order they are received. In at least one embodiment, the application can run on a GPU-accelerated instance generated in cloud 1026, and the inference service can perform inference on the GPU.

[0115] In at least one embodiment, visualization service 1020 can be used to generate visualizations for viewing the output of application and / or deployment pipeline 1010. In at least one embodiment, visualization service 1020 can utilize GPU 1022 to generate visualizations. In at least one embodiment, visualization service 1020 can implement rendering effects such as ray tracing or other light transport simulation techniques to generate higher quality visualizations. In at least one embodiment, visualizations can include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slicing, virtual reality display, augmented reality display, etc. In at least one embodiment, a virtualized environment can be used to generate virtual interactive displays or environments (e.g., virtual environments) for system users (e.g., doctors, nurses, radiologists, etc.) to interact with. In at least one embodiment, visualization service 1020 can include an internal visualizer, cinematic and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0116] In at least one embodiment, hardware 922 may include GPU 1022, AI system 1024, cloud 1026, and / or any other hardware for executing training system 904 and / or deployment system 906. In at least one embodiment, GPU 1022 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that can be used to perform processing tasks for any feature or function of computing service 1016, collaborative content creation service 1017, AI service 1018, simulation service 1019, visualization service 1020, other services, and / or software 918. For example, for AI service 1018, GPU 1022 may be used to perform preprocessing on imaging data (or other data types used by machine learning models), postprocessing on the output of machine learning models, and / or inference (e.g., to execute machine learning models). In at least one embodiment, cloud 1026, AI system 1024, and / or other components of system 1000 may use GPU 1022. In at least one embodiment, cloud 1026 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1024 may use a GPU, and one or more AI systems 1024 may be used to perform cloud 1026 (or at least part of a task for deep learning or inference). Similarly, although hardware 922 is shown as a discrete component, this is not intended to be limiting, and any component of hardware 922 may be combined with or utilized by any other component of hardware 922.

[0117] In at least one embodiment, the AI ​​system 1024 may include a specially built computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, in addition to a CPU, RAM, memory, and / or other components, features, or functions, the AI ​​system 1024 (e.g., NVIDIA's DGX) TM It may also include GPU-optimized software (e.g., a software stack) that can be executed using multiple GPUs 1022. In at least one embodiment, one or more AI systems 1024 may be implemented in the cloud 1026 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1000.

[0118] In at least one embodiment, cloud 1026 may include GPU-accelerated infrastructure (e.g., NVIDIA's NGC). TM The cloud 1026 can provide a GPU-optimized platform for performing processing tasks of system 1000. In at least one embodiment, cloud 1026 may include AI system 1024 for performing one or more AI-based tasks of system 1000 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1026 may be integrated with application coordination system 1028 utilizing multiple GPUs to achieve seamless scaling and load balancing between and within applications and services 920. In at least one embodiment, as described herein, cloud 1026 may be responsible for performing at least some of the services 920 of system 1000, including computing service 1016, AI service 1018, and / or visualization service 1020. In at least one embodiment, cloud 1026 may perform large and small batch inference (e.g., perform NVIDIA's TENSOR RT), providing accelerated parallel computing APIs and platform 1030 (e.g., NVIDIA's...). The application coordination system 1028 (e.g., KUBERNETES) provides graphics rendering APIs and platforms (e.g., for ray tracing, 2D graphics, 3D graphics and / or other rendering techniques to produce higher quality cinematic effects), and / or may provide other functions for the system 1000.

[0119] In at least one embodiment, to protect patient confidentiality (e.g., in cases of off-site use of patient data or records), cloud 1026 may include a registry—e.g., a deep learning container registry. In at least one embodiment, the registry may store containers for instantiating applications that can perform preprocessing, post-processing, or other processing tasks on patient data. In at least one embodiment, cloud 1026 may receive data, including patient data and sensor data in containers, perform the requested processing only on the sensor data in those containers, and then forward the output and / or visualization to appropriate parties and / or devices (e.g., local medical devices for visualization or diagnosis), without extracting, storing, or otherwise accessing the patient data. In at least one embodiment, the confidentiality of patient data is maintained in accordance with HIPAA and / or other data regulations.

[0120] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0121] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar references, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (referring to a physical connection where not modified) should be interpreted as partially or wholly contained, attached to, or joined together, even with some intervention. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0122] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” means multiple items). In at least one embodiment, the number of items in the multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” rather than “based on only”.

[0123] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that are executed jointly on one or more processors via hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within transient signal transceivers. In at least one embodiment, code (e.g., executable code or source code) is stored on one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media lack all the code, but the multiple non-transitory computer-readable storage media collectively store all the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0124] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the processes described herein individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating in different ways, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0125] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not constitute a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0126] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference as if each reference were individually and specifically indicated to be incorporated herein by reference and the entire contents of which are described herein.

[0127] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0128] Unless otherwise expressly stated, it will be understood that throughout this specification, terms such as “processing,” “computing,” “determining,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that process and / or convert data represented as physical quantities (e.g., electrons) in the registers and / or memory of the computing system into other data represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0129] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or intermittently, sequentially, or in parallel. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0130] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an application programming interface, or an inter-process communication mechanism.

[0131] While the description herein illustrates exemplary embodiments of the described technologies, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for descriptive purposes, various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0132] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A method, the method comprising: The images of the samples are processed using at least one backbone neural network (NN) to generate sample features representing the samples; Multiple reference features are obtained, each of which represents a corresponding reference image among multiple reference images; Multiple differential features are generated, each of the multiple differential features representing a corresponding difference between the sample feature and a corresponding reference feature among the multiple reference features; Predict one or more characteristics of the sample based on aggregated features obtained by aggregating the multiple differential features; and Utilize one or more of the aforementioned characteristics to adjust one or more operations of the manufacturing process.

2. The method according to claim 1, wherein, Obtaining the plurality of reference features includes: The plurality of reference images are processed using at least one backbone neural network (NN).

3. The method according to claim 1, wherein, Obtaining the plurality of reference features includes: Retrieve from the memory device the plurality of reference features previously generated by processing the plurality of reference images using the at least one backbone neural network NN.

4. The method according to claim 1, wherein, Generating the multiple differential features includes: The input, including the sample features and the corresponding reference features, is processed using at least one differential neural network (NN).

5. The method according to claim 1, wherein, Predicting one or more characteristics of the sample includes: The aggregated features are processed using a classification head neural network (NN) to generate the probability that the sample conforms to the specifications of the manufacturing process.

6. The method according to claim 1, wherein, Predicting one or more characteristics of the sample includes: The aggregated features are processed using a classification head neural network (NN) to generate multiple probabilities, each of which represents the likelihood that the sample conforms to a corresponding test category among multiple test categories.

7. The method according to claim 1, wherein, The at least one backbone neural network NN includes at least one of the following: Convolutional Neural Network (NN) Transformer Neural Network (NN), or Visual Transformer Neural Network (NN).

8. The method according to claim 1, wherein, The aggregation of the multiple reference features includes: The aggregated feature is obtained by averaging the multiple differential features.

9. The method according to claim 1, wherein, The aggregation of the multiple reference features includes: The multiple differential features are processed by a convergent neural network (NN).

10. The method according to claim 9, wherein, The aggregated neural network NN includes at least one of the following: One or more fully connected neuron layers, or One or more neuronal attention blocks.

11. The method according to claim 9, wherein, The operation of updating the at least one backbone neural network NN includes: The training images of the training samples are processed using the at least one backbone neural network NN to generate training sample features representing the training samples. The at least one backbone neural network NN is used to process multiple training reference images to generate multiple corresponding training reference features; Multiple training differential features are generated, each of the multiple training differential features representing a corresponding difference between the training sample feature and the corresponding training reference feature among the multiple training reference features; Aggregate the multiple training differential features to obtain training aggregated features; The training aggregate features are processed using a classification head neural network (NN) to predict one or more training features of the training samples; and Based on a comparison of one or more training features of the training samples with the true values ​​of the training samples, one or more parameters of at least one of the backbone neural network NN or the classification head neural network NN are modified.

12. The method according to claim 11, wherein, Generating the plurality of training differential features includes using at least one differential neural network (NN), and wherein aggregating the plurality of training differential features includes using the aggregation neural network (NN), the method further includes: Modify one or more parameters of at least one of the at least one differential neural network NN or the at least one aggregate neural network NN.

13. A system comprising: Memory devices; as well as One or more processing devices, said one or more processing devices being communicatively coupled to said memory device, said one or more processing devices being used for: The images of the samples are processed using at least one backbone neural network (NN) to generate sample features representing the samples; Obtain multiple reference features representing multiple reference images associated with acceptable tolerances in the manufacturing process; Generate multiple differential features, each of which represents a corresponding difference between the sample feature and a corresponding reference feature among the multiple reference features; and Integrate one or more characteristics of the sample into the manufacturing process, wherein the one or more characteristics are predicted based on the plurality of differential features.

14. The system according to claim 13, wherein, In order to obtain the plurality of reference features, the one or more processing devices are configured to perform at least one of the following: The plurality of reference images are processed using the at least one backbone neural network NN; or Retrieve from the memory device the plurality of reference features previously generated by processing the plurality of reference images using the at least one backbone neural network NN.

15. The system according to claim 13, wherein, In order to generate the plurality of differential features, the one or more processing devices are used to: The input, including the sample features and the corresponding reference features, is processed using at least one differential neural network (NN).

16. The system according to claim 13, wherein, In order to predict one or more characteristics of the sample, the one or more processing devices are configured to perform at least one of the following: The multiple differential features are processed using a classification head neural network (NN) to generate the probability that the sample conforms to the specifications of the manufacturing process; or The multiple differential features are processed using a classification head neural network (NN) to generate multiple probabilities, each of which represents the likelihood that the sample conforms to a corresponding test category among multiple test categories.

17. The system according to claim 13, wherein, In order to aggregate the plurality of reference features, the one or more processing devices are configured to perform at least one of the following: The aggregated feature is obtained by averaging the multiple differential features. or The multiple differential features are processed by a convergent neural network (NN).

18. The system according to claim 13, wherein, The at least one backbone neural network NN includes at least one of the following: Convolutional Neural Network (NN) Transformer Neural Network (NN), or Visual Transformer Neural Network (NN).

19. The system according to claim 13, wherein, The system is included in at least one of the following: In-vehicle infotainment systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system used to perform one or more medical procedures; A system used to perform one or more plant operations; A system used to perform one or more analytical operations; A system that implements one or more inference microservices; A system for performing optical transmission simulation; A system for collaborative content creation using 3D assets; A system used to perform deep learning operations; Systems implemented using edge devices; A system for generating or presenting at least one of virtual reality content, mixed reality content, or augmented reality content; Systems implemented using robots; A system for performing one or more conversational AI operations; A system that implements one or more large language model LLMs; A system that implements one or more small language models (SLMs); A system that implements one or more Visual Language Models (VLMs); A system that implements one or more multimodal language models; A system that implements one or more language models; A system for performing one or more generative AI operations; A system for generating synthetic data; A system that includes one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

20. One or more processors, said one or more processors comprising: The processing circuit is used to perform a first stage and a second stage of neural network operation. The first stage of the neural network operation represents an image of the manufactured product and multiple reference images via corresponding image features. The second stage of the neural network operation jointly processes the image features to identify the degree of conformity between the manufactured product and the specifications of the manufactured product.

Images