Pipeline for efficient training and deployment of machine learning models

Abstract

Pipelines for efficient training and deployment of machine learning models are disclosed, specifically apparatuses, systems, and techniques that provide pipelines for efficient training, retraining, configuration, deployment, and use of machine learning models for inference in user-specific platforms.

Description

Technical Field

[0001] At least one embodiment relates to processing resources for performing and facilitating artificial intelligence. For example, at least one embodiment relates to providing pipelines for efficiently training, configuring, deploying, and using machine learning models on a user-specific platform. Background Technology

[0002] Machine learning is commonly used in office and hospital environments, robotic automation, security applications, autonomous transportation, law enforcement, and many other settings. In particular, machine learning is applied to audio and video processing, such as sound, speech, and object recognition. A popular approach to machine learning involves training a computational system using training data (sound, images, and / or other data) to identify data patterns that facilitate data classification, such as training on the presence of specific types of objects in an image or training on the presence of specific words in speech. Training can be supervised or unsupervised. Machine learning models can utilize various computational algorithms, such as decision tree algorithms (or other rule-based algorithms), artificial neural networks, and more. During the subsequent deployment phase (also known as the "inference phase"), new data is fed into the trained machine learning model, and the patterns and features established during training can be used to identify various target objects, sounds, or sentences of interest. Attached Figure Description

[0003] Figure 1 It is a block diagram of an example architecture for a customizable pipeline that supports training, configuring, and deploying one or more machine learning models, according to at least some embodiments;

[0004] Figure 2A According to at least some embodiments Figure 1 A block diagram of an example architecture for a training engine with a customizable pipeline;

[0005] Figure 2B According to at least some embodiments Figure 1 A block diagram of an example architecture for the build and deployment phases of a customizable pipeline;

[0006] Figure 3 It is an example computing device according to at least some embodiments capable of implementing a determinate pipeline that supports training, configuring and deploying one or more machine learning models;

[0007] Figure 4 This is a block diagram of an example customizable pipeline for natural language processing of speech using one or more machine learning models, according to at least some embodiments.

[0008] Figure 5 This is a flowchart illustrating example methods for providing a customizable pipeline that supports the training, configuration, and deployment of one or more machine learning models, according to at least some embodiments.

[0009] Figure 6 This is a flowchart of an example method using a deployed customizable machine pipeline machine learning model according to at least some embodiments;

[0010] Figure 7A The inference and / or training logic according to at least one embodiment is illustrated;

[0011] Figure 7B The inference and / or training logic according to at least one embodiment is illustrated;

[0012] Figure 8 The training and deployment of a neural network according to at least one embodiment are illustrated;

[0013] Figure 9 This is an example data flow diagram of an advanced computing pipeline according to at least one embodiment;

[0014] Figure 10 This is a system diagram of an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment. Detailed Implementation

[0015] Machine learning has become indispensable in many industries and activities where at least some levels of decision-making can be delegated to computer systems. Currently, machine learning models (MLMs) are developed for specific target domains and applications. Because the goals of various machine learning applications can be highly diverse, MLMs may need to be set up, configured, and trained differently depending on the intended users of the trained MLM. However, models belonging to the same general type (e.g., speech recognition models) can be set up very differently in different use cases. For example, a speech recognition model designed for automated customer telephone support may differ from a model developed for hospital applications, such as recording a doctor's analysis of patient diagnostic data or responding to patient requests. Furthermore, even MLMs operating within the same target domain (e.g., the medical field) may require different configurations in different contexts. For example, a model designed to recognize speech in an operating room may need to be trained or configured differently than a model designed for observing or recovering in a ward.

[0016] Currently, configuring an MLM for a user-specific domain application can require significant development work. MLM developers may need to design the model architecture (e.g., in the case of a neural network MLM, designing the topology of layers and node connections), train one or more MLMs on relevant domain-specific training data, and so on. In many cases, one or more MLMs may simply be part of a larger codebase involving various additional computational support stages, such as audio preprocessing, audio artifact removal, filtering, post-processing, spectral Fourier analysis, etc. Multiple MLMs may coexist in the same computational pipeline, and developers may have to merge multiple MLMs into a single computational pipeline, each providing different functionalities. For example, developers of natural language processing applications might need to integrate feature extraction (which uses spectral analysis), acoustic MLM (which processes the extracted features), acoustic post-processing modules (removing artifacts, filler words, and stop words, etc.), language preprocessing (which performs word segmentation, lexicalization, etc.), language MLM (which identifies topics, speaker intent, punctuation, etc.), language post-processing (for performing rule-based corrections / validations on the language MLM output), and so on. In such a pipeline, the output of the acoustic MLM can be fed into a language MLM, which in turn can feed data into an intent recognition model, and so on.

[0017] Currently, to integrate one or more MLMs with various support stages into a single computing pipeline or workflow, developers must create and manage code encompassing the entire workflow. For example, developers might create platform-specific code, such as a speech recognition MLM pipeline tailored to meet the needs of a medical clinic. Such code can make sharing and extending MLM applications beyond the original use case cumbersome and technically challenging. In particular, such pipelines may not be easily customizable to other domains or computing platforms. Specifically, another developer attempting to reconfigure an MLM pipeline for a different domain (e.g., from a stock trading firm to an investment brokerage firm) may not only have to reconfigure the actual MLM but may also need to redesign and comprehensively modify most of the entire code, even if some parts of the code implement one or more standard modules of the pipeline (e.g., digital audio signal preprocessing). Therefore, users (e.g., clients) who have access to the MLM pipeline but lack the expertise of senior developers may not be able to customize the pipeline to their specific needs, such as reconfiguring a natural language programming MLM pipeline for use in different language domains of interest to the user (e.g., sports broadcasting). Consequently, users may have to use a suboptimal MLM pipeline to meet their goals. Alternatively, users may have to acquire additional resources and hire professional developers to configure the pipeline, and may need to retrain part or all of the MLM in the pipeline.

[0018] The various aspects and embodiments of this disclosure address these and other challenges of modern technology by describing methods and systems capable of efficiently managing and configuring MLM pipelines. Implementations allow for training and retraining MLMs for user-specific target platforms, modifying the parameters and architecture of previously trained MLMs, selecting and configuring previously trained MLMs, adapting selected MLMs to serve user-specific needs, integrating selected MLMs into further customizable workflows, deploying customized workflows on user and cloud hardware, inputting actual inference data, reading, storing, and managing inference outputs, and so on.

[0019] System Architecture

[0020] Figure 1 This is a block diagram of an example architecture for a customizable pipeline (CP) 100 that supports the training, configuration, and deployment of one or more machine learning models, according to at least some embodiments. Figure 1As shown, CP 100 can be implemented on computing device 102, but it should be understood that any engine and components of computing device 102 can be implemented (or shared) on any number of computing devices or in the cloud. Computing device 102 can be a desktop computer, laptop computer, smartphone, tablet, server, computing device accessing a remote server, computing device utilizing a virtualized computing environment, game console, wearable computer, smart TV, etc. Users of CP 100 can have local or remote (e.g., via a network) access to computing device 102. Computing device 102 may have (not in...) Figure 1 (As shown herein) Any number of central processing units (CPUs) and graphics processing units (GPUs), including virtual CPUs and / or virtual GPUs, or any other suitable processing device capable of performing the techniques described herein. Computing device 102 may further have (not shown herein) Figure 1 (As shown) Any number of memory devices, network controllers, peripheral devices, etc. Peripheral devices may include cameras (e.g., video cameras) for capturing images (or image sequences), microphones for capturing sound, scanners, sensors, or any other devices for data acquisition.

[0021] In some embodiments, CP 100 may include several engines and components for efficient MLM implementation. Users (customers, end users, developers, data scientists, etc.) can interact with CP 100 via user interface UI 104, which may include command line, graphical UI, web-based interface (e.g., web browser-accessible interface), mobile application-based UI, or any combination thereof. UI 104 may display menus, tables, charts, flowcharts, graphs, and / or text representations of software, data, and workflows. UI 104 may include selectable items that allow users to access various pipeline settings, providing training / retraining and other data, as described in more detail below. User actions input via UI 104 can be transmitted to the pipeline coordinator 110 of CP 100 via pipeline API 106. In some embodiments, before receiving pipeline data from pipeline coordinator 110, the user (or a remote computing device that the user is using to access the pipeline) can download API packages to the remote computing device. The downloaded API package can be used to install Pipeline API 106 on a remote computing device, enabling users to communicate bidirectionally with Pipeline Coordinator 110 during the setup and use of CP 100.

[0022] Pipeline coordinator 110 can provide users with various data via pipeline API 106, which can be used to configure and deploy one or more MLMs and use the deployed MLMs to process (infer) various input user data. For example, pipeline coordinator 110 can provide users with information about available pre-trained MLMs, enabling the retraining of pre-trained MLMs on user-specific data provided by the user or the training of new (previously untrained) MLMs. Pipeline coordinator 110 can then construct CP 100 based on the information received from the user. For example, pipeline coordinator 110 can configure a user-selected MLM and deploy the selected MLM along with various other stages (e.g., preprocessing and postprocessing) for implementing the selected MLM. To perform these and other tasks, pipeline coordinator 110 can coordinate and manage several engines, each implementing a part of the overall pipeline functionality.

[0023] In some embodiments, CP 100 may access one or more previously trained (pre-trained) MLMs and thus provide users with access to at least some of these pre-trained MLMs (e.g., based on user subscriptions). The MLMs can be trained for common tasks within the CP's domain of expertise. For example, a CP specializing in speech processing may access one or more trained MLMs to recognize typical speech patterns, such as customer service requests, public conversations, etc. CP 100 may further include a training engine 120. The training engine 120 can perform retraining (additional training) of the pre-trained MLMs. Retraining can be performed using retraining data tailored to the user's specific domain application. In some embodiments, the retraining data may be provided by the user. For example, a user may provide retraining data to enhance the natural language processing capabilities of one of the pre-trained MLMs to improve speech recognition that may be encountered in investment brokerage or securities trading environments. The data can be provided as audio tracks for video recordings, television programs, etc. (e.g., by the user's financial company's technical experts) in any available (compressed or uncompressed) digital format of digital audio recordings, such as WAV, WavPack, WMA, MP3, MPEG-4.

[0024] The pre-trained MLM 122 can be stored in a trained model repository 124, which is accessible to computing device 102 via network 140. The pre-trained MLM 122 can be trained by training server 162. Network 140 can be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wireless network, a personal area network (PAN), or a combination thereof. In some embodiments, training server 162 can be part of computing device 102. In other embodiments, training server 162 can be directly or communicatively coupled to computing device 102 via network 140. Training server 162 can be (and / or include) a rack server, router computer, personal computer, laptop computer, tablet computer, desktop computer, media center, or any combination thereof. Training server 162 can include training engine 160. Training engine 160 on training server 162 can be the same as (or similar to) training server 162 on computing device 102. In some embodiments, the training engine 120 on computing device 102 may be absent; instead, all training and retraining can be performed via training engine 160 on training server 162. In some embodiments, training engine 160 can perform off-site training of pre-trained MLM 122, while training engine 120 on computing device 102 can perform retraining of pre-trained MLM 122 as well as training of a new (customized) MLM 125.

[0025] During training or retraining, training engine 160 (120) can generate and configure one or more MLMs. MLMs can include regression algorithms, decision trees, support vector machines, K-means clustering models, neural networks, or any other machine learning algorithms. Neural network MLMs can include convolutional, recurrent, fully connected, long short-term memory models, Hopfield, Boltzmann, or any other type of neural network. Generating an MLM can include setting the MLM type (e.g., neural network), architecture, number of layers of neurons, connection types between these layers (e.g., fully connected, convolutional, deconvolutional, etc.), number of nodes in each layer, type of activation function used in each layer / node of the network, type of loss function used in network training, and so on. Generating an MLM can include setting (e.g., randomly) the initial parameters (weights, biases) of each node in the network. The generated MLM can be trained by training engine 160 using training data that may include one or more training inputs 165 and corresponding one or more target outputs 167.

[0026] For example, for training a speech recognition MLM 122, training input 165 may include one or more digital recordings containing utterances of words, phrases, and / or sentences that the MLM is being trained to recognize. Target output 167 may include an indication of the presence of target words and phrases in training input 165. Target output 167 may also include a transcription of the utterances, and so on. In some embodiments, target output 167 may include the identification of the speaker's intent. For example, a customer calling a food delivery service may express a limited number of intents (to order food, check the status of an order, cancel an order, etc.) but may do so in an unlimited number of ways. While the specific words and sentences uttered may not be very meaningful, determining the intent may be important. Therefore, in such an embodiment, target output 167 may include the correct intent category. Similarly, target output 167 for training input 165 (which includes the utterances of a customer calling customer service) may be either a recording of the utterances or an indication of the customer's emotional state (e.g., anger, worry, satisfaction, etc.). Furthermore, training engine 160 can generate mapping data 166 (e.g., metadata) that associates one or more training inputs 165 with one or more correct target outputs 167. During training of MLM 122 (or a custom MLM 125), training engine 160 (or 120) can identify patterns in one or more training inputs 165 based on the desired one or more target outputs 167 and train the respective MLMs to perform the desired task. During the inference phase, in future processing of new speech, the predictive utility of the identified patterns can subsequently be verified using additional training input / target output associations before use. For example, upon receiving a new voice message, the trained MLM 122 can identify that a customer wants to check the status of a previously placed order, identify the customer's name, order number, etc.

[0027] In some embodiments, multiple MLMs can be trained simultaneously or individually. A speech recognition pipeline may involve multiple models, such as an acoustic model for sound processing (e.g., parsing speech into words), a language model for recognizing the parsed words, a model for intent recognition, a model for understanding the question, or any other model. In some embodiments, some models can be trained independently while others can be trained simultaneously. For example, the acoustic model can be trained separately from all other language processing models, the intent recognition model can be trained together with the speech transcription model, and so on.

[0028] In some embodiments, each or some of the MLMs 122 (and / or MLM 125) can be implemented as a deep learning neural network with multiple layers of linear or nonlinear operations. For example, each or some of the speech recognition MLMs can be a convolutional neural network, a recurrent neural network (RNN), a fully connected neural network, and so on. In some embodiments, each or some of the MLMs 122 (and / or MLM 125) can include multiple neurons, each receiving its input from other neurons or external sources and producing an output by applying an activation function to the sum of (trainable) weighted input and bias values. In some embodiments, each or some of the MLMs 122 (and / or 125) can include multiple neurons arranged in layers, including an input layer, one or more hidden layers, and an output layer. Neurons from adjacent layers can be connected by weighted edges. Initially, some initial values ​​(e.g., random values) can be assigned to the edge weights. For each training input 165, the training engine 160 can cause each or some of the MLMs 122 (and / or MLM 125) to generate one or more outputs. The training engine 137 can then compare one or more observed outputs with one or more desired target outputs 167. The resulting error or mismatch, such as the difference between the expected target output 167 and one or more actual outputs of the neural network, can be backpropagated through the respective neural networks, and the weights in the neural networks can be adjusted to make the actual outputs closer to the target outputs 167. This adjustment can be repeated until the output error for a given training input 165 meets a predetermined condition (e.g., below a predetermined value). Subsequently, different training inputs 165 can be selected to generate new outputs and implement a series of new adjustments until the respective neural networks are trained to acceptable accuracy.

[0029] Training engine 120 may include additional (compared to training engine 160) components for retraining a previously trained MLM 122 for a domain-specific application. For example, training engine 120 may include a data augmentation module for augmenting existing training data (e.g., training input 165) using domain-specific data. For instance, existing audio recordings can be augmented with target words and phrases frequently encountered in the target domain. For example, the data augmentation module can augment existing training inputs with phrases such as “short selling,” “capital gains tax,” “hedge funds,” “economic fundamentals,” and “initial public offering.” Target output 167 can also be augmented similarly. For example, the data augmentation module can update target output 167 with various artistic terms with domain-specific meanings (such as “options” and “future”). Training engine 120 may also have a pruning module for reducing the number of nodes, and an evaluation module for determining whether node pruning has reduced the accuracy of the retrained model below a minimum threshold accuracy.

[0030] Figure 2A According to at least some embodiments Figure 1 A block diagram of an example architecture 200 for a training engine (e.g., training engine 120) with a customizable pipeline 100. (See diagram for example.) Figure 2A As shown, the training engine architecture 200 may include several modules (sub-engines) capable of performing the above operations, such as an initial training module 210, an evaluation module 210, a retraining module 230, etc. For example, the initial training module 210 may use initial data 202 to train MLM 122. The initial training module 210 may also use custom (user-specific and / or user-provided) data 204 to train a custom MLM 125. The evaluation module 220 may determine whether the training of MLM 122 (or custom MLM 125) has been successful, or whether additional training should be performed. For example, the evaluation module 220 may use a portion of the initial data 202 (or custom data 204) reserved for testing / evaluation. If the respective MLM does not meet the minimum accuracy or confidence level, as indicated by the reverse arrow, the initial training module 210 may provide additional training on the MLM. Once the MLM successfully passes the evaluation, the MLM may be stored (e.g., stored in...). Figure 1The stored MLM is stored in a training model repository 124 for immediate or future use by the user. The retraining module 230 can retrain the stored MLM using new / adjusted data 232 to produce a retrained MLM 123 or a retrained custom MLM 127. For example, a previously trained MLM can be retrained for applications in different domains. Optionally, a previously trained MLM can be retrained to account for changing or additional conditions, such as changes in terminology used in the domain, hiring a new employee with different language characteristics than other employees, etc. Retraining can also be performed until evaluation criteria (e.g., determined by the evaluation module 220) are met. The retraining criteria may differ from the initial training criteria.

[0031] Please return to the previous page. Figure 1 During MLM retraining, the user can interact with the pipeline coordinator 110 via pipeline API 106 to monitor the retraining process. For example, at the start of retraining (or at any other stage), the user can select a first set of pre-trained MLMs 122 as the model, which will be used as is without retraining. The user can further select a second set of pre-trained MLMs 122 for retraining by training engine 120 (or training engine 160) to produce a retrained MLM 123. Furthermore, the user can instruct training engine 120 (or training engine 160) to produce a custom (user-trained) MLM 125. The user can select the architecture and network parameters for the custom MLM 125 via UI 104; training engine 120 (or training engine 160) can receive the user-specified parameters via pipeline coordinator 110 and perform training of the custom MLM 125 based on the received parameters. The parameters of the pre-trained MLM 122, the retrained MLM 123, and / or the custom MLM 125 can be stored in a memory device accessible to the pipeline coordinator 110. The memory device storing the MLMs can be local (e.g., non-volatile) memory on the computing device 102, or remote (e.g., cloud-based) memory accessible to the computing device 102 via the network 140. When a user logs in to / authenticates a user session on CP 100, a list of some or all of the MLMs (e.g., MLMs 122, 123, and / or 125) available to the user can be provided to the user of CP 100. The user-accessible list may include MLMs that have been retrained (123) or user-trained (125) in the current session, as well as MLMs that have been retrained or user-trained in any previous user session. Therefore, during or after each user session, a new user-trained custom MLM 125 can be stored in the trained model repository 124 for future use.

[0032] CP 100 may also include an export engine 130. The export engine 130 allows the user to select any number of pre-trained MLMs 122, retrained MLMs 123, or custom MLMs 125 for subsequent deployment during the current (or future) user session. The export engine can output the user-selected MLMs in an implementation-independent format. In some embodiments, the output MLM may include identifying and retrieving the topology of the MLM, the number and type of neural network layers in the MLM, and the values ​​of weights determined during training performed by the training engine. In these embodiments, the export engine 130 may generate a representation of the user-selected MLM and display the generated representation on UI 104. The displayed representation may include graphs, tables, numbers, text entries, and other objects characterizing the architecture of the selected MLM, such as the number of layers, nodes, edges, topology, etc., for each selected MLM. The displayed representation may also include parameters of the selected MLM, such as weights, biases, activation functions of individual nodes, etc. The metadata loaded by the export engine 130 can also indicate which additional components and modules the deployment user-selected MLM might require. In speed recognition, these additional components may include a spectral analyzer for speech sounds, a speech feature extractor for acoustic models, an acoustic post-processing component to remove speech artifacts, filler words, and stop words, a language preprocessing component to perform word segmentation, lexicalization, etc., a language post-processing component to perform correction / verification of rule-based language MLM output, and other components.

[0033] CP 100 may also include a build engine 150. The build engine 150 allows users to configure exported pre-trained MLM 122, retrained MLM 123, or custom MLM 125 before deployment to a user platform. The export engine 130 provides a representation of the exported MLM that informs the user about the MLM's architecture and properties. These representations can indicate to the user which aspects of the MLM are static (parameters) and which aspects (settings) are customizable. For example, the type of neural network (e.g., convolutional vs. fully connected), the number of neuron layers in the neural network, the topology of the edges connecting nodes in the network, the types of activation functions used in various nodes, etc., can be fixed parameters. If a user wants to change some fixed parameters, the user may have to retrain their respective neural network model for the new architecture using the training engine 120. On the other hand, the user can change the MLM settings without retraining the model. These configurable settings can include chunk sizes, such as the size of the audio buffer to be processed in the streaming speech recognition application, the alphabet used to map the output (e.g., Latin, Cyrillic, etc.), the language (e.g., English, German, Russian, etc.), the window size used for FFT processing of the input speech data, window overlap (e.g., 25%, 50%, 75%, etc.), Hamming window parameters, speech end detection parameters, audio buffer size, delay settings, etc. The build engine 150 can also allow the user to select from available domains (e.g., the financial industry, the medical field, etc.) that have already been trained on the selected MLM (during initial training, retraining, or training on user-specific data). The build engine 150 (or pipeline coordinator 110) enables the settings and parameters of the exported MLM to be displayed together on the UI 104. The display can be annotated with instructions indicating which modifications (settings) should be processed by the build engine 150 and which modifications (parameters or settings) should be processed by the training engine 120. For example, a call to modify the language model can be handled by the build engine 150 (without calling the training engine 120), while a call to modify the acoustic model (e.g., audio data processing during the speech recognition decoder stage) can be handled by the training engine 120. In some embodiments, a call to modify the language model can include, but is not limited to, adjusting the weights of one or more layers in the model. As a further example, a modification to the setup between models, such as from an acoustic model of one language trained using adult speech data to an acoustic model of the same language trained using speech data from children, can trigger a call to the training engine 120 and retraining of the acoustic model with the new data.

[0034] Some settings may relate to a single exported MLM; for example, alphabet (speech spelling or standard English spelling) settings may affect the language model but not the acoustic model. Some settings may relate to multiple MLMs; for example, configuring a language model for Chinese speech recognition may also invoke changes in acoustic model settings (e.g., automatic or default) to adjust for different rhythms and tones of speech. Some settings may affect the interaction of one or more MLMs with various preprocessing and postprocessing components of the pipeline. For example, changing the language model settings to use the model for speech recognition in different language domains may also require modifying preprocessing block settings, which could remove stop words. Specifically, a language model performing speech recognition on formal presentations at a professional conference might use less aggressive stop / filler word removal than one used for transcription in an informal brainstorming business meeting.

[0035] Figure 2B According to at least some embodiments Figure 1 A block diagram of an example architecture 200 for the build and deployment phases of the customizable pipeline 100. While the build engine 150 allows users to configure the exported MLM prior to deployment, the deployment engine 170 executes the actual implementation of CP 100 on a user-accessible platform. Figure 2B As shown, the MLM exported by export engine 130 may have various model artifacts 252 (e.g., modules, dependencies, metadata, etc.) and one or more configuration files 254 that configure the actual execution of the exported MLM. Users can input modified configuration settings 256 (e.g., via...). Figure 1The configuration settings (received by UI 104) can then be processed by build engine 150. Modified configuration settings can be written back to one or more configuration files 254, which contain various fields, such as default fields initially provided by training engine 120, overridden by user-specified settings. Build engine 150 can process one or more configuration files 254 and model artifacts 252 (e.g., source code, libraries, and other dependencies) for the exported MLM, and can produce executable artifacts, configuration files, and various other dependencies for the exported MLM, such as libraries, executable code for preprocessing components and postprocessing components of the pipeline, etc. The output of build engine 150 can be an intermediate representation (IR) of the pipeline. In some embodiments, the IR of the MLM pipeline can be packaged as a Docker image 260, or an image in any similar platform for containerized application execution. In some implementations, images in different formats (other than Docker) can be used, for example, any proprietary format can be used. In some implementations, an appropriate archive in a format that bundles various executable components, libraries, data, and metadata together can be used. The content of the IR can be stored in the memory device of the user's computing device (e.g., computing device 102) (e.g., as an archive), or in a memory device accessible to computing device 102 (e.g., in the cloud). The build engine 150 can be a tool or application implemented in Python, C++, Java, or any other programming language.

[0036] Configurable pipelines (e.g., Figure 1The deployment engine 170 of CP 100 can implement the pipeline on user-accessible hardware resources (target platform). The user can access a (e.g., local) computing device 102 with several CPUs, GPUs, and memory devices. Optionally or additionally, the user can access one or more cloud computing servers providing virtualization services. The deployment engine 170 can allow the user to input a description or identification of the user-accessible target platform resource 262, which may include identifiers of available computing, storage, network, and other resources. In some embodiments, the deployment engine 170 can use any available metric collection device or driver to collect information about the local resources of the computing device 102. Optionally or additionally, the deployment engine 170 or pipeline coordinator 110 can collect information about available virtual (cloud) processing resources (e.g., using authentication services of a remote access server or remote virtualization server). The collected information may include (but is not limited to) CPU speed, number of CPU cores (physical or virtual), number and type of GPUs (physical or virtual), amount of available system memory and / or GPU memory, number of remote processing nodes available for pipeline deployment, type and version of the operating system installed on computing device 102 (or type / version of guest operating system instantiated on the virtualization environment), and so on. After the user-selected configuration settings have been implemented by build engine 150, deployment engine 170 enables the execution of the MLM pipeline (e.g., CP100) on user-accessible computing resources without reconfiguring the features of the MLM pipeline.

[0037] In some embodiments, deployment engine 170 can access the IR of the MLM pipeline stored by build engine 150 (locally or in the cloud) and generate an inference ensemble of executable code (e.g., implemented as object code or bytecode), configuration files, libraries, dependencies, and other resources for use by inference engine 180. In embodiments where the MLM pipeline IR is a Docker (or similar) image 260, deployment engine 170 can instantiate a pipeline Docker container 270 based on Docker image 260, for example, using a containerized service accessible to the target platform. In some embodiments, the inference ensemble can be a Triton ensemble for a Triton inference server, which facilitates MLM deployment and allows users to run MLM using various available frameworks (e.g., TensorFlow, TersorRT, PyTorch, ONNX runtime, etc.) or user-provided custom frameworks. Deployment engine 170 can execute commands specified in the IR to run the executable code, libraries, and other dependencies generated by build engine 150. Furthermore, the deployment engine 170 can perform mappings of the MLM pipeline configurations generated by the build engine 150 to handle the computational resources of a specific target platform, including available GPUs and / or CPUs. The configuration files generated by the deployment engine 170 can be stored using a platform-neutral protocol buffer, which can be ASCII serialized. The use of a protocol buffer minimizes typesetting and serialization-related errors that may occur when users (developers) add support for the MLM architecture.

[0038] Refer again Figure 1 After deployment engine 170 converts the IR into a set of inferences ready to be executed by inference engine 180 (e.g., into pipeline Docker containers), the MLM pipeline can be ready to process input user data 182. User data 182 can be any data that can be configured to be used by the MLM pipeline. For example, for speech processing, user data 182 can include voice recordings, such as digital recordings of conversations, presentations, narrations, or any other audio recordings to be transcribed. In some embodiments, user data can include questions (or a series of questions) to be answered. In some embodiments, user data 182 can be images (or sequences of images) with objects to be identified, motion patterns to be detected, etc. Any other user data 182 can be input to inference engine 180, and the type of user data depends on the user-specific domain. In embodiments involving natural language processing, the user-specific domain can include customer service support, medical inquiries, educational settings, courtroom settings, emergency personnel conversations, or any other type of environment.

[0039] The inference engine 180 can process user data 182 and generate inference output 184. The inference output 184 can have any suitable type and format. For example, the inference output 184 can be a transcription of speech or dialogue, recognition of the intent or emotion of one or more speakers, an answer to a question (e.g., in text or numerical form), and so on. The format of the inference output 184 can be text, numbers, spreadsheets, audio files, video files, or any combination thereof.

[0040] The various engines of CP 100 do not need to be applied to linear progressions. In some embodiments, the various engines of CP 100 can be applied multiple times. For example, user data 182 can be used as test data, and the obtained inference output 184 can be used as feedback on the current state of the pipeline. This feedback can inform the user about how to modify the pipeline to improve its performance. In some embodiments, such modifications can be performed iteratively. For example, after receiving feedback, the user can use training engine 120 to initiate retraining of some MLMs included in the pipeline. The user can also replace some MLMs with other (e.g., pre-trained or user-trained) MLMs and export the newly added MLMs using export engine 130. The user can use build engine 150 to change the configuration of some old, retrained, or newly trained MLMs. Build engine 150 can be configured to update only (e.g., by generating updated IR) the model constructs and configuration files of these MLMs and components of the modified pipeline (e.g., by updating configuration settings), without changing the models and components that remain unchanged, for faster installation. In some embodiments, this faster installation can be achieved by using a PIP (Package Installer for Python) wheel to build package formats (.whl). The deployment engine 170 can then deploy updated pipelines on the target platform using the updated IR. In some embodiments, the user can retain the MLM and its respective configuration settings, but modify (by adding, removing, or otherwise altering) the available resources on the target platform.

[0041] Figure 3 This is an example computing device 300 according to at least some embodiments, which is capable of implementing a customizable pipeline to support the training, configuration, and deployment of one or more machine learning models. In some embodiments, the computing device 300 may include some or all of the engines of the customizable pipeline, for example, Figure 1 The CP 100 in the model includes a training engine 120, an export engine 130, a build engine 150, a deployment engine 170, and an inference engine 180. Although Figure 3All engines are described as part of the same computing device, but in some implementations, any engine shown may actually be implemented on different computing devices, including virtual computing devices, cloud-based processing devices, etc. For example, computing device 300 may include inference engine 180, but not other engines with customizable pipelines. Inference engine 180 (and / or any other engines in the pipeline) may be executed by one or more GPUs 310 to perform speech recognition, object recognition, or any other inference involving machine learning. In some embodiments, GPU 310 includes multiple cores 311, each capable of executing multiple threads 312. Each core may run multiple threads 312 concurrently (e.g., in parallel). In some embodiments, threads 312 may access registers 313. Registers 313 may be thread-specific registers, which may access registers limited to their respective threads. Furthermore, shared registers 314 may be accessible by all threads of the core. In some embodiments, each core 311 may include a scheduler 315 for distributing computational tasks and processes among different threads 312 of the core 311. The dispatch unit 316 can use the correct private registers 313 and shared registers 314 to implement the scheduled task on the appropriate thread. The computing device 300 may include one or more input / output components 334 to facilitate the exchange of information with one or more users or developers.

[0042] In some embodiments, GPU 310 may have a (high-speed) cache 318, access to which may be shared by multiple cores 311. Furthermore, computing device 300 may include GPU memory 319, where GPU 310 may store intermediate and / or final results (outputs) of various computations performed by GPU 310. After completing a specific task, GPU 310 (or CPU 330) may move the output to (main) memory 304. In some embodiments, CPU 330 may execute processes involving serial computation tasks (assigned by one engine of the pipeline), while GPU 310 may execute tasks easily parallelized (e.g., multiplying the input of a neural node by weights and adding a bias). In some embodiments, the respective engines of the pipeline (e.g., build engine 150, inference engine 180, etc.) may determine which processes managed by their respective engines will be executed on GPU 310 and which processes will be executed on CPU 330. In some embodiments, CPU 330 may determine which processes will be executed on GPU 310 and which processes will be executed on CPU 330.

[0043] Figure 4This is a block diagram of an example customizable pipeline 400 according to at least some embodiments, which uses one or more machine learning models for natural language processing of speech. Part or all of the MLM in pipeline 400 may be a trained neural network model, such as a pre-trained and / or custom-trained deep learning neural network. Speech input 402 entering pipeline 400 may be an analog signal, such as a digital file generated by a microphone and converted into any audio format readable by the processing device. Input speech 402 may undergo audio preprocessing 410, which may include spectral analysis and other processing. For example, input speech 402 may undergo filtering, upsampling or downsampling, pre-emphasis, windowing (e.g., using a 20ms window that advances every 10ms), application of the Mel-frequency cepstral coefficients (MFCC) algorithm, and / or other processing. A multidimensional vector representing the extracted features may be input into a first MLM 420, which may be an acoustic MLM (e.g., an acoustic neural network model). Acoustic MLM 420 may be a trained neural network model for outputting recognition of various phonemes (the sounds of vowels), each phoneme being assigned a probability. Acoustic post-processing 430 may include speech decoding (e.g., assigning probabilities to individual words) and may also include removal of sound artifacts, filler words, or stop words, or some other type of acoustic post-processing. Language pre-processing 440 may include word rooting or lemmatization (determining the root form of words), word segmentation (recognition of character and word sequences), etc. The output of language pre-processing 440 may be used as input to a second MLM 450, which may include one or more language MLMs (e.g., language neural network models). The second MLM 450 may be another trained neural network for producing text output 460, which may be a transcription of speech input 402. In some embodiments, the second MLM 450 may be configured to generate punctuation detection (450-1), utterance detection (450-2), and speaker intent detection (450-3), which can be achieved through additional output / neuron layers of the same second MLM 450 or as an additional neural network model. The second MLM 450 may include a language understanding neural network model 450-4 and / or a question-and-answer (QA) neural network model 450-5. The output of the QA model 450-5 may be a representation (e.g., a text representation) of the answer to a question contained in the speech input 402. In some embodiments, the output of the QA model 450-5 may be provided to a speech synthesizer, which may be a trained third MLM 470 for outputting artificial speech as part of the speech output 472.

[0044] As mentioned above Figure 1As depicted in Figure 2, the first MLM 420, the second MLM 450, and / or the third MLM 470 can be pre-trained or custom-trained by training engine 120, as schematically shown with dashed arrows. In some embodiments, some or all of MLMs 420, 430, and 470 can be retrained using domain-specific user data. Furthermore, as... Figure 4 As further described, some or all of the MLM 420, 430 and 470 can be configured using the Build Engine 150 based on user-provided configuration settings (without retraining).

[0045] Figure 5 and Figure 6 The flowcharts are, respectively, of example methods 500 and 600 related to providing a customizable machine learning pipeline according to at least some embodiments. Methods 500 and 600 can be executed to deploy an MLM for voice recognition, speech recognition, speech synthesis, object detection, motion detection, hazard detection, robotics applications, prediction, and many other contexts and applications where machine learning can be used. In at least one embodiment, methods 500 and 600 can be executed by a processing unit of computing device 102, computing device 300, or a combination of other computing devices. Methods 500 and 600 can be executed by one or more processing units (e.g., CPU and / or GPU), which may include (or communicate with) one or more memory devices. In at least one embodiment, methods 500 and 600 can be executed by multiple processing threads (e.g., CPU threads and / or GPU threads), each thread executing one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, the processing threads implementing method 500 (and similarly, method 600) can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Optionally, the processing threads implementing method 500 (and similarly, method 600) can execute asynchronously with each other. Various operations of methods 500 and 600 can be synchronized with... Figure 5 and Figure 6 The order shown is compared to a different order of execution. Some operations of these methods can be performed concurrently with other operations. In at least one embodiment, they may not always be executed. Figure 5 and Figure 6 One or more operations are shown.

[0046] Figure 5 This is a flowchart of an example method 500 according to at least some embodiments, providing support for training, configuring, and deploying one or more machine learning models. The customizable pipeline may be CP 100, which may include features as described above. Figure 1 and Figure 2A-B describes various engines, modules, and components. In block 510, the processing unit executing method 500 can access multiple trained MLMs, which can be pre-trained MLMs 122 (e.g., provided by a pipeline service provider) or custom MLMs 125 (trained by a pipeline user). The selected MLMs can be pre-trained using a first set of training data, which can be provided previously by the pipeline service provider or by the user. Maintaining trained MLMs can include storing files and data sufficient for deploying and executing the MLMs, or storing references to such files and data, such as links to downloadable files and data stored elsewhere (e.g., on cloud storage). Some or each of the maintained MLMs can be associated with initial configuration settings for their respective MLMs, which can be stored in configuration files, databases, etc. The processing unit executing method 500 can also access these configuration settings. In some embodiments, at least some of the selected MLMs can be (or include) neural network models with multiple neuron layers. Some neural network models can be deep learning neural network models.

[0047] At box 520, the processing unit performing method 500 can provide a user interface (UI) for receiving user input indicating the selection of one or more of a plurality of trained MLMs. For example, a user may be attempting to set up a machine learning pipeline and can select from available MLMs those that can solve a specific problem or set of problems for the user. For example, a user attempting to set up a pipeline for speech recognition can select an acoustic model for decoding input speech and a language model for recognizing the decoded speech. To assist in selecting an MLM for the pipeline, before receiving a user selection of one or more MLMs, the processing unit may cause one or more lists of a plurality of trained MLMs available to the user to be displayed on the UI. In some embodiments, the list may be available in the form of enumeration items, clickable buttons, icons, menus, or any other prompts. For some or each of the listed MLMs, the list may include a representation of the architecture of the respective MLM, which may be in the form of a graph, table, description of layers, description of the topology of the MLM, etc. In some embodiments, the list may also include parameters of the respective MLM, such as the number of nodes, edges, activation functions, or any other specification of the MLM's properties.

[0048] In response to viewing the list of MLMs, a user can decide whether to modify some of the selected MLMs to better correspond to the details of their project. In some cases, modifications to the MLMs may be significant enough to necessitate retraining the selected MLMs, for example, using second training data, which may be user-selected data, to retrain the selected MLMs for a specific domain to which the MLM pipeline is to be applied. Specifically, the processing unit performing method 500 may receive a second set of training data for domain-specific training of one or more selected MLMs. For example, the processing unit may receive an instruction on which MLMs to be retrained based on user input via the UI. For example, from the selected acoustic MLMs and language MLMs for the speech recognition pipeline, the user may instruct the language MLM to be retrained for Japanese (which has already been trained for English). The user may also identify the location of the retraining data (e.g., the second data) (e.g., a cloud storage address). The retraining data may include training inputs (e.g., audio files), target outputs (e.g., transcriptions of speech from audio files), and mapping data (indicators of the correspondence between training inputs and target outputs). In response to receiving an instruction to retrain an MLM and retraining data, the processing unit may cause the execution pipeline's training engine (e.g., training engine 120) to perform domain-specific training on one or more selected MLMs, as combined above. Figure 1 A more detailed description is provided. Retraining can be performed on both the pre-trained MLM 122 and the previously trained custom MLM125.

[0049] In some cases, the necessary modifications to the MLM may not be significant enough to require retraining. In some embodiments, the modified configuration settings may include language settings for the language neural network model. Using the previous example, to implement a change from English to Japanese, the user could decide to modify the settings of the acoustic MLM, for example, by changing the size of the sliding window. In response to receiving initial configuration settings from user input specifying how to modify one or more MLMs, at block 530, the processing unit performing method 500 may determine the modified configuration settings for one or more selected MLMs based on the user input. In some embodiments, the modified configuration settings for one or more selected MLMs may include audio buffer size, speech end settings, or delay settings for the acoustic MLM. Other natural language processing MLMs that can be configured similarly include speech synthesis MLMs, language understanding MLMs, question-answering MLMs, or any other MLM.

[0050] In box 540, the processing unit executing method 500 may cause (e.g., to be initiated by issuing a command) to execute the build engine of the MLM pipeline to modify one or more selected MLMs according to the modified configuration settings, as described above. Figure 1A more detailed description follows. In box 550, the method can continue: the processing unit enables the deployment engine of the execution pipeline to deploy one or more modified MLMs, as described above. Figure 1 A more detailed description follows. In box 560, the processing unit executing method 500 can enable the display of representations of one or more deployed MLMs on the UI, as described above. Figure 1 A more detailed description is provided. The displayed representation demonstrates to the user that the deployed MLM is ready to process user data. The configurable MLM pipeline can be used for training, deploying, and inference of any type and any number of machine learning models.

[0051] Figure 6 This is a flowchart of an example method 600 using a deployed customizable machine learning pipeline model according to at least some embodiments. In some embodiments, method 600 may be used in conjunction with method 500. Method 600 may be executed after providing a list of pre-trained MLMs 122 and custom MLMs 125 to be presented to a user, and optionally after retraining of an MLM selected by the user and retrained by a training engine. At box 610, the processing unit executing method 600 may receive a user selection of one or more MLMs (to place them in a configurable pipeline) and may cause the pipeline's export engine to be executed to initialize one or more selected MLMs. At box 620, method 600 may continue to make (e.g., using the export engine) one or more selected MLMs available for processing user input data. Furthermore, method 600 may include causing the execution of a build engine (box 540) and a deployment engine (box 560), as per [the relevant documentation]. Figure 5 Method 500 is described in the text.

[0052] At box 630, the processing unit performing method 600 may receive user input data, and at box 640, it may be possible to apply one or more deployed MLMs (Multi-Level Models) to the user input data (e.g., using an inference engine) to generate output data. Applying the output of one or more MLMs may enable at least one of displaying a representation of the output data or a reference to a stored representation of the output data on a UI at box 650. For example, if the user input data includes speech, the representation of the output data may be text transcribed from the input speech, recognition of the speaker's intent, punctuation of the input speech, an audio response to a question asked (e.g., using synthesized speech), and so on. In some embodiments, the representation of the output data may be text or sound directly presented to the user, for example, on a UI. In some embodiments, the representation of the output data may be an indication on which the output data is stored (on a local machine or in the cloud).

[0053] Reasoning and training logic

[0054] Figure 7A Inference and / or training logic 715 is shown for performing inference and / or training operations associated with one or more embodiments.

[0055] In at least one embodiment, inference and / or training logic 715 may include, but is not limited to, code and / or data storage 701 for storing forward and / or output weights and / or input / output data, and / or other parameters configuring neurons or layers of a neural network trained for and / or used for inference in one or more embodiments. In at least one embodiment, training logic 715 may include or be coupled to code and / or data storage 701 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic, including integer and / or floating-point units (collectively referred to as arithmetic logic units (ALUs) or simple circuits). In at least one embodiment, code (such as graph code) loads weight or other parameter information into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, code and / or data storage 701 stores weight parameters and / or input / output data of each layer of a neural network trained or used in one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inference using one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 701 may be included within other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0056] In at least one embodiment, any portion of the code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 701 may be a cache memory, dynamic random-addressable memory (“DRAM”), static random-addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether the code and / or data storage 701 is internal or external to the processor, for example, or composed of DRAM, SRAM, flash memory, or some other storage type, may depend on the available on-chip or off-chip storage space, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in the inference and / or training of the neural network, or some combination of these factors.

[0057] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 705 for storing backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in one or more embodiments. In at least one embodiment, during training and / or inference using one or more embodiments, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of a neural network trained or used in one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 705 for storing graph code or other software to control timing and / or sequence, wherein weight and / or other parameter information is loaded to configure logic including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).

[0058] In at least one embodiment, code (such as graph code) causes the architecture of the neural network corresponding to that code to load weights or other parameter information into the processor ALU. In at least one embodiment, any portion of the code and / or data storage 705 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice between the code and / or data storage 705 being internal or external to the processor, for example, whether it consists of DRAM, SRAM, flash memory, or some other type of storage, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the data batch size used in the inference and / or training of the neural network, or some combination of these factors.

[0059] In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be separate storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be combined storage structures. In at least one embodiment, code and / or data storage 701 and code and / or data storage 705 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data storage 701 and code and / or data storage 705 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0060] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 710 (including integer and / or floating-point units) for performing logical and / or mathematical operations at least in part based on or instructed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from layers or neurons within a neural network) stored in activation storage 720, which are functions of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, activation is activated in response to execution instructions or other code, linear algebraic and / or matrix-based mathematical generation performed by ALU 710, and the activation is stored in activation storage 720. The weight values ​​stored in code and / or data storage 705 and / or data storage 701 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters. Any or all of these can be stored in code and / or data storage 705 or code and / or data storage 701 or another on-chip or off-chip memory.

[0061] In at least one embodiment, one or more processors or other hardware logic devices or circuits include one or more ALUs 710, while in another embodiment, one or more ALUs 710 may be located outside the processor or other hardware logic device or the circuitry that uses them (e.g., a coprocessor). In at least one embodiment, one or more ALUs 710 may be included within an execution unit of a processor, or otherwise included in a group of ALUs accessible by the execution unit of the processor, which may be within the same processor or distributed among different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed-function unit, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may share a processor or other hardware logic device or circuitry, while in another embodiment, they may be located in different processors or other hardware logic devices or circuitry, or in some combination of the same and different processors or other hardware logic devices or circuitry. In at least one embodiment, any portion of activation storage 720 may be included together with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Furthermore, inference and / or training code may be stored together with other code accessible to the processor or other hardware logic or circuitry, and may be retrieved and / or processed using the processor’s fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0062] In at least one embodiment, the active memory 720 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the active memory 720 may be wholly or partially located inside or outside one or more processors or other logic circuits. In at least one embodiment, the choice of whether the active memory 720 is internal to or external to the processor may depend on the available on-chip or off-chip storage, the latency requirements for training and / or inference functions, the batch size of data used in inference and / or training the neural network, or some combination of these factors. For example, it may include DRAM, SRAM, flash memory, or some other memory type.

[0063] In at least one embodiment, Figure 7A The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7AThe inference and / or training logic 715 shown can be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, data processing unit (“DPU”) hardware or other hardware (such as field programmable gate array (“FPGA”)).

[0064] Figure 7B An inference and / or training logic 715 according to at least one embodiment is illustrated. In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, hardware logic, wherein computational resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown can be used in conjunction with an application-specific integrated circuit (ASIC), such as those from Google. Processing unit, from Graphcore TM Inference processing units (IPUs) or from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, data processing unit (“DPU”) hardware, or other hardware (e.g., field-programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 7B In at least one embodiment shown, each of code and / or data storage 701 and code and / or data storage 705 is associated with dedicated computing resources (e.g., computing hardware 702 and computing hardware 706), respectively. In at least one embodiment, each of computing hardware 702 and computing hardware 706 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) only on the information stored in code and / or data storage 701 and code and / or data storage 705, respectively, and the results of the function execution are stored in activation storage 720.

[0065] In at least one embodiment, each of the code and / or data storage 701 and 705 and the corresponding computing hardware 702 and 706 corresponds to a different layer of the neural network, such that activations obtained from one storage / computation pair 701 / 702 of the code and / or data storage 701 and computing hardware 702 provide input as input to the next storage / computation pair 705 / 706 of the code and / or data storage 705 and computing hardware 706, in order to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / computation pair 701 / 702 and 705 / 706 may correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic 715 after or in parallel with the storage / computation pairs 701 / 702 and 705 / 706.

[0066] Neural network training and deployment

[0067] Figure 8 Training and deployment of a deep neural network according to at least one embodiment are illustrated. In at least one embodiment, an untrained neural network 806 is trained using a training dataset 802. In at least one embodiment, the training framework 804 is the PyTorch framework, while in other embodiments, the training framework 804 is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training frameworks. In at least one embodiment, the training framework 804 trains the untrained neural network 806 and enables it to be trained using the processing resources described herein to generate a trained neural network 808. In at least one embodiment, the weights may be randomly selected or pre-trained using a deep belief network. In at least one embodiment, training may be performed in a supervised, partially supervised, or unsupervised manner.

[0068] In at least one embodiment, supervised learning is used to train an untrained neural network 806, wherein training dataset 802 includes inputs paired with desired outputs for input, or wherein training dataset 802 includes inputs with known outputs and neural network 806 is manually graded output. In at least one embodiment, the untrained neural network 806 is trained in a supervised manner, and inputs from training dataset 802 are processed, and the resulting output is compared with a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through the untrained neural network 806. In at least one embodiment, training framework 804 adjusts the weights controlling the untrained neural network 806. In at least one embodiment, training framework 804 includes tools for monitoring the degree to which the untrained neural network 806 converges to a model (e.g., a trained neural network 808) adapted to generate the correct answer (e.g., result 814) based on input data (e.g., a new dataset 812). In at least one embodiment, training framework 804 repeatedly trains the untrained neural network 806 while adjusting the weights to improve the output of the untrained neural network 806 using a loss function and tuning algorithms (e.g., stochastic gradient descent). In at least one embodiment, the training framework 804 trains the untrained neural network 806 until the untrained neural network 806 reaches the desired accuracy. In at least one embodiment, the trained neural network 808 can then be deployed to perform any number of machine learning operations.

[0069] In at least one embodiment, unsupervised learning is used to train an untrained neural network 806, wherein the untrained neural network 806 attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised learning training dataset 802 will include input data without any associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 806 can learn groupings within the training dataset 802 and can determine how each input relates to the untrained dataset 802. In at least one embodiment, unsupervised training can be used to generate a self-organizing graph in the trained neural network 808, which is capable of performing operations useful for reducing the dimensionality of the new dataset 812. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows the identification of data points in the new dataset 812 that deviate from the normal patterns of the new dataset 812.

[0070] In at least one embodiment, semi-supervised learning can be used, a technique in which a mixture of labeled and unlabeled data is included in the training dataset 802. In at least one embodiment, the training framework 804 can be used to perform incremental learning, for example, through transfer learning techniques. In at least one embodiment, incremental learning enables the trained neural network 808 to adapt to a new dataset 812 without forgetting the knowledge injected into the trained neural network 808 during initial training.

[0071] Reference Figure 9 , Figure 9 This is an example data flow diagram of process 900 for generating and deploying a processing and inference pipeline according to at least one embodiment. In at least one embodiment, process 900 may be deployed to perform game name recognition analysis and inference on user feedback data at one or more facilities 902, such as a data center.

[0072] In at least one embodiment, process 900 may be executed within training system 904 and / or deployment system 906. In at least one embodiment, training system 904 may be used to train, deploy, and implement machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use with deployment system 906. In at least one embodiment, deployment system 906 may be configured to offload processing and computing resources in a distributed computing environment to reduce infrastructure requirements of facility 902. In at least one embodiment, deployment system 906 may provide a pipeline platform for selecting, customizing, and implementing virtual instruments for use with computing devices at facility 902. In at least one embodiment, virtual instruments may include software-defined applications for performing one or more processing operations on feedback data. In at least one embodiment, one or more applications in the pipeline may use or invoke services of deployment system 906 (e.g., inference, visualization, computation, AI, etc.) during application execution.

[0073] In at least one embodiment, some applications used in the advanced processing and inference pipeline may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at facility 902 using feedback data 908 (e.g., feedback data) stored at facility 902 or feedback data 908 from another or more facilities, or a combination thereof. In at least one embodiment, training system 904 may be used to provide applications, services, and / or other resources to generate a deployable machine learning model for the operation of deployment system 906.

[0074] In at least one embodiment, the model registry 924 may be supported by an object storage system that supports version control and object metadata. In at least one embodiment, it may be available from within a cloud platform via, for example, cloud storage (e.g., Figure 10 The system uses a Cloud 1026-compatible Application Programming Interface (API) to access object storage. In at least one embodiment, machine learning models within the model registry 924 can be uploaded, listed, modified, or deleted by the developer or partner of the system interacting with the API. In at least one embodiment, the API can provide access to methods that allow users with appropriate credentials to associate models with applications, enabling the models to be executed as part of the containerized instantiation of the application.

[0075] In at least one embodiment, training pipeline 1004 ( Figure 10 This can include situations where facility 902 is training its own machine learning model or has an existing machine learning model that needs optimization or updating. In at least one embodiment, feedback data 908 can be received from various channels (such as forums, web forms, etc.). In at least one embodiment, once feedback data 908 is received, AI-assisted annotation 910 can be used to help generate annotations corresponding to the feedback data 908 for use as ground truth data for the machine learning model. In at least one embodiment, AI-assisted annotation 910 can include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that can be trained to generate annotations corresponding to certain types of feedback data 908 (e.g., from certain devices), and / or certain types of anomalies in the feedback data 908. In at least one embodiment, AI-assisted annotation 910 can then be used directly, or it can be adjusted or fine-tuned using annotation tools to generate ground truth data. In at least one embodiment, in some examples, labeled data 912 can be used as ground truth data for training the machine learning model. In at least one embodiment, AI-assisted annotation 910, labeled data 912, or a combination thereof can be used as ground truth data for training the machine learning model. In at least one embodiment, the trained machine learning model may be referred to as output model 916 and may be used by deployment system 906, as described herein.

[0076] In at least one embodiment, training pipeline 1004 ( Figure 10This can include situations where facility 902 requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 906, but facility 902 may not currently have such a machine learning model (or may not have an efficient or effective model optimized for this purpose). In at least one embodiment, an existing machine learning model can be selected from model registry 924. In at least one embodiment, model registry 924 can include machine learning models trained to perform various inference tasks on imaging data. In at least one embodiment, the machine learning model in model registry 924 can be trained on imaging data from a different facility (e.g., a remote facility) instead of facility 902. In at least one embodiment, the machine learning model may have already been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a specific location, training can be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of the imaging data off-site (e.g., in compliance with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model has been trained or partially trained at one location, a machine learning model can be added to a model registry 924. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be used in the model registry 924. In at least one embodiment, a machine learning model (referred to as output model 916) can then be selected from the model registry 924 and used in a deployment system 906 to perform one or more processing tasks for one or more applications of the deployment system.

[0077] In at least one embodiment, training pipeline 1004 ( Figure 10This can be used in scenarios including facility 902, which requires a machine learning model to perform one or more processing tasks for deploying one or more applications in system 906, but facility 902 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for this purpose). In at least one embodiment, the machine learning model selected from model registry 924 may not be fine-tuned or optimized for the feedback data 908 generated at facility 902 due to population differences, genetic variation, robustness, anomalous diversity of the training data, and / or other problems with the training data used to train the machine learning model. In at least one embodiment, AI-assisted annotation 910 can be used to help generate annotations corresponding to the feedback data 908 for use as ground-based data for retraining or updating the machine learning model. In at least one embodiment, labeled data 912 can be used as ground-based data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 914. In at least one embodiment, model training 914 (e.g., AI-assisted annotation 910, labeled data 912, or a combination thereof) can be used as ground-based data for retraining or updating the machine learning model.

[0078] In at least one embodiment, the deployment system 906 may include software 918, service 920, hardware 922, and / or other components, features, and functions. In at least one embodiment, the deployment system 906 may include a software "stack" such that software 918 can be built on top of service 920 and can be used to perform some or all of the processing tasks, and service 920 and software 918 can be built on top of hardware 922 and can be used to perform processing, storage, and / or other computing tasks of the deployment system 906.

[0079] In at least one embodiment, software 918 may include any number of different containers, each of which may perform an instantiation of an application. In at least one embodiment, each application may perform one or more processing tasks (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in an advanced processing and inference pipeline. In at least one embodiment, for each type of computing device, there may be any number of containers that may perform data processing tasks on feedback data 908 (or other data types, such as those described herein). In at least one embodiment, in addition to the containers that receive and configure imaging data for use by each container and / or for use by facility 902 after processing through the pipeline, the advanced processing and inference pipeline may also be defined based on the selection of different containers desired or required for processing feedback data 908 (e.g., to convert output back to a usable data type for storage and display in facility 902). In at least one embodiment, a combination of containers within software 918 (e.g., which constitute a pipeline) may be referred to as a virtual instrument (as described in more detail herein), and the virtual instrument may utilize service 920 and hardware 922 to perform some or all of the processing tasks of the application instantiated in the container.

[0080] In at least one embodiment, data may be preprocessed as part of a data processing pipeline to prepare it for processing by one or more applications. In at least one embodiment, post-processing may be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare output data for the next application and / or to prepare output data for user transmission and / or use (e.g., as a response to an inference request). In at least one embodiment, the inference task may be performed by one or more machine learning models, such as trained or deployed neural networks, which may include the output model 916 of the training system 904.

[0081] In at least one embodiment, the tasks of the data processing pipeline can be encapsulated in one or more containers, each container representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications can be published to a private (e.g., limited access) area of ​​a container registry (described in more detail herein), and trained or deployed models can be stored in a model registry 924 and associated with one or more applications. In at least one embodiment, an image of the application (e.g., a container image) can be used in the container registry, and once a user selects an image from the container registry for deployment in the pipeline, that image can be used to generate containers for instantiation of the application for use by the user's system.

[0082] In at least one embodiment, the developer can develop, publish, and store an application (e.g., as a container) for performing processing and / or inference on the provided data. In at least one embodiment, a software development kit (SDK) associated with the system can be used to perform development, publication, and / or storage (e.g., to ensure that the developed application and / or container conforms to or is compatible with the system). In at least one embodiment, the developed application can be tested locally using the SDK (e.g., at a first facility, on data from a first facility), the SDK serving as a system (e.g.,...). Figure 10 System 1000 may support at least some services 920. In at least one embodiment, once verified by system 1000 (e.g., for accuracy, etc.), the application becomes available in the container registry for users (e.g., hospitals, clinics, laboratories, healthcare providers, etc.) to select and / or implement one or more processing tasks on data at the user's facility (e.g., a second facility).

[0083] In at least one embodiment, the developer can then share the application or container over a network for the system (e.g., Figure 10 The system 900 allows for user access and use. In at least one embodiment, a completed and validated application or container may be stored in a container registry, and the associated machine learning model may be stored in a model registry 924. In at least one embodiment, a requesting entity (which provides an inference or image processing request) may browse the container registry and / or model registry 924 for applications, containers, datasets, machine learning models, etc., select the desired combination of elements to include in the data processing pipeline, and submit a processing request. In at least one embodiment, the request may include input data necessary to execute the request, and / or may include a selection of the application and / or machine learning model to be executed when the request is processed. In at least one embodiment, the request may then be passed to one or more components of the deployment system 906 (e.g., the cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 906 may include referencing elements (e.g., applications, containers, models, etc.) selected from the container registry and / or model registry 924. In at least one embodiment, once results are generated through the pipeline, the results may be returned to the user for reference (e.g., for viewing in a suite of viewing applications executed on a local machine, local workstation, or terminal).

[0084] In at least one embodiment, service 920 may be utilized to assist in processing or executing applications or containers in the pipeline. In at least one embodiment, service 920 may include computing services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, service 920 may provide functionality common to one or more applications in software 918, thus abstracting functionality into services that can be invoked or utilized by applications. In at least one embodiment, the functionality provided by service 920 can operate dynamically and more efficiently, while also allowing applications to process data in parallel (e.g., using...). Figure 10 The parallel computing platform 1030 in the system can be scaled well. In at least one embodiment, it is not required that each application providing the same functionality as the shared service 920 must have a corresponding instance of the service 920, but rather that the service 920 can be shared between and among various applications. In at least one embodiment, as a non-limiting example, the service may include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included, which can provide the ability to train and / or retrain machine learning models.

[0085] In at least one embodiment, where service 920 includes an AI service (e.g., an inference service), as part of application execution, one or more machine learning models associated with an application for anomaly detection (e.g., tumors, growth abnormalities, scarring, etc.) can be executed by invoking (e.g., as an API call) the inference service (e.g., an inference server) to execute one or more machine learning models or their processing. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, the application can invoke the inference service to execute the machine learning models for performing one or more processing operations associated with the segmentation task. In at least one embodiment, the software 918 implementing the advanced processing and inference pipeline can be pipelined because each application can invoke the same inference service to execute one or more inference tasks.

[0086] In at least one embodiment, hardware 922 may include a GPU, CPU, DPU, graphics card, AI / deep learning system (e.g., an AI supercomputer, such as NVIDIA's DGX supercomputer system), cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 922 may be used to provide efficient, specially designed support for software 918 and services 920 deployed in system 906. In at least one embodiment, GPU processing may be used to perform local processing (e.g., at facility 902) within the AI / deep learning system, in the cloud system, and / or other processing components of system 906 to improve the efficiency, accuracy, and performance of game name recognition.

[0087] In at least one embodiment, as a non-limiting example, with respect to deep learning, machine learning, and / or high-performance computing, software 918 and / or services 920 may be optimized for GPU processing. In at least one embodiment, at least some of the computing environments of deployment system 906 and / or training system 904 may be executed in a data center, one or more supercomputers, or high-performance computing systems with GPU-optimized software (e.g., the hardware and software combination of NVIDIA DGX systems). In at least one embodiment, as described herein, hardware 922 may include any number of GPUs that can be invoked to perform data processing in parallel. In at least one embodiment, the cloud platform may also include GPU-optimized execution for deep learning tasks, GPU processing for machine learning tasks, or other computing tasks. In at least one embodiment, an AI / deep learning supercomputer and / or GPU-optimized software (e.g., as provided on NVIDIA's DGX systems) may be used as a hardware abstraction and scaling platform to execute the cloud platform (e.g., NVIDIA's NGC). In at least one embodiment, the cloud platform may integrate application container cluster systems or coordination systems (e.g., KUBERNETES) across multiple GPUs to achieve seamless scaling and load balancing.

[0088] Figure 10 This is a system diagram of an example system 1000 for generating and deploying a deployment pipeline according to at least one embodiment. In at least one embodiment, system 1000 can be used to implement Figure 9 The process 900 and / or other processes include advanced processing and inference pipelines. In at least one embodiment, system 1000 may include a training system 904 and a deployment system 906. In at least one embodiment, the training system 904 and deployment system 906 may be implemented using software 918, services 920, and / or hardware 922, as described herein.

[0089] In at least one embodiment, system 1000 (e.g., training system 904 and / or deployment system 906) may be implemented in a cloud computing environment (e.g., using cloud 1026). In at least one embodiment, system 1000 may be implemented locally (regarding facilities) or as a combination of cloud computing resources and local computing resources. In at least one embodiment, access to the API in cloud 1026 may be restricted to authorized users by establishing security measures or protocols. In at least one embodiment, the security protocol may include a network token, which may be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and may carry appropriate authorization. In at least one embodiment, the API of the virtual instrument (described herein) or other instances of system 1000 may be restricted to a set of public IPs that have been audited or authorized for interaction.

[0090] In at least one embodiment, the various components of system 1000 may communicate with each other and among themselves using any of a variety of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communication between facilities and components of system 1000 (e.g., for sending inference requests, for receiving the results of inference requests, etc.) may be transmitted via one or more data buses, wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0091] In at least one embodiment, similar to the description herein. Figure 9 As described, training system 904 can execute training pipeline 1004. In at least one embodiment, where deployment system 906 uses one or more machine learning models in deployment pipeline 1010, training pipeline 1004 can be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1006 (e.g., without retraining or updating). In at least one embodiment, as a result of training pipeline 1004, output model 916 can be generated. In at least one embodiment, training pipeline 1004 can include any number of processing steps, AI-assisted annotation 910, tagging or annotating feedback data 908 to generate labeled data 912, selecting a model from a model registry, model training 914, training, retraining, or updating the model, and / or other processing steps. In at least one embodiment, different training pipelines 1004 can be used for different machine learning models used by deployment system 906. In at least one embodiment, similar to the approach regarding... Figure 9 The training pipeline 1004 described in the first example can be used for the first machine learning model, similar to the one described above. Figure 9The training pipeline 1004 described in the second example can be used for a second machine learning model, similar to the one described above. Figure 9 The training pipeline 1004 of the third example described can be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 904 can be used according to the requirements of each respective machine learning model. In at least one embodiment, one or more machine learning models may have already been trained and are ready for deployment, so the training system 904 may not perform any processing on the machine learning models, and the machine learning models may be implemented by the deployment system 906.

[0092] In at least one embodiment, according to an embodiment or example, one or more output models 916 and / or pre-trained models 1006 may include any type of machine learning model. In at least one embodiment, and not limited thereto, the machine learning model used by system 1000 may include models using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), k-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recursion, perceptrons, long / short-term memory (LSTM), Bi-LSTM, Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models.

[0093] In at least one embodiment, training pipeline 1004 may include AI-assisted annotation. In at least one embodiment, labeled data 912 (e.g., conventional annotation) can be generated using any number of techniques. In at least one embodiment, in some examples, labels or other annotations can be generated by drawing programs (e.g., annotation programs), computer-aided design (CAD) programs, tagging programs, another type of application suitable for generating annotations or labels for ground reality, and / or can be hand-drawn. In at least one embodiment, ground reality data can be synthetically generated (e.g., generated from computer models or renderings), realistically generated (e.g., designed and generated from real-world data), automatically generated by machines (e.g., extracting features from data using feature analysis and learning, and then generating labels), manually annotated (e.g., taggers or annotation experts, defining the placement of labels), and / or combinations thereof. In at least one embodiment, for each instance of feedback data 908 (or other data types used by the machine learning model), there may be corresponding ground reality data generated by training system 904. In at least one embodiment, AI-assisted annotation can be performed as part of deployment pipeline 1010; supplementing or replacing AI-assisted annotation included in training pipeline 1004. In at least one embodiment, system 1000 may include a multi-layer platform, which may include a software layer (e.g., software 918) of a diagnostic application (or other application type) that can perform one or more medical imaging and diagnostic functions.

[0094] In at least one embodiment, the software layer may be implemented as a secure, encrypted, and / or certified API that can invoke (e.g., call) an application or container from an external environment (e.g., facility 902). In at least one embodiment, the application may then invoke or execute one or more services 920 to perform computational, AI, or visualization tasks associated with their respective applications, and the software 918 and / or service 920 may utilize hardware 922 to perform processing tasks efficiently and effectively.

[0095] In at least one embodiment, deployment system 906 may execute deployment pipeline 1010. In at least one embodiment, deployment pipeline 1010 may include any number of applications, which may be sequential, non-sequential, or otherwise applied to feedback data (and / or other data types) – including AI-assisted annotations, as described above. In at least one embodiment, as described herein, deployment pipeline 1010 for an individual device may be referred to as a virtual instrument for the device. In at least one embodiment, for a single device, there may be more than one deployment pipeline 1010, depending on the desired information from the data generated from the device.

[0096] In at least one embodiment, the applications that can be used to deploy pipeline 1010 may include any application that can perform processing tasks on feedback data or other data from the device. In at least one embodiment, since various applications can share common image operations, in some embodiments, data augmentation libraries (e.g., as one of services 920) may be used to accelerate these operations. In at least one embodiment, to avoid bottlenecks in traditional processing methods that rely on CPU processing, parallel computing platform 1030 may be used for GPU acceleration of these processing tasks.

[0097] In at least one embodiment, deployment system 906 may include user interface 1014 (e.g., graphical user interface, web interface, etc.) which may be used to select applications to be included in deployment pipeline 1010, deploy applications, modify or change applications or their parameters or configurations, use and interact with deployment pipeline 1010 during setup and / or deployment, and / or otherwise interact with deployment system 906. In at least one embodiment, although not shown with respect to training system 904, user interface 1014 (or different user interfaces) may be used to select models to be used in deployment system 906, to select models to be trained or retrained in training system 904, and / or to otherwise interact with training system 904.

[0098] In at least one embodiment, in addition to the application coordination system 1028, a pipeline manager 1012 may be used to manage interactions between applications or containers deployed through the pipeline 1010 and services 920 and / or hardware 922. In at least one embodiment, the pipeline manager 1012 may be configured to facilitate interactions from application to application, from application to service 920, and / or from application or service to hardware 922. In at least one embodiment, although shown as included in software 918, this is not intended to be limiting, and in some examples, the pipeline manager 1012 may be included in service 920. In at least one embodiment, the application coordination system 1028 (e.g., Kubernetes, DOCKER, etc.) may include a container coordination system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline 1010 (e.g., rebuilding applications, splitting applications, etc.) with individual containers, each application can execute in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0099] In at least one embodiment, each application and / or container (or its image) can be developed, modified, and deployed independently (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer). This allows focus on the tasks of a single application and / or container without being hindered by the tasks of another application or container. In at least one embodiment, the pipeline manager 1012 and the application coordination system 1028 can facilitate communication and collaboration between different containers or applications. In at least one embodiment, the application coordination system 1028 and / or the pipeline manager 1012 can facilitate communication and resource sharing between and within each application or container, provided that the expected inputs and / or outputs of each container or application are known to the system (e.g., based on the application or container's architecture). In at least one embodiment, since one or more applications or containers in the deployment pipeline 1010 can share the same services and resources, the application coordination system 1028 can coordinate, load balance, and determine the sharing of services or resources between and within the various applications or containers. In at least one embodiment, the scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. Therefore, in at least one embodiment, taking into account the system's needs and availability, the scheduler can allocate resources to different applications and distribute resources between and among applications. In some examples, the scheduler (and / or other components of the application coordination system 1028) can determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as Quality of Service (QoS), the urgency of data output (e.g., to determine whether to perform real-time processing or delayed processing), etc.

[0100] In at least one embodiment, service 920, utilized and shared by applications or containers in deployment system 906, may include computing service 1016, AI service 1018, visualization service 1020, and / or other service types. In at least one embodiment, an application may invoke (e.g., execute) one or more services 920 to perform processing operations for the application. In at least one embodiment, an application may utilize computing service 1016 to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more computing services 1016 may be utilized to perform parallel processing (e.g., using parallel computing platform 1030) to process data substantially simultaneously through one or more applications and / or one or more tasks of a single application. In at least one embodiment, parallel computing platform 1030 (e.g., NVIDIA's CUDA) may implement general-purpose computing on a GPU (GPGPU) (e.g., GPU 1022). In at least one embodiment, the software layer of parallel computing platform 1030 may provide access to the GPU's virtual instruction set and parallel computing elements to execute computing kernels. In at least one embodiment, the parallel computing platform 1030 may include memory, and in some embodiments, memory may be shared between and within multiple containers, and / or between and within different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within containers to use the same data from a shared memory segment of the parallel computing platform 1030 (e.g., where multiple different stages of one or more applications are processing the same information). In at least one embodiment, instead of copying data and moving it to different locations in memory (e.g., read / write operations), the same data in the same memory location can be used for any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, this information about the new location of the data can be stored and shared between applications because the resulting data from processing is used to generate new data. In at least one embodiment, the location of the data, and the location of the updated or modified data, may be part of the definition of how the payload in the container is understood.

[0101] In at least one embodiment, AI service 1018 may be used to perform an inference service for executing a machine learning model associated with the application (e.g., a task to perform one or more processing tasks of the application). In at least one embodiment, AI service 1018 may utilize AI system 1024 to execute a machine learning model (e.g., a neural network such as a CNN) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, the application deploying pipeline 1010 may use one or more output models 916 from training system 904 and / or other models from the application to perform inference on imaging data (e.g., DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.). In at least one embodiment, two or more examples of inference using application coordination system 1028 (e.g., a scheduler) may be available. In at least one embodiment, the first category may include a high-priority / low-latency path that can implement a higher service level protocol, such as for performing inference on urgent requests in emergency situations or for radiologists during diagnostic procedures. In at least one embodiment, the second category may include standard priority paths, which can be used for requests that may not be urgent or for situations where analysis can be performed at a later time. In at least one embodiment, the application coordination system 1028 may allocate resources (e.g., services 920 and / or hardware 922) based on priority paths for different inference tasks of the AI ​​service 1018.

[0102] In at least one embodiment, shared memory may be installed into AI service 1018 in system 1000. In at least one embodiment, shared memory may operate as a cache (or other storage device type) and may be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a set of API instances of deployment system 906 may receive the request and may select one or more instances (e.g., for best fit, for load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request may be fed into a database, and if not already in the cache, a machine learning model may be located from model registry 924. A verification step may ensure that an appropriate machine learning model is loaded into the cache (e.g., shared memory), and / or a copy of the model may be saved to the cache. In at least one embodiment, if the application is not already running or there are not enough instances of the application, a scheduler (e.g., the scheduler of pipeline manager 1012) may be used to start the application referenced in the request. In at least one embodiment, if an inference server has not yet been started to execute the model, an inference server may be started. In at least one embodiment, any number of inference servers may be started for each model. In at least one embodiment, in a pull model that clusters inference servers, the model can be cached whenever load balancing is favorable. In at least one embodiment, the inference servers can be statically loaded into the corresponding distributed servers.

[0103] In at least one embodiment, an inference server running in a container can be used to perform inference. In at least one embodiment, an instance of the inference server can be associated with a model (and optionally multiple versions of the model). In at least one embodiment, if an instance of the inference server does not exist when a request to perform inference on the model is received, a new instance can be loaded. In at least one embodiment, when the inference server is started, a model can be passed to the inference server, allowing the same container to be used to serve different models, as long as the inference server runs as different instances.

[0104] In at least one embodiment, during application execution, an inference request for a given application can be received, and a container (e.g., an instance of a hosted inference server) can be loaded (if not already loaded), and a launcher can be invoked. In at least one embodiment, preprocessing logic within the container can (e.g., using a CPU and / or GPU and / or DPU) load, decode, and / or perform any additional preprocessing on the incoming data. In at least one embodiment, once the data is ready for inference, the container can infer the data as needed. In at least one embodiment, this can include a single inference call for an image (e.g., a hand X-ray) or can request inference for hundreds of images (e.g., a chest CT scan). In at least one embodiment, the application can summarize the results before completion, which may include, but is not limited to, a single confidence score, pixel-level segmentation, voxel-level segmentation, generating visualizations, or generating text to summarize the results. In at least one embodiment, different priorities can be assigned to different models or applications. For example, some models may have a real-time (TAT less than 1 minute) priority, while other models may have a lower priority (e.g., TAT less than 10 minutes). In at least one embodiment, model execution time can be measured from the requesting agency or entity, and may include cooperative network traversal time and inference service execution time.

[0105] In at least one embodiment, the transfer of requests between service 920 and the inference application can be hidden behind a software development kit (SDK) and robust transfer can be provided via queues. In at least one embodiment, requests are placed in queues via an API for individual application / tenant ID combinations, and the SDK pulls requests from the queues and provides them to the application. In at least one embodiment, the name of the queue can be provided in the environment where the SDK picks up the queue. In at least one embodiment, asynchronous communication via queues may be useful because it allows any instance of the application to pick up work as it becomes available. In at least one embodiment, results can be sent back via queues to ensure no data loss. In at least one embodiment, queues can also provide the ability to partition work, as the highest priority work can go into a queue connected to a majority of instances of the application, while the lowest priority work can go into a queue connected to a single instance that processes tasks in the order they are received. In at least one embodiment, the application can run on a GPU-accelerated instance generated in cloud 1026, and the inference service can perform inference on the GPU.

[0106] In at least one embodiment, visualization service 1020 can be used to generate visualizations for viewing the output of application and / or deployment pipeline 1010. In at least one embodiment, visualization service 1020 can utilize GPU 1022 to generate visualizations. In at least one embodiment, visualization service 1020 can implement rendering effects such as ray tracing to generate higher quality visualizations. In at least one embodiment, visualizations can include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slicing, virtual reality display, augmented reality display, etc. In at least one embodiment, a virtualized environment can be used to generate virtual interactive displays or environments (e.g., virtual environments) for system users (e.g., doctors, nurses, radiologists, etc.) to interact with. In at least one embodiment, visualization service 1020 can include an internal visualizer, cinematic and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0107] In at least one embodiment, hardware 922 may include GPU 1022, AI system 1024, cloud 1026, and / or any other hardware for executing training system 904 and / or deployment system 906. In at least one embodiment, GPU 1022 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that can be used to perform processing tasks for any feature or function of computing service 1016, AI service 1018, visualization service 1020, other services, and / or software 918. For example, for AI service 1018, GPU 1022 may be used to perform preprocessing on imaging data (or other data types used by machine learning models), postprocessing on the output of machine learning models, and / or inference (e.g., to execute machine learning models). In at least one embodiment, cloud 1026, AI system 1024, and / or other components of system 1000 may use GPU 1022. In at least one embodiment, cloud 1026 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1024 may use a GPU, and one or more AI systems 1024 may be used to perform cloud 1026 (or at least part of a task for deep learning or inference). Similarly, although hardware 922 is shown as a discrete component, this is not intended to be limiting, and any component of hardware 922 may be combined with or utilized by any other component of hardware 922.

[0108] In at least one embodiment, AI system 1024 may include a specially built computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, in addition to CPU, RAM, memory, and / or other components, features, or functions, AI system 1024 (e.g., NVIDIA's DGX) may also include GPU-optimized software (e.g., a software stack) that can be performed using multiple GPUs 1022. In at least one embodiment, one or more AI systems 1024 may be implemented in a cloud 1026 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1000.

[0109] In at least one embodiment, cloud 1026 may include GPU-accelerated infrastructure (e.g., NVIDIA's NGC) that can provide a GPU-optimized platform for performing processing tasks of system 1000. In at least one embodiment, cloud 1026 may include AI system 1024 for performing one or more AI-based tasks of system 1000 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1026 may be integrated with application coordination system 1028 utilizing multiple GPUs to achieve seamless scaling and load balancing between and within applications and services 920. In at least one embodiment, as described herein, cloud 1026 may be responsible for performing at least some of the services 920 of system 1000, including computing service 1016, AI service 1018, and / or visualization service 1020. In at least one embodiment, cloud 1026 may perform large and small batch inference (e.g., perform NVIDIA's TENSOR RT), provide accelerated parallel computing APIs and platform 1030 (e.g., NVIDIA's CUDA), perform application coordination system 1028 (e.g., KUBERNETES), provide graphics rendering APIs and platform (e.g., for ray tracing, 2D graphics, 3D graphics and / or other rendering techniques to produce higher quality cinematic effects), and / or provide other functionalities for system 1000.

[0110] In at least one embodiment, to protect patient confidentiality (e.g., in cases of off-site use of patient data or records), cloud 1026 may include a registry—e.g., a deep learning container registry. In at least one embodiment, the registry may store containers for instantiating applications that can perform preprocessing, post-processing, or other processing tasks on patient data. In at least one embodiment, cloud 1026 may receive data, including patient data and sensor data in containers, perform the requested processing only on the sensor data in those containers, and then forward the output and / or visualization to appropriate parties and / or devices (e.g., local medical devices for visualization or diagnosis), without extracting, storing, or otherwise accessing the patient data. In at least one embodiment, the confidentiality of patient data is maintained in accordance with HIPAA and / or other data regulations.

[0111] At least one embodiment of this disclosure may be described in accordance with the following terms:

[0112] In Clause 1, a method for deploying a custom machine learning model (MLM) includes: accessing a plurality of trained MLMs, each trained MLM associated with initial configuration settings; providing a user interface (UI) for receiving user input indicating selection of one or more of the plurality of trained MLMs and modification of the initial configuration settings of the one or more MLMs; determining modified configuration settings of the one or more selected MLMs based on the user input; causing a build engine to be executed to modify the one or more selected MLMs according to the modified configuration settings; causing a deployment engine to be executed to deploy the one or more modified MLMs; and causing a representation of the one or more deployed MLMs to be displayed on the UI.

[0113] In Clause 2, the method described in Clause 1 is used, wherein the one or more selected MLMs are pre-trained using a first set of training data.

[0114] In Clause 3, the method according to Clause 2 further includes: receiving a second set of training data for performing domain-specific training on the one or more selected MLMs; and causing a training engine that executes the pipeline to perform the domain-specific training on the one or more selected MLMs.

[0115] In Clause 4, the method according to Clause 1 further includes: before receiving the user input indicating the selection of the one or more MLMs, causing a list of the plurality of trained MLMs to be displayed on the UI; and after receiving the user input indicating the selection of the one or more MLMs, causing an export engine to be executed to initialize the one or more selected MLMs.

[0116] In Clause 5, the method according to Clause 4, wherein causing the export engine to be executed further includes: causing a representation of at least one of the schema of the one or more selected MLMs or parameters of the one or more selected MLMs to be displayed on the UI.

[0117] In Clause 6, the method according to Clause 4 further includes enabling the execution of the export engine by making the one or more selected MLMs available for processing user input data.

[0118] In Clause 7, the method according to Clause 6 further includes: receiving the user input data; causing the one or more deployed MLMs to be applied to the user input data to generate output data; and causing at least one of a representation of the output data or a reference to a stored representation of the output data to be displayed on the UI.

[0119] In Clause 8, according to the method described in Clause 1, the one or more selected MLMs include neural network models arranged in the pipeline.

[0120] In Clause 9, according to the method described in Clause 1, the one or more selected MLMs include an acoustic neural network model and a language neural network model.

[0121] In Clause 10, according to the method described in Clause 9, the modified configuration settings of said one or more selected MLMs include at least one of the following: the audio buffer size of the acoustic preprocessing stage of the acoustic neural network model; the speech end setting of the acoustic neural network model; the delay setting of the acoustic neural network model; or the language setting of the language neural network model.

[0122] In Clause 11, the method described in Clause 8, wherein the one or more selected MLMs include at least one of a speech synthesis neural network model, a language understanding neural network model, or a question-answering neural network model.

[0123] In Clause 12, a system includes: a memory device; and one or more processing devices communicatively coupled to the memory device for: accessing a plurality of trained machine learning model (MLM) models, each trained MLM associated with initial configuration settings; providing a user interface (UI) for receiving user input indicating selection of one or more of the plurality of trained MLM models and modification of the initial configuration settings of the one or more MLM models; determining modified configuration settings of one or more selected MLM models based on the user input; causing a build engine to execute a pipeline to modify the one or more selected MLM models according to the modified configuration settings; causing a deployment engine to execute the pipeline to deploy one or more modified MLM models; and causing a representation of one or more deployed MLM models to be displayed on the UI.

[0124] In Clause 13, the system according to Clause 12, wherein the one or more processing devices are further configured to: display a list of the plurality of trained MLMs on the UI before receiving the user input indicating selection of the one or more MLMs; and after receiving the user input indicating selection of the one or more MLMs, execute the pipeline's export engine to initialize the one or more selected MLMs.

[0125] In Clause 14, the system pursuant to Clause 13, wherein in order to enable the execution of the export engine, the one or more processing devices are further configured to: display on the UI a representation of at least one of the schema of the one or more selected MLMs or parameters of the one or more selected MLMs.

[0126] In Clause 15, the system according to Clause 12, wherein the one or more processing devices are further configured to: receive user input data; cause the one or more deployed MLMs to be applied to the user input data to generate output data; and cause at least one of a representation of the output data or a reference to a stored representation of the output data to be displayed on the UI.

[0127] In Clause 16, the system described in Clause 12, wherein the modified configuration settings of the one or more selected MLMs include at least one of the following: audio buffer size of the acoustic preprocessing stage of the acoustic neural network model; speech end setting of the acoustic neural network model; delay setting of the acoustic neural network model; or language setting of the language neural network model.

[0128] In Clause 17, the system described in Clause 12, wherein the one or more selected MLMs include at least one of a speech synthesis neural network model, a language understanding neural network model, or a question-answering neural network model.

[0129] In Clause 18, a non-transitory computer-readable medium storing instructions, wherein the instructions, when executed by a processing device, cause the processing device to: access a plurality of trained machine learning model (MLM) models, each trained MLM associated with an initial configuration setting; provide a user interface (UI) for receiving user input indicating the selection of one or more of the plurality of trained MLM models and modification of the initial configuration settings of the one or more MLM models; determine, based on the user input, a modified configuration setting of one or more selected MLM models; cause a pipeline build engine to modify the one or more selected MLM models according to the modified configuration settings; cause a pipeline deployment engine to deploy one or more modified MLM models; and cause a representation of one or more deployed MLM models to be displayed on the UI.

[0130] In Clause 19, the computer-readable medium pursuant to Clause 18, wherein the instructions further cause the processing device to: display a list of the plurality of trained MLMs on the UI before receiving the user input indicating the selection of the one or more MLMs; and after receiving the user input indicating the selection of the one or more MLMs, cause the pipeline's export engine to be executed to initialize the one or more selected MLMs.

[0131] In Clause 20, the computer-readable medium pursuant to Clause 19, wherein, in order to cause the export engine to be executed, the instructions further cause the processing device to: cause a representation of at least one of the schema of the one or more selected MLMs or the parameters of the one or more selected MLMs to be displayed on the UI.

[0132] In Clause 21, the computer-readable medium pursuant to Clause 18, wherein the instructions further cause the processing device to: receive user input data; cause the one or more deployed MLMs to be applied to the user input data to generate output data; and cause at least one of a representation of the output data or a reference to a stored representation of the output data to be displayed on the UI.

[0133] In Clause 22, the computer-readable medium pursuant to Clause 20, wherein the modified configuration settings of the one or more selected MLMs include at least one of the following: audio buffer size of the acoustic preprocessing stage of the acoustic neural network model; speech end setting of the acoustic neural network model; delay setting of the acoustic neural network model; or language setting of the language neural network model.

[0134] In Clause 23, the computer-readable medium pursuant to Clause 20, wherein the one or more selected MLMs include at least one of a speech synthesis neural network model, a language understanding neural network model, or a question-answering neural network model.

[0135] Other variations are within the spirit of this disclosure. Therefore, although the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0136] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar references, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (referring to a physical connection where not modified) should be interpreted as partially or wholly contained, attached to, or joined together, even with some intervention. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0137] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” means multiple items). In at least one embodiment, the number of items in the multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” rather than “based on only”.

[0138] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is executed jointly on one or more processors via hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) within transient signal transceivers. In at least one embodiment, code (e.g., executable code or source code) is stored on one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media lack all the code, but the multiple non-transitory computer-readable storage media collectively store all the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions, while the graphics processing unit (“GPU”) and / or data processing unit (“DPU”) (which may be together with the GPU) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0139] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the operations of the processes described herein, either individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating in different ways, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0140] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not constitute a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0141] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference as if each reference were individually and specifically indicated to be incorporated herein by reference and the entire contents of which are described herein.

[0142] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0143] Unless otherwise expressly stated, it will be understood that throughout this specification, terms such as “processing,” “computing,” “determining,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that process and / or convert data represented as physical quantities (e.g., electrons) in the registers and / or memory of the computing system into other data represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0144] Similarly, the term "processor" can refer to any device or part of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or intermittently, sequentially, or in parallel. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0145] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an application programming interface, or an inter-process communication mechanism.

[0146] While the description herein illustrates exemplary embodiments of the described technologies, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for descriptive purposes, various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0147] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A method for deploying a custom machine learning model (MLM), the method comprising: Enables the execution of the build engine to: Modify one or more selected MLMs from a plurality of trained MLMs according to one or more updated profiles; each updated profile corresponds to at least one corresponding selected MLM from the one or more selected MLMs and includes at least one modification to the initial set of configuration settings for the at least one corresponding selected MLM; as well as Configure the image container for executing one or more modified MLMs using the one or more updated configuration files; as well as Enables the deployment engine to: Metrics are collected for one or more processing units that can be used to deploy the one or more modified MLMs, wherein the metrics include at least one of the following: CPU speed, number of CPU cores, number of GPUs, amount of available system memory and / or GPU memory, number of remote processing nodes available for pipeline deployment; Based on the collected metrics and the image container, an execution container is generated, the execution container including one or more executable codes for executing the one or more selected MLMs according to the one or more updated configuration files; and Using the generated execution container, deploy the one or more selected MLMs on the one or more processing units; as well as This enables the display of one or more representations of the deployed MLM on the user interface (UI).

2. The method of claim 1, wherein the one or more selected MLMs are pre-trained using a first set of training data.

3. The method according to claim 2, further comprising: Receive a second set of training data for domain-specific training of the one or more selected MLMs; as well as This enables the execution of the training engine to perform domain-specific training on the one or more selected MLMs.

4. The method of claim 1, wherein the one or more selected MLMs are selected by user input via the UI, indicating a selection of the one or more MLMs, and further comprising: Before receiving the user input indicating the selection of one or more MLMs, cause a list of the plurality of trained MLMs to be displayed on the UI; as well as Upon receiving the user input indicating the selection of one or more MLMs, the export engine is executed to initialize the one or more selected MLMs.

5. The method of claim 4, wherein executing the export engine further comprises: This enables the display on the UI of a representation of at least one of the schema of the one or more selected MLMs or the parameters of the one or more selected MLMs.

6. The method of claim 4, wherein executing the export engine further comprises: This makes one or more selected MLMs available for processing user input data.

7. The method according to claim 6, further comprising: Receive the user input data; This allows one or more deployed MLMs to be applied to the user input data to generate output data; as well as This enables the display on the UI of at least one of a representation of the output data or a reference to a stored representation of the output data.

8. The method of claim 1, wherein the one or more selected MLMs comprise neural network models arranged in a pipeline.

9. The method of claim 1, wherein the one or more selected MLMs comprise an acoustic neural network model and a language neural network model.

10. The method of claim 9, wherein the initial set of configuration settings for the one or more selected MLMs comprises at least one of the following: The audio buffer size of the acoustic preprocessing stage of the acoustic neural network model; The speech end setting of the acoustic neural network model; The delay setting of the acoustic neural network model; or The language settings of the language neural network model.

11. The method of claim 8, wherein the one or more selected MLMs include at least one of a speech synthesis neural network model, a language understanding neural network model, or a question-answering neural network model.

12. A system comprising: Memory devices; and One or more processing devices, communicatively coupled to the memory device, are used for: This enables the execution pipeline's build engine to: Modify one or more selected MLMs from a plurality of trained MLMs according to one or more updated profiles; each updated profile corresponds to at least one corresponding selected MLM from the one or more selected MLMs and includes at least one modification to the initial set of configuration settings for the at least one corresponding selected MLM; as well as Configure the image container for executing one or more modified MLMs using the one or more updated configuration files; as well as This enables the deployment engine that executes the pipeline to: Metrics are collected for one or more processing units that can be used to deploy the one or more modified MLMs, wherein the metrics include at least one of the following: CPU speed, number of CPU cores, number of GPUs, amount of available system memory and / or GPU memory, number of remote processing nodes available for pipeline deployment; Based on the collected metrics and the image container, an execution container is generated, the execution container including one or more executable codes for executing the one or more selected MLMs according to the one or more updated configuration files; and Using the generated execution container, deploy the one or more selected MLMs on the one or more processing units; as well as This enables the display of one or more representations of the deployed MLM on the user interface (UI).

13. The system of claim 12, wherein the one or more selected MLMs are selected via user input through the UI, indicating a selection of the one or more MLMs, and the one or more processing devices are further configured to: Before receiving the user input indicating the selection of one or more MLMs, cause a list of the plurality of trained MLMs to be displayed on the UI; and Upon receiving the user input indicating the selection of one or more MLMs, the pipeline's export engine is executed to initialize the one or more selected MLMs.

14. The system of claim 13, wherein, in order to execute the export engine, the one or more processing devices are further configured to: This enables the display on the UI of a representation of at least one of the schema of the one or more selected MLMs or the parameters of the one or more selected MLMs.

15. The system of claim 12, wherein the one or more processing devices are further configured to: Receive user input data; This causes one or more deployed MLMs to be applied to the user input data to generate output data; and This enables the display on the UI of at least one of a representation of the output data or a reference to a stored representation of the output data.

16. The system of claim 12, wherein the initial set of configuration settings for the one or more selected MLMs comprises at least one of the following: The size of the audio buffer in the acoustic preprocessing stage of an acoustic neural network model; The speech end setting of the acoustic neural network model; The delay setting of the acoustic neural network model; or Language settings for language neural network models.

17. The system of claim 12, wherein the one or more selected MLMs include at least one of a speech synthesis neural network model, a language understanding neural network model, or a question-answering neural network model.

18. A non-transitory computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processing device, cause the processing device to: This enables the execution pipeline's build engine to: Modify one or more selected MLMs from a plurality of trained MLMs according to one or more updated profiles; each updated profile corresponds to at least one corresponding selected MLM from the one or more selected MLMs and includes at least one modification to the initial set of configuration settings for the at least one corresponding selected MLM; as well as Configure the image container for executing one or more modified MLMs using the one or more updated configuration files; as well as This enables the deployment engine that executes the pipeline to: Metrics are collected for one or more processing units that can be used to deploy the one or more modified MLMs, wherein the metrics include at least one of the following: CPU speed, number of CPU cores, number of GPUs, amount of available system memory and / or GPU memory, number of remote processing nodes available for pipeline deployment; Based on the collected metrics and the image container, an execution container is generated, the execution container including one or more executable codes for executing the one or more selected MLMs according to the one or more updated configuration files; and Using the generated execution container, deploy the one or more selected MLMs on the one or more processing units; as well as This enables the display of one or more representations of the deployed MLM on the user interface (UI).

19. The computer-readable medium of claim 18, wherein the one or more selected MLMs are selected by user input via the UI, indicating a selection of the one or more MLMs, and the instruction further causes the processing device to: Before receiving the user input indicating the selection of one or more MLMs, cause a list of the plurality of trained MLMs to be displayed on the UI; and Upon receiving the user input indicating the selection of one or more MLMs, the pipeline's export engine is executed to initialize the one or more selected MLMs.

20. The computer-readable medium of claim 19, wherein, in order to cause the export engine to be executed, the instructions further cause the processing device to: This enables the display on the UI of a representation of at least one of the schema of the one or more selected MLMs or the parameters of the one or more selected MLMs.

21. The computer-readable medium of claim 18, wherein the instructions further cause the processing device to: Receive user input data; This causes one or more deployed MLMs to be applied to the user input data to generate output data; and This enables the display on the UI of at least one of a representation of the output data or a reference to a stored representation of the output data.

22. The computer-readable medium of claim 20, wherein the initial set of configuration settings for the one or more selected MLMs comprises at least one of the following: The size of the audio buffer in the acoustic preprocessing stage of an acoustic neural network model; The speech end setting of the acoustic neural network model; The delay setting of the acoustic neural network model; or Language settings for language neural network models.

23. The computer-readable medium of claim 20, wherein the one or more selected MLMs comprise at least one of a speech synthesis neural network model, a language understanding neural network model, or a question-answering neural network model.

Images