Spatial Neural Audio Processing in Content Generation Systems and Applications

Abstract

The approaches presented here enable the identification of sound from a sound source with respect to an array of microphones with a potentially unknown configuration, partly by utilizing differences in the audio signals received by the microphones. In at least one embodiment, audio signals are captured using an array of microphones, and audio features are extracted from these signals. The audio features can be processed with a first neural network to generate a feature vector representing the spatial location of an audio source with respect to the array of microphones. This spatial location is derived based on audio differences and independently of the availability of information indicating the physical configuration of the array of microphones.The feature vector can be provided to a task-specific model to perform at least one audio-related task that is partially based on spatial location.

Description

BACKGROUND

[0001] In many applications – such as video conferencing or voice recording – there may be a desire to improve the quality of the recorded audio. This may include, for example, extracting, enhancing and / or suppressing audio from specific sources or directions. During a video conference, this may include extracting or enhancing audio corresponding to the speech of a person speaking from a specific location or area, e.g., an area in front of a computer or podium used by that person for the video conference, as well as suppressing noise or audio from other sources or locations near the computer or podium, such as background noise that might be picked up from other directions.In scenarios where multiple microphones are used to capture audio signals, it is possible to leverage spatial information to extract, enhance, and / or reject audio signals originating from different directions, regions, or locations. This can be achieved, for example, by focusing on signals from a specific direction or range of directions. One approach is to leverage a direction and / or distance estimate, or a steering vector, determined relative to a fixed microphone array with a known configuration. However, estimating the direction of arrival of an audio signal requires knowledge of the geometry of the microphone array, which may limit the use of such an approach in situations where a random microphone configuration is used or where microphones (or other devices capable of capturing audio signals) may be rearranged.If a random set of microphones is not properly calibrated and placed in an unknown configuration, the direction of arrival of signals from an audio source captured by those microphones may be unknown or at least uncertain, which may impair the ability to extract, enhance, and / or reject audio signals from those unknown directions. BRIEF DESCRIPTION OF THE INVENTION

[0002] The invention is defined by the claims. To illustrate the invention, aspects and embodiments are described herein, which may or may not fall within the scope of the claims.

[0003] Disclosed are approaches for identifying sound from a sound source relative to an array of microphones having a potentially unknown configuration, using, in part, differences in the audio signals received by the microphones. In at least one embodiment, audio signals are acquired using an array of microphones, and audio features are extracted from those signals. The audio features may be processed with a first neural network to generate a feature vector representing a spatial location of an audio source relative to the plurality of microphones, wherein the spatial location is derived based on audio differences and independent of the availability of information indicative of a physical configuration of the plurality of microphones.The feature vector can be provided to a task-specific model to perform at least one audio-related task based in part on spatial location.

[0004] The disclosure extends to all novel aspects or features described and / or illustrated herein.

[0005] Further features of the disclosure are characterized by the independent and dependent claims.

[0006] Any feature of one aspect of the disclosure may be applied to other aspects of the disclosure in any suitable combination. In particular, method aspects may be applied to device or system aspects, and vice versa.

[0007] Furthermore, features implemented in hardware may be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly.

[0008] Any system or device feature described herein may also be provided as a method feature, and vice versa. System and / or device aspects described functionally (including means plus functional features) may alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0009] It is also to be understood that certain combinations of the various features described and defined in each aspect of the disclosure may be implemented and / or provided and / or used independently of one another.

[0010] The disclosure also provides computer programs and computer program products comprising software code configured to, when executed on a data processing device, perform any of the methods described herein and / or embody any of the device and system features described herein, including any or all component steps of a method.

[0011] The disclosure also includes a computer or computer system (including networked or distributed systems) having an operating system that supports a computer program for performing the methods described herein and / or for embodying the device or system features described herein.

[0012] The disclosure also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0013] The disclosure also provides a signal carrying one or more of the aforementioned computer programs.

[0014] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0015] Aspects and embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Various embodiments according to the present disclosure will be described with reference to the drawings in which: Fig. 1 illustrates an example system for performing location-based neural audio processing in accordance with at least one embodiment; Fig. 2A, Fig. 2B, Fig. 2C and Fig. 2D illustrate a physical configuration of microphones and sound sources of a system for performing location-based neural audio processing, as well as representations of audio signals captured by such microphones, according to at least one embodiment; Fig. 3 illustrates details associated with processing audio features using a neural network to generate feature vectors representing spatial locations of audio sources relative to the microphones and audio sources, according to at least one embodiment; Fig. 4-5 illustrate example processes that may be used with a system for performing location-based neural audio processing in accordance with at least one embodiment; Fig. 6 illustrates components of a distributed system that may be used for location-based neural audio processing according to at least one embodiment; Fig. 7A illustrates inference and / or training logic according to at least one embodiment; Fig. 7B illustrates inference and / or training logic according to at least one embodiment; Fig. 8 illustrates an example data center system according to at least one embodiment; Fig. 9 illustrates a computer system according to at least one embodiment; Fig. 10 illustrates a computer system according to at least one embodiment; Fig. 11 illustrates at least portions of a graphics processor according to one or more embodiments. Fig. 12 illustrates at least portions of a graphics processor according to one or more embodiments. Fig. 13 is an example data flow diagram for an advanced data processing pipeline according to at least one embodiment; Fig. 14 is a system diagram for an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computational pipeline according to at least one embodiment; and Fig. 15A and Fig. 15B illustrates a data flow diagram for a process for training a machine learning model and a client-server architecture for enhancing annotation tools with pre-trained annotation models, according to at least one embodiment. DETAILED DESCRIPTION

[0017] In the following description, various embodiments are described. For explanatory purposes, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified in order not to obscure the described embodiments.

[0018] The systems and methods described herein may be used, without limitation, by non-autonomous vehicles or machines, semi-autonomous or autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS), one or more in-vehicle infotainment systems, one or more emergency vehicle sensing systems), guided and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying vessels, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely operated vehicles such as drones, and / or other types of vehicles.Furthermore, the systems and methods described herein may be used for a variety of purposes, including, without limitation, machine control, machine locomotion, machine propulsion, synthetic data generation, model training or updating, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or other suitable applications.

[0019] Included embodiments may include a variety of different systems, such as automotive systems (e.g., an infotainment system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, aviation systems, media systems, boat systems, systems for monitoring smart areas, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems,that implement one or more language models - such as large language models (LLMs), systems for performing light transport simulations, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0020] Approaches according to various illustrative embodiments enable location-aware processing of content, independent of knowledge of the location or configuration of devices used to capture or acquire that content, or the location of the sources of that content. In at least one embodiment, one or more neural networks (NNs) may be trained (using captured and / or simulated training data) to derive location-aware embeddings or feature vectors associated with at least one sound source. The neural embeddings may be estimated for various physical configurations of microphones, e.g., for three or more microphones, to enable accurate directional or location inference.The physical configurations may include various configurations with a different number of microphones, different relative placements of the microphones, varying noise and reverberation in the microphones, and other related aspects. One or more neural networks or other machine learning models may also be trained to draw inferences for one or more location-based (or at least partially location-based) tasks, such as determining the location of audio sources useful for extracting, enhancing, and / or suppressing audio signals, among other such tasks. For example, a neural network may use cross-channel features, temporal features, and / or spectral features to generate embeddings or feature vectors that encode information corresponding to a location (or direction, or region, etc.).) associated with at least one sound source relative to the individual microphones of the microphone array. Once these embeddings or feature vectors are generated, they can be used to enhance, extract, and / or suppress audio signals from inferred locations, distances, and / or directions with respect to one or more unknown microphone configurations, thereby achieving robustness that allows a task-specific model to be implemented using location-aware neural audio processing without obtaining specific information about the location or direction that can be determined using a known microphone array.

[0021] In at least one embodiment, systems and methods for location-based neural audio processing may include receiving audio signals recorded with an array (or other set or collection) of microphones and extracting (or encoding) audio features from the audio signals. The audio features may relate to aspects that enable the derivation of the above-mentioned cross-channel features, temporal features, and spectral features. The audio features may be processed using at least one neural network to generate a feature vector representing a spatial location of a sound source relative to the array of microphones, independent of the availability of information indicating the physical configuration of the array of microphones.A generated feature vector may be provided to a task-specific model to perform at least one audio-related task, such as extracting speech corresponding to the spatial location, performing echo cancellation, enhancing audio signals received from an inferred spatial location associated with a sound source, and / or suppressing audio signals received from locations other than a spatial location associated with the sound source, among other such options.

[0022] In at least one embodiment, audio enhancement for location-based speech enhancement may be performed when an utterance from (or for) a particular location is available. For example, embeddings or feature vectors may contain features useful for determining location-relevant information corresponding to an utterance. The location-relevant information may be provided as input to a speech enhancement system to extract speech from a location of a multi-channel audio signal containing various sounds from different locations.Since the configuration of the microphones may not be known, the position-relevant information may not refer to an actual position, region, or direction, but may identify features or aspects useful in identifying sound originating from a particular region, so that sound originating from that region can be amplified and sound originating from outside that region can be suppressed for speech enhancement or other tasks. For example, the time at which sound from a source is received by microphones at different positions will be similar, so the relative timing can be used to identify sound from one location of that source and distinguish it from other sounds at other locations that have different relative timing information. Such an approach can, for example,used in in-vehicle communication where the goal is to extract the speech of different people in the vehicle or to enhance the speech of a person in a specific location, e.g., in the driver's seat. In one example, the speech may be separated from the sound source, including from a front left, front right, rear left, and rear right position. A system according to at least one embodiment may identify and extract speech (or other audio signals—such as singing or clapping) corresponding to these different positions in the vehicle, regardless of the identity of the sound sources at these positions—although the system may not be able to precisely identify which sound corresponds to which position.The system may be able to attribute sounds coming from the driver's seat position without knowing that the position corresponds to the driver's seat and not to another position in the vehicle. Such an approach is advantageous for scenarios such as ridesharing or when passengers cannot (and perhaps need not) be identified. Different car models may have different positions of sound sources and microphones, and other physical configurations may change; however, a system such as the one described here can provide robustness to different and / or unknown microphone placements and array geometries by using location-aware neural embeddings to attribute sounds from specific positions or directions to specific sound sources.In at least one embodiment, sound sources at a factory or other service location may be recorded and used in training or calibrating such systems. Alternatively, location-based registration may be performed during the activity of a single speaker, using other sensors in the vehicle (including a camera) to confirm the speaker's current location, among other such options.

[0023] In at least one embodiment, such approaches may also be used to suppress one or more sound sources, whether from a particular direction or outside a range of directions of particular sound sources. When the activity of such an unwanted sound source is detected or recorded, a system may extract information from the audio signal, in some embodiments independent of the availability of location information indicating a physical configuration of the plurality of microphones. This information may be used in an audio processing system to infer direction, distance, and / or location and then suppress a signal corresponding to that direction, distance, or location.In one operational example, such an approach enables playback suppression, placing microphones in an environment where a speaker or TV is playing content and selectively suppressing sources external to the speaker or TV.

[0024] A system for performing location-based neural audio processing according to at least one embodiment may use neural embeddings or feature vectors to encode location-based information of a sound source instead of, for example, geometric information. The neural embeddings or feature vectors may be used to generalize the estimation processes of a direction or location of an audio signal, which would otherwise only be defined when the microphone position is known in advance. Furthermore, such neural embeddings or feature vectors may be used to generalize a steering vector estimation process. A steering vector may be based on a signal propagation model estimated by eigenvalue decomposition of a covariance matrix that assumes little to no reverberation in the environment.However, such an assumption may have limitations in scenarios where microphones are placed at some distance. The robustness of using neural embeddings or feature vectors by location-aware neural audio processing according to at least one embodiment can address these limitations through the above-mentioned generalizations that are not based on informed positions and distances at the time of utterance.

[0025] Variations of these and other such functions may be used within the various embodiments as will be apparent to those skilled in the art in light of the teachings and suggestions contained herein.

[0026] Approaches according to various illustrated embodiments provide an efficient and accurate process for creating, modifying, and / or enhancing content. Fig. 1 illustrates an example system 100 for performing location-based neural audio processing according to at least one embodiment. In this example, a microphone array 102, or a collection of typically three or more microphones or audio capture devices, may be provided, whose relative positions or orientations may not be known. Audio data corresponding to the audio signals captured by the microphones of array 102 may be passed to a data augmentation module 104. The data augmentation module 104 is capable of augmenting, modifying, or enhancing the received audio signals. If the audio data is to be used as training data, the data augmentation may attempt to include data for various conditions associated with the captured audio signals.In at least one embodiment, such an extension may help increase the robustness of the processed audio data for a particular environment. In at least one embodiment, the data extension module 104 may include or be used with memory for storing instructions and a processing component capable of executing those instructions to simulate as many different conditions as possible with the received audio signals.

[0027] In one example, the microphones of array 102 may be a simulated array of microphones to provide audio signals that may be subject to channel swapping and channel dropouts. A data enhancement module 104 may permute channels associated with the audio signals to expose system 100 to additional configurations and different configurations. Likewise, data enhancement module 104 may perform channel blanking, using only audio signals from selected microphones (and not others), based in part on an environment. This may be performed to simulate certain physical configurations, such as the availability of microphones in an environment. In another example, data enhancement module 104 may simulate nulls in some channels associated with the audio signals to assist in simulating channel dropouts.In another example, different amounts and types of noise (or other sounds and speech) can be integrated into the audio signals to simulate different physical environments.

[0028] An output of a data augmentation module 104 may be passed to a location-based embedding process or module 120, which may include a feature extraction module 106 and a neural network (NN) 108 (or a NN-based inference module). In at least one example, a location-based embedding module 120 is associated with memory containing instructions and processing capabilities based in part on the execution of those instructions to perform multiple modules, including feature extraction, training, and / or testing of one or more neural networks. Separately, dedicated processing or execution units enable individual modules for feature extraction, training, and testing of one or more neural networks without requiring the instructions to be executed from memory.

[0029] In at least one embodiment, a feature extraction module 106 of the location-based embedding module 120 is capable of extracting features from the audio data captured by the individual microphones of the array 102 without knowing a physical configuration of the multiple microphones. At least one NN of the neural network module can be trained to distinguish audio data corresponding to different locations, directions, and / or positions of one or more sound sources with respect to the microphones of the array 102 without explicitly providing any actual information about a physical direction, distance, and / or configuration of the multiple microphones of the array 102. However, it is noted that such approaches can also be applied in systems where some information is known, such asthe direction or orientation of one or more microphones in an array, but not all information is available, so traditional approaches would still be unusable or at least potentially unreliable.

[0030] In one example, it is not necessary to determine an azimuth direction relative to a microphone array. Instead, a feature extraction module 106 may extract audio features that may be provided to a trained NN to infer a spatial location of a sound source, where the physical position may be unknown, but it may be determined that the sound may originate from the same relative position or direction as a previously detected sound based on the extracted audio feature information. For example, in an operation related to a smart speaker, instead of an azimuth (or other physical angle) estimated from an utterance, an embedding or feature vector may be used to determine proximity to a particular location of the source of the utterance from which information is to be extracted, even if the location in the physical world is unknown.

[0031] A neural network of the location-based embedding process 120 described here can be trained to perform inferences based on a varying number of input channels and therefore support the phase differences between the channels 106C as a feature, as described for the system 200 in Fig. 2. For example, there may be four or more channels, and each channel may include real and imaginary components 106A of the complex spectrum in the audio signals provided by the microphones of array 102. The real and imaginary components 106A may be treated as features other than magnitude and phase 106B, level and arrival time, and / or other aspects or features of such audio signals. Furthermore, reference to a NN generally includes one or more NNs working toward a common goal to enable location-aware neural audio processing. A neural network may be trained to support feature inputs with averaging and cross-channel attention. A channel may be used interchangeably with data associated with the channel, which may originate from one or more microphones of array 102.The channel may provide a portion of a feature for the NN during training. Channel attention may reflect meaningful aspects associated with a provided training input, such as an audio feature of the audio signals. In at least one embodiment, the attention may be associated with spatial aspects in the audio features.

[0032] One or more such features 106A-C may, for example, be interleaved with temporal features and spectral features. Furthermore, spatial features may be encoded using a power or amplitude of the received audio signals with respect to at least one point in time via the physical configurations of the microphones capturing the audio data. Thus, a four-channel input provided to the feature extraction module 106 may generate a four-channel output of associated features that may be provided to the NN. The NN may include multiple training layers, such as a channel processing layer, an inter-channel processing layer, a spectral processing layer, and a temporal processing layer, as described in Fig. 3 will be explained in more detail.

[0033] In at least one embodiment, one or more features may correspond to the spectrograms of the audio signals. For example, a Short-Time Fourier Transform (STFT) may be associated with a magnitude and phase of each channel of the audio signals. One or more of the magnitude and phase may also include complex-valued features, where the different channels represent real and imaginary parts of the audio signals. In at least one embodiment, the magnitude and phase may be processed differently than the real and imaginary parts of the audio signals.The different training layers may be provided to process the STFT of the different channels before a cross-channel processing layer can process nested features to account for the cross-channel phase difference features, all of which are part of channel processing enabled by the feature extraction module 106 and the NN module 108.

[0034] Additionally, temporal processing for each channel can be performed using a temporal processing layer of the NN module 108. For example, one or more NN layers of the NN module 108 can be used with each channel or with a mixed STFT from the cross-channel processing layer to capture temporal correlations within each channel. This allows the system 100 to utilize the dynamics inherent in the audio signals from the multiple microphones of the array 102. Furthermore, various temporal models can be used, including recurrent layers, temporal convolutional layers, and / or introspection layers along with position encoding as part of the layers of the NN module 108.

[0035] In at least one embodiment, cross-channel attention may be performed to process the streams associated with the channels. Furthermore, one or more of the NN layers may utilize merging of the audio signals from the audio channels into a single stream. Thus, cross-channel averaging and attention may be performed with one or more layers of the NN; and such processes may be applied to any number of input channels of the audio signals. For testing purposes, fusion enables part of the classification process for received test audio signals. In one example, high-performance, transformer-like models may be used with the NN layers for such aspects of merging the audio signals.

[0036] A trained neural network may provide a first inference output, which may be an embedding or feature vector. An embedding or feature vector may be used to represent information about differences in sound sources, including differences in the physical configurations of the multiple microphones of the array 102 from the training inputs, including at least channel-related, inter-channel, spectral, and temporal information. Thus, the embedding or feature vector may contain a certain number of elements, e.g., 512 or 1024 elements without restriction, allowing flexible generalization across different microphone arrays. The minimum number of elements may, e.g.,determined by a specific formulation, and the values ​​presented are merely examples without limitation, except that there should be a sufficient number of elements to ensure reasonable accuracy. In one example, an embedding or feature vector may be a vector of significant length that can encode information about each sound source relative to the microphone array. A NN module 108 may include at least a first NN with embedding layers trained to identify or distinguish the locations of the various sound sources. The embedding or feature vector need not specifically encode the fundamental frequency or pitch of the sound sources, but may also encode other aspects that contain relationships between all of these fundamental features.

[0037] In at least one embodiment, a task-specific model 110 may be used to perform a location-based task using the embeddings or feature vectors derived by the NN module 108. In at least one embodiment, a location-based task may be performed using a second NN or another machine learning model, which may use the embeddings or feature vector to derive location-, direction-, and / or distance-related information associated with one or more sound sources for further processing. The second NN or another model may be trained to derive sounds associated with a location at which a task is to be solved based on at least a portion of the embeddings or feature vector of the first NN and for a provided test input.The task and its associated location may, for example, include suppressing the sound from one of the sound sources, amplifying the sound from one of the sound sources, and / or echo cancellation with respect to the sound sources, even if the physical locations of these sources are indeterminate. The second NN is trained to provide a target based in part on the embeddings or feature vector of the first NN and may be unsupervised.

[0038] Another task that can be performed using such a task-specific model 110 relates to distinguishing between different sound sources. For example, two microphones from two different smart speakers may receive utterances requiring a response. However, the task of determining which smart speaker should be used to respond to the utterance can be solved by identifying the source and inferring its proximity to one of the smart speakers, without the utterance containing information about the location or distance of the source itself. When using multiple smart speakers, a smart speaker can provide a response to an utterance based on its physical proximity to the utterance.

[0039] In at least one embodiment, a task-specific model 110 associated with a second NN may be trained to estimate proximity using the embeddings or feature vector of the first NN. For example, the second NN may provide a second inference that includes information about which smart speaker is closer to the utterance or to provide relative distances for each smart speaker to the sound source. For example, the embeddings or feature vectors of the first NN may be used to train the second NN so that an inference on a test input including the utterance may be associated with the positional information between the sound source and a smart speaker.The position information may be used in a post-processing feature of the task-specific model 110 to provide a response from a smart speaker that is closer to the sound source than one that is farther from the sound source.

[0040] A System 100, as it is in Fig. 1 can therefore determine differences, such as cross-channel levels and delays, in audio signals received by multiple microphones from a sound source. Since the location or physical configuration of the microphones may not be known, a spatial representation can be determined based on features of the audio signal and used to train a neural network to enhance or filter one or more sounds of the audio signal. For example, even if the physical configuration of the multiple microphones, including the direction or location of the multiple microphones, is unknown, a neural network trained with different physical configurations and different sound sources can provide embeddings or feature vectors that enable estimates of the direction or location for each of the microphones.The embeddings or feature vectors, in at least one embodiment, are based in part on differences in the audio signals received by the different microphones, which is generalizable and does not rely on specific directional or location information.

[0041] Fig. 2A illustrates at least one physical configuration 230, among other aspects, of various microphones and sound sources in a system 200 for performing location-based neural audio processing, according to at least one embodiment. Fig. 2A illustrates, by way of example, a plurality of microphones 202 in a representation of an array of microphones 202. Furthermore, a sound source 208 that is different from one or more target sound sources 212 is located within an audio detection distance of the array of microphones 202. In one example, the one or more target sound sources 212 may include a desired sound source and provide a desired portion of an audio signal received by the array of microphones 202. The array of microphones 202 records audio signals A and B 204, which are subjected to processing for audio features 208. However, a sound source 208 that provides an undesired signal 206 may be suppressed, for example, as part of the processing of the audio features 208.In at least one embodiment, the physical configuration 230 may include a configuration with a different number of microphones 202, different relative placements 210 of the microphones, varying noise and reverberation in the microphones, among other related aspects.

[0042] The array of microphones 202 may capture audio signals A and B 204 from the environment and, in at least one embodiment, provide the audio signals to a data enhancement module. However, the audio signals A and B 204 may also be a basis for audio features 208 to be extracted. In at least one embodiment, STFT processes (or other such methods) may be used to extract one or more features from the audio signals A and B 204, including real and imaginary components 106A, magnitude and phase 106B, or phase differences between channels 106C. In one example, the audio signals A and B 204 may be represented as a mixed signal recorded over time by the array of microphones 202. The audio signals A and B 204 may represent a desired signal from a target source 212 (such asan utterance from a target source 212) and may include an unwanted signal 206 from an unwanted source 208. In one example, the unwanted signal 206 pertains to different sounds than the target source signal and may include environmental impulse responses (such as different acoustics or audio reflections) associated with a function associated with the sound source 208 and at least one microphone 202. The audio signals A and B 204 may include a four-channel mixed signal time-matched across the array of microphones 202.

[0043] The audio features 208 may correspond to representations following data augmentation by a data augmentation module. In at least one embodiment, the audio features 208 may correspond to, among other things, temporal features, spectral features, and / or cross-channel features, which are generally included in the features 106A-C and may have been extracted as part of the audio features using the feature extraction module 106. In at least one embodiment, the temporal features may be determined in the time domain, while the spectral features may be determined by frequency conversion of the time domain, e.g., by STFT of the audio signals A and B 204.

[0044] In at least one embodiment, a NN module 108 may be trained to learn at least one representation of an audio signal that can be associated with at least one direction or location. Feature extraction may provide aspects of the audio signal including at least real or imaginary parts of the complex spectrum 106A, magnitude and phase 106B, and the phase difference between channels 106C, which are used in training the NN. However, an estimated reverberation ratio along with level differences between channels may also be used for training. In one example, the reverberation ratio is a ratio of direct sound to reverberation. These features may be used to create the embeddings 204, which represent spatial location information independent of physical knowledge from the physical configuration 230.For example, a trained NN is capable of providing feature vectors or embeddings corresponding to features received in a test input that relate to specific locations, directions, or positions of a source of the test input relative to at least one microphone.

[0045] Fig. 2B, Fig. 2C and Fig. 2D illustrate a general example of determining spatial information that may be used in accordance with at least one embodiment. It should be noted that various approaches to determining spatial information using features extracted from audio data may be used, as presented herein, and that this example is presented to facilitate explaining how spatial information may be determined when physical location information is not available. Various other types of features may be used, as described herein, corresponding to differences in the audio signals received by different microphones of an array to thereby generate a "signature" for a particular sound source relative to the microphone array.While the differences may correspond to features such as timing of reception or pitch, various other types of features may also be used, some of which can be identified by a neural network but are not easily understood by a human.

[0046] In order 220 of Fig. 2B, there are three microphones 224, 226, 228, each of which can capture sound from a sound source 222. One or more aspects of the recorded audio signal will differ between the three microphones, based at least on the differences in distance from the sound source. For example, the same sound will reach the three microphones at different times, which may correspond to the distance differences d1 and d2, as shown in diagram 240 of Fig. 2C. All sounds emanating from the sound source 222 at this location will arrive at the microphones 224-226 with similar delays (or other location- or distance-related aspects). If the orientations 264, 266, 268 of the microphones are known, as in the arrangement 260 of Fig. 2D, the location (or at least the direction) of the sound source can be determined based on these particular differences in the signals, since there is only one location where these distance vectors can all end up at the same point 262 in space if they originate from these microphone locations. If the orientation of the microphone array is known, this relative physical position 262 or direction can be determined based on these signal differences determined for the various microphones.

[0047] However, if the physical orientations or locations of the microphones are unknown, the physical location or direction cannot be determined with any degree of certainty. Looking again at the distance vectors from Fig. 2C, these vectors can only be used to determine a single physical location if the starting points of these vectors (or the locations of the microphones) are known; otherwise, there may be many (or even infinite) possible locations where these vectors could converge from unknown starting points. However, it is known that for audio data originating from this spatial location, the differences in the three distance vectors are approximately equal, and that in many cases, the audio data contains additional audio features that, to a trained NN, are representative of this spatial location.Thus, even if the physical location is not provided with the audio features, one or more feature vectors determined by the trained NN based on the audio features of the sound source 222 from that location can be identified, since the vectors may contain at least some of the same or similar features for other audio data recorded by the microphones of the array with the same or similar relative positioning or location with respect to the sound source 222. In this way, location-based processing of audio data can be performed even if the exact physical location is not known, since features of the audio data recorded by a microphone array can be used to identify audio data originating from the same location.Thus, if a speaker can be determined to be located at a location that exhibits certain audio features or embeddings, then audio from that location can be enhanced for audio that exhibits similar features, and audio from sources at other locations or directions that exhibit different features in the recorded audio signals can be suppressed. Thus, all audio features, as in . Fig. 2C illustrates that differences that can be determined as consistent differences attributable to different locations can be used to distinguish between audio signals from different locations, even if those actual physical locations are not known.

[0048] Fig. 3 illustrates details 300 associated with processing audio features using a neural network to generate feature vectors representing the spatial locations of audio sources relative to the microphones and audio sources, according to at least one embodiment. The audio features (e.g., 106A-C, as shown in Fig. 2A) of the training input 314 may be processed (e.g., in their own input vectors 302A, B, 304, 306) using embedded layers 308 of a first NN of a NN module 108. This process may, regardless of the availability of information indicating a physical configuration 230 of an array of microphones, generate a feature vector or embeddings 204 representing a spatial location of an audio source with respect to the array of microphones 202. Once trained, the embedded layers 308 of the first NN are capable of, for example, generating embeddings or feature vectors 204 representing a first inference of locations associated with various sound sources and microphones estimated for a test input 316.

[0049] In at least one embodiment, the embeddings or feature vectors 204 can be used together with a test input 316 to determine the active periods of certain sound sources. For example, if an audio signal associated with a target sound source 212 is part of the test input 316, an inference can be created using the embeddings or feature vectors 204 to determine that the target sound source 212 is active for a specific period of time. Furthermore, the embeddings or feature vectors 204 of the second inference 310 can be assigned to one or more second NN layers 312, which can be generated and used with specialized applications, e.g., for audio signal enhancement.Thus, additional classifications or inferences may be created using different embeddings or feature vectors 204, where each embedding or feature vector contains a different spatial aspect, including a location, direction, relative positioning, etc., of a target sound source providing the test input 316 and at least one microphone in the microphone array.

[0050] In one example, location-based neural audio processing is not tied to a specific geometry and does not need to refer to a specific physical location—such as direction and distance—for the test input 316. Instead, the embeddings or feature vectors 204 represent an encoding of a specific location in space. For a given microphone geometry, it is possible to encode a specific location in space using the features described here and the embedded layers 308, such that a determination can be approximated by inference of a coordinate axis, a relative axial orientation (including in the x, y, and z axes), or a derivative in the Euclidean system with two angles and a distance based on a test input 316.This approach can be generalized to different microphone geometries, provided the embeddings or feature vectors are sized to contain enough information to allow discrimination for different microphone arrangements at different sources.

[0051] In at least one embodiment, an environment as in Fig. 2A with the physical configuration 230, may be a vehicle or refer to a computer or Internet of Things (IoT) device. To extract speech corresponding to an occupant, e.g., a driver, in the vehicle or a user of the computer or IoT device (e.g., a digital kiosk displaying a digital avatar or other digital assistant) from other people in the environment, the approaches described herein enable the use of a trained first neural network to provide embeddings or feature vectors 204 for various locations, directions, or relative positioning for at least one user with respect to the other users.For example, the first NN may be trained to provide a location-based embedding or feature vector 204 that encodes information about the location, direction, or relative positioning of the target sound source with respect to the microphone array 202, which may include a relative spatial location of other occupants or which may include other sound sources.

[0052] In one example, an audio signal for a target sound source 212 may be extracted by generating an embedding or feature vector 204 corresponding to the audio patterns of the target sound source 212. In a smart speaker environment, a smart speaker may be configured to detect only one target sound source 212, for example, using multiple recordings of sentences. However, in addition to detecting the target sound source 212, the embedding or feature vector 204 is also trained to learn a profile corresponding to a particular position of the target sound source 212 relative to the position of an array of microphones 202.Furthermore, there may be embeddings or feature vectors 204 that allow generalizations across different geometries of the microphone array, where each embedding or feature vector corresponds to a microphone in the array; then, a test input with each embedding or feature vector may be used to infer whether a location or other spatial feature can be provided for the test input.

[0053] Location-based neural audio processing has an advantage over pure direction estimation because if the initial neural network is trained to generalize from the information available to it, then it is not necessary to track a specific microphone configuration. This approach can be used with a calibration or testing phase and is used to extract audio signals from a specific location to generate embeddings or feature vectors of that location with respect to the microphone arrangement.A second inference is then performed by a task-specific model, which may be a convolutional neural network (CNN) or other types of neural networks, which may be configured to perform speech extraction, as well as audio interference, suppression, enhancement, and / or echo cancellation, on one or more audio signals that are part of a test input 316, using the embeddings or feature vectors of the embedded layers 308. The task-specific model 110 is capable of performing post-processing 318 based, in part, on an audio signal determined to originate from a particular position or general location.

[0054] Location-based neural audio processing does not require specific knowledge of the position of the microphone array 202 corresponding to the occupant or person of interest during training. For example, simulated or recorded sounds from multiple positions in an environment can be used during training for the first NN as part of the training input 314 to create robust embeddings or feature vectors that can be used with the test input 316 in a live environment to estimate the location of a sound source in the test input. Furthermore, since no geometric requirements are placed on the positions of the microphone array 202 with respect to the test input 316 and the relative distances or placements 210, and since other spatial information is encoded in the embeddings 204, the location-based neural audio processing described herein is scalable.

[0055] In at least one embodiment, real or simulated audio data may be used to generate audio signals received by an array of microphones 202; the derived embeddings or feature vectors 204 are robust and generalizable to cover as many different configurations as possible. The training input (such as the training input 314 in Fig. 3) may include various arrangements of microphones in different configurations, e.g., in different rotations, placements, and positioning. There may be no restrictions on the environments associated with the arrangement of microphones for training.

[0056] In one example, an environment may include rooms, open or closed areas, and other real estate or layout configurations. Furthermore, barriers and physical installations may also be considered in location-based neural audio processing. For example, deflections and other mountings of one or more microphone arrays would not affect inferences about the spatial information in the embeddings or feature vectors. Some microphones providing training inputs may be attached to different devices, e.g., computers. However, it is possible to simulate sound sources around these microphone configurations to create environmental effects such as barriers, which may then have no influence on the derived embeddings or feature vectors for a test input.In at least one embodiment, the embeds 204 can be used to detect the presence of a person or other obstacle in front of a sound source during testing and enable response actions. A response action may include automatically switching the screen of a kiosk when it is determined that someone is near or in front of the device, or enabling better reception of audio signals around the barrier.

[0057] With a trained first NN with the embedded layers 308, it is also possible to use the embeddings 204 to adapt aspects of the test input with audio signals from an environment. For example, for a test input 316, any audio signal source located near one or more microphones 202 can be provided along with the embeddings or the feature vector 204 to enable the determination of an estimated direction or other estimated spatial information. Further post-processing 318 in the task-specific model 110 enables the passing, suppression, and / or enhancement of an audio signal of the estimated direction. Alternatively, the suppression of audio signals coming from other directions of the test input can be performed while passing or enhancing the audio signal of the estimated direction.

[0058] In at least one embodiment, such an approach may eliminate the need to know the system's microphone geometry, which is part of the physical configuration of the system's microphones. Furthermore, to enable scalability to random devices with random physical configurations in the system for location-aware neural audio processing, a service may be provided that extracts the voice of a user, e.g., a vehicle driver, from different locations in the vehicle. Since different vehicle manufacturers may provide a different microphone configuration, such as a different number of microphones and a different location, the approaches presented here eliminate the need to know in advance the location of the test input microphones and to know or estimate the placement of the test input microphones.

[0059] In at least one embodiment, it is possible to dynamically update the embedded layers 308 over time, e.g., to track where a user moves relative to the microphone arrangement. For example, a target sound signal (of a target source 212) can be tracked to provide dynamic training input 314 and perform dynamic training of the first NN of the NN module 108. As a result, the embeddings 204 can be updated to account for position changes. In at least one embodiment, the embeddings 204, the second inference 310, or the post-processing 318 of the task-specific model 110 enable configuring a direction indicator or other aspects associated with a second inference 310.

[0060] The directional or other related aspects include a proximity or location range in an environment relative to a target source signal that is the focus of post-processing 318. Therefore, for certain embeddings or feature vectors 204, an estimated direction may be derived using the embeddings or feature vectors 204 that include an adjustment of, for example, + / - 15 degrees in a direction associated with a target source signal. For example, there may be a distribution in multiple axes around an estimated position or direction. To enable the extraction of information around a location or direction associated with a target sound signal, a space may be defined with respect to at least the embedded layers 308 so that the embeddings or feature vectors 204 can utilize such information.

[0061] Therefore, instead of training the embedded layers 308 for a specific direction, the embedded layers 308 may be trained to generate embeddings 204 for proximity to a specific location or direction. Alternatively, the trained embedded layers 308 may be used with a test input and with an additional input requiring more general inference to generate proximity to a specific location or direction without changes to the trained embedded layers 308. Therefore, it is possible to determine whether information associated with proximity to the embeddings should be extracted. In at least one embodiment, an embedding 204 from the embedded layers 308 may be used to represent spectral information corresponding to a target source signal.The embedding can be used to determine speaker extraction and other actions for the task-specific model 110. The embeddings 204 can also be used to represent spatial information of a specific sound source 208, independent of the speaker or the sound. This spatial information can be generalized to different microphone geometries through the embeddings or feature vectors 204, which incorporate learning from the features extracted and processed by the embedding layers 308.

[0062] In at least one embodiment, the embedded layers 308 represent a NN trained across multiple geometries and using various features of the audio signals associated with the microphone arrangement. Furthermore, the use of data augmentations, e.g., for speech enhancement, ensures that the NN is robust and generalizable to both speech and non-speech audio signals. Thus, the NN can be trained with speech, noise, music, or various other types of sounds from various other sound sources. The embedded layers 308 represent a NN trained for location estimation or equivalence, and trained on a large training corpus comprising various microphone arrangements and physical configurations, as well as various locations of audio sources relative to these configurations.The embeddings and feature vectors enable the discrimination between different locations and physical configurations and can help teach another NN a task-specific model to encode location information from the microphone signals.

[0063] Location-aware neural audio processing can account for barriers that may exist between a sound source and a microphone. Thus, instead of determining a steering vector or a relative transfer function, which can be an analytical model for signal propagation, training the NN on embeddings or feature vectors using the large training corpus overcomes the exceptions or assumptions associated with such barriers. While a steering vector provides information about a sound source in a particular direction, audio signals received at a microphone may exhibit differences and relative delays due to the barriers that have nothing to do with the direction itself. Location-aware neural audio processing eliminates the need to make assumptions about specific properties of a microphone array related to the obstacles.For example, distraction effects can be avoided if the distractions can be attributed to one or more microphones. Location-based neural audio processing does not require the assumption of free-field propagation or other practical or theoretical assumptions.

[0064] In at least one embodiment, a neural network may be trained to distinguish different locations based on at least the audio signals of a target sound source (as part of a test input) using features extracted from the audio signals. A test input 316 may represent features extracted from an example target sound signal. Multiple instances of a neural network may be trained to classify whether sounds within an audio signal originate from the same location or from different locations or directions in space. In at least one embodiment, at least two sound sources may be used during training to improve the robustness and generalizations of the system. It may be advantageous to use multiple sound sources, as three or more sound sources may be advantageously used to ensure the certainty of the spatial location.Furthermore, use cases can also benefit from at least two or three or more microphones recording audio signals for use with a trained NN of the system in question. Using only two microphones to localize a source can lead to confusion, as the same differences can be determined for two different positions in front of and behind the microphone pair (or in different directions relative to a line between the two microphones).

[0065] In at least one embodiment, a single physical microphone may contain multiple sensors or parts for which differences in the signals can be detected. In this way, inference with two microphones can be used to a limited extent, including for noise suppression outside spatial regions associated with at least two sound sources or microphones. For example, an NN of the NN module 108 trained to provide embeddings or feature vectors can be used to distinguish different locations associated with audio signals received and processed by the NN module 108. To localize an audio source, it is also possible to train the NN of the NN module 108 to always estimate the direction and distance to the microphones of the array 102.

[0066] Fig. 4 illustrates an example process 400 that may be used with a system for location-based neural audio processing according to at least one embodiment. It should be noted that for this and other processes discussed herein, there may be additional, fewer, or alternative steps performed in a similar or alternative order or at least partially in parallel within the various embodiments, unless explicitly stated otherwise. In this example, audio signals 402 recorded with multiple microphones are received, where the positions or directions of one or more of these microphones may be unknown to the system or unavailable to the process. A set of audio features 404 may be extracted from the received audio signals, and these extracted audio features may be used, among other things, to generate a set of embeddings or latent features.The audio features of the methods may be processed 406 using a trained neural network to generate at least one feature vector (or other set of embeddings or representations) representing a spatial location of an audio source relative to the microphones, independent of the availability of information indicative of a physical configuration of the plurality of microphones. The feature vector (or other representation), along with other inputs such as the acquired audio data or signals, may be provided 408 as input to a task-specific model (or module). The task-specific model (or module) may perform 410 at least one audio-related task on the audio data or signals based in part on the spatial location information derived from the neural network.

[0067] Such a process 400 may be supported by including, in the audio-related task performed, an extraction of speech corresponding to the spatial location, echo cancellation, enhancement of audio data received from the spatial location associated with the audio source, or suppression of audio signals received from locations other than the spatial location associated with the audio source. Furthermore, the audio features of step 404 include differences in the audio signals corresponding to at least one of the real or imaginary parts of a complex audio spectrum, signal magnitude, signal phase, or the ratio of direct sound to reverberation.

[0068] In at least one embodiment, such a process 400 is supported by including a minimum number of elements in the feature vector, such as at least 512 or 1,024 elements, wherein none of the elements need (or can) relate to a physical direction or physical location with respect to one or more of the plurality of microphones. Furthermore, such a method may include a further step or sub-step of merging the audio signals into a single audio signal prior to the step 410 of performing the at least one audio-related task. A further step or sub-step of the method 400 may include analyzing audio data indicated as being generated by the sound source to identify audio data of the sound source represented in the plurality of audio signals.In at least one embodiment, such a process 400 may be assisted by one or more of the feature vectors representing a location-based embedding of the audio source in a latent space; the audio features include cross-channel features, temporal features, and spectral features; or the microphones may include various types of audio capture devices.

[0069] Fig. 5 illustrates another example process 500 that may be used with a system to perform location-based neural audio processing according to at least one embodiment. In this example process 500, audio signals captured with multiple microphones are received 502, where these audio signals may include audio data generated by one or more audio sources within an audio pickup range of at least one of the microphones. Audio features extracted from the audio signals may be processed 504 to generate at least one feature vector representative of at least one spatial location of a sound source captured by the multiple microphones. The locations or orientations of some or all of the microphones may not be known, so the spatial location itself has no known correlation to a physical location.The at least one feature vector may be provided 506 as input to at least one task-specific model. The at least one task-specific model may determine 508 whether a spatial location corresponding to the sound source matches a previous spatial location determined for a previously identified sound source, e.g., if the spatial locations match the same spatial location within an allowable tolerance or threshold for location determination. If it is determined 510 that the spatial locations are the same, and the sounds then likely correspond to the same sound source or a new sound source at the same location, then the same task 512 as for the previously identified source may be performed, such as enhancing audio data originating from a person speaking from a particular location, region, or direction.If it is determined that the spatial locations are not the same, e.g., if the spatial locations correspond to different spatial locations that are outside an allowable tolerance or threshold for location determination, then a different task 514 may be performed (using the same task-specific model or a different task-specific model) than that performed for the previously identified source, such as suppressing audio data received from another person speaking from a different location, which may relate to background noise or unwanted speech.

[0070] In at least one embodiment, the spatial location of a sound source may be monitored and updated over time to track where a user or sound source moves relative to the array of microphones, reflecting a change in the position of the target sound signal. In one example, a target audio signal (of a target source) may be tracked to provide dynamic training input and to perform dynamic training of the first NN of a NN module. As a result, the embeddings or feature vectors may be updated to account for position changes.In at least one embodiment, the updates to the embeddings or feature vectors enable updates to the following inferences, and post-processing aspects of a task-specific model enable configuring a directionality or other related aspects associated with changes to the previously made inferences.

[0071] Fig. 6 illustrates an exemplary networked system configuration 600 that may be used to provide, generate, modify, encode, process, and / or transmit audio data or other such content. In at least one embodiment, a client device 602 may generate or receive data for a session using components of a control application 604 on a client device 602 and data stored locally on that client device.In at least one embodiment, a content application 624 executing on a server 620—for example, a cloud or edge server—may initiate a session associated with at least the client device 602, using a session manager and user data stored in a user database 636, and may cause content such as one or more audio representations—such as embeddings or feature vectors—to be selected from an object store 634 by a content manager 626 for processing.

[0072] A content manager 626 may additionally or alternatively cooperate with a content generator 628 to generate new content, e.g., updates to embeddings or feature vectors. The embeddings or feature vector may be generated for audio signals of one or more object classes for which one or more models or networks have been trained using a training module 630. In at least one embodiment, this content generator 628 may receive audio inputs and perform aspects of location-based neural audio processing. At least a portion of the generated embeddings or feature vector—which may correspond to synthesized data (e.g., simulated audio data) useful in generating such embeddings or feature vectors—may be transmitted to the client device 602 using a suitable transmission manager 622 for delivery via download, streaming, or other such transmission channel.

[0073] An encoder may be used to encode and / or compress at least a portion of this data prior to transmission to the client device 602. In at least one embodiment, the client device 602 receiving this content may pass this content to a corresponding control application 604, which may also or alternatively include a graphical user interface (GUI) 614, a content manager 610, and a content generator 612 to select, deliver, synthesize, render, modify, or use content for presentation (or other purposes) on or by the client device 602. A decoder may also be used to decode data received over the network 640 for presentation via the client device 602, e.g., audio or video content via a display 606 and audio data, e.g., sounds and music, via at least one audio playback device 608, e.g.,Speakers or headphones.

[0074] In at least one embodiment, at least a portion of this content may already be stored on, played back on, or accessible by the client device 602, such that transfer over the network 640 is not required for at least that portion of the content, e.g., if that content was previously downloaded or stored locally on a hard drive or optical disk. In at least one embodiment, a transfer mechanism such as data streaming may be used to transfer this content from the server 620 or user database 636 to the client device 602. In at least one embodiment, at least a portion of this content may be obtained, enhanced, and / or streamed from another source, such as a third-party service 660 or another client device 650, which may also include a content application 662 for creating, enhancing, or delivering content.In at least one embodiment, portions of this functionality may be performed using multiple computing devices or multiple processors within one or more computing devices, which may include, for example, a combination of CPUs and GPUs.

[0075] In this example, these client devices may include any suitable computing device, such as a desktop computer, a notebook computer, a set-top box, a streaming device, a game console, a smartphone, a tablet computer, a VR headset, AR glasses, a wearable computer, or a smart TV. Each client device may submit a request over at least one wired or wireless network, which may include, but is not limited to, the Internet, an Ethernet network, a local area network (LAN), or a cellular network. In this example, these requests may be delivered to an address associated with a cloud provider that operates or controls one or more electronic resources in a cloud provider environment that may include, for example, a data center or server farm.In at least one embodiment, the request may be received or processed by at least one edge server located at a network edge and outside of at least one security layer associated with the cloud provider's environment. In this way, latency may be reduced by allowing client devices to interact with servers located in closer proximity while simultaneously improving the security of resources in the cloud provider's environment.

[0076] In at least one embodiment, such a system may be used to perform graphical rendering operations. In other embodiments, such a system may be used for other purposes, e.g., to provide image or video content to test or validate autonomous machine applications, or to perform deep learning operations. In at least one embodiment, such a system may be implemented with an edge device or may include one or more VMs. In at least one embodiment, such a system may be implemented at least partially in a data center or at least partially using cloud computing resources. INFERENCE AND TRAINING LOGIC

[0077] Fig. 7A illustrates the inference and / or training logic 715 used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are described below in connection with Fig. 7A and / or 7B.

[0078] In at least one embodiment, the inference and / or training logic 715 may include, without limitation, code and / or data storage 701 to store feedforward and / or output weights and / or input / output data and / or other parameters to configure neurons or layers of a neural network that is trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 701 to store graph code or other software that controls the timing and / or order in which weight and / or other parameter information is loaded to configure logic, including integer and / or floating-point units (collectively, arithmetic logic unit(s) (“ALU(s)”). In at least one embodiment, code, such as arithmetic logic unit(s), loads the weights and / or other parameters to the neural network.B. a graph code, based on a neural network architecture to which the code corresponds, stores weight or other parameter information into the processor ALU(s). In at least one embodiment, the code and / or data storage 701 stores weight parameters and / or input / output data of each layer of a neural network being trained or used in connection with one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of the code and / or data storage 701 may be included in other on-chip or off-chip data storage, including the L1, L2, or L3 cache or system memory of a processor.

[0079] In at least one embodiment, any portion of the coding memory and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 701 may be cache memory, dynamic randomly addressable memory ("DRAM"), static randomly addressable memory ("SRAM"), non-volatile memory (e.g., flash memory), or other memory.In at least one embodiment, the choice of whether the code and / or data memory 701 is internal or external to, for example, the processor, or comprises DRAM, SRAM, flash memory, or another memory type may depend on the available on-chip or off-chip memory, latency requirements of the training and / or inference functions performed, batch size of the data used in inference and / or training of a neural network, or a combination of these factors.

[0080] In at least one embodiment, the inference and / or training logic 715 may include, without limitation, a code and / or data storage 705 to store backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the code and / or data storage 705 stores weighting parameters and / or input / output data of each layer of a neural network trained using aspects of one or more embodiments or used in connection with one or more embodiments during the backpropagation of input / output data and / or weighting parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, training logic 715 may include or be coupled to code and / or data storage 705 for storing graph code or other software that controls the timing and / or order in which weight and / or other parameter information is loaded to configure logic, including integer and / or floating-point units (collectively, ALU(s)). In at least one embodiment, code, such as graph code, loads weight or other parameter information into the processor ALU(s) based on a neural network architecture to which the code conforms. In at least one embodiment, any portion of code and / or data storage 705 may be included in other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.In at least one embodiment, any portion of the coding memory and / or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the choice of whether the code and / or data storage 705 is internal or external to, for example, the processor, or comprises DRAM, SRAM, flash memory, or another memory type may depend on the available on-chip or off-chip memory, latency requirements of the training and / or inference functions performed, batch size of the data used in inference and / or training of a neural network, or a combination of these factors.

[0081] In at least one embodiment, code and / or data memory 701 and code and / or data memory 705 may be separate memory structures. In at least one embodiment, code and / or data memory 701 and code and / or data memory 705 may be the same memory structure. In at least one embodiment, code and / or data memory 701 and code and / or data memory 705 may be partially the same memory structure and partially separate memory structures. In at least one embodiment, any portion of code and / or data memory 701 and code and / or data memory 705 may be included in other on-chip or off-chip data storage, including the L1, L2, or L3 cache or system memory of a processor.

[0082] In at least one embodiment, the inference and / or training logic 715 may include, without limitation, one or more arithmetic logic units ("ALU(s)") 710, including integer and / or floating point units, to perform logical and / or mathematical operations based at least in part on or specified by training and / or inference code (e.g., graph code), the result of which may generate activations (e.g., output values ​​of layers or neurons within a neural network) stored in an activation memory 720 and which are functions of input / output and / or weight parameter data stored in the code and / or data memory 701 and / or code and / or data memory 705.In at least one embodiment, the activations stored in activation memory 720 are generated according to linear algebraic and / or matrix-based mathematics performed by ALU(s) 710 in response to execution instructions or other code, using weight values ​​stored in code and / or data memory 701 and / or code and / or data memory 705 as operands, along with other values ​​such as bias values, gradient information, momentum values, or other parameters or hyperparameters, some or all of which may be stored in code and / or data memory 701 or code and / or data memory 705 or other on-chip or off-chip memory.

[0083] In at least one embodiment, the ALU(s) 710 are included in one or more processors or in other hardware logic devices or circuits, while in another embodiment, the ALU(s) 710 may be external to a processor or other hardware logic device or circuitry that utilizes them (e.g., a co-processor). In at least one embodiment, the ALU(s) 710 may be included in the execution units of a processor or otherwise in a bank of ALU(s) accessible by the execution units of a processor, either within the same processor or distributed across different processors of different types (e.g., CPUs, GPUs, fixed function units, etc.).In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and enablement storage 720 may be located on the same processor or other hardware logic device or circuit, while in another embodiment, they may be located on different processors or other hardware logic devices or circuits, or in a combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of enablement storage 720 may be included in other on-chip or off-chip data storage, including the L1, L2, or L3 cache or system memory of a processor.Furthermore, the inference and / or training code may be stored along with other code accessible by a processor or other hardware logic or circuitry and retrieved and / or processed via a processor's fetch, decode, scheduling, execution, elimination, and / or other logic circuitry.

[0084] In at least one embodiment, the activation memory 720 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, the activation memory 720 may be located wholly or partially within or external to one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation memory 720 is internal or external, for example, to the processor, or comprises DRAM, SRAM, flash memory, or another type of memory may depend on the available on-chip or off-chip memory, latency requirements of the training and / or inference functions performed, batch size of the data used in the inference and / or training of a neural network, or a combination of these factors. In at least one embodiment, the Fig. 7A may be used in conjunction with an application-specific integrated circuit (“ASIC”), such as the Tensorflow® Processing Unit from Google, an Inference Processing Unit (“IPU”) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, the inference and / or training logic 715 illustrated in Fig. 7A may be used in conjunction with CPU hardware, GPU hardware, or other hardware such as field programmable gate arrays (“FPGAs”).

[0085] Fig. 7B illustrates the inference and / or training logic 715 according to at least one or more embodiments. In at least one embodiment, the inference and / or training logic 715 may include, without limitation, hardware logic in which computational resources are dedicated or otherwise used exclusively in connection with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the inference and / or training logic 715 may Fig. 7B may be used in conjunction with an ASIC, such as the Tensorflow® Processing Unit from Google, an IPU from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, the inference and / or training logic 715 illustrated in Fig. 7B may be used in conjunction with CPU hardware, GPU hardware, or other hardware such as FPGAs. In at least one embodiment, the inference and / or training logic 715 includes, without limitation, a code and / or data memory 701 and a code and / or data memory 705, which may be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. In at least one embodiment illustrated in Fig. 7B, each of the code and / or data memories 701 and code and / or data memories 705 is associated with a dedicated computing resource, such as computing hardware 702 and computing hardware 706, respectively. In at least one embodiment, each of the computing hardware 702 and computing hardware 706 includes one or more ALU(s) that perform mathematical functions, such as linear algebraic functions, only on information stored in the code and / or data memory 701 and code and / or data memory 705, respectively, and the result of which is stored in the activation memory 720.

[0086] In at least one embodiment, each of the code and / or data memories 701 and 705 and the corresponding computational hardware 702 and 706 correspond to different layers of a neural network, such that the resulting activation from a "memory / compute pair 701 / 702" of code and / or data memory 701 and computational hardware 702 is provided as input to the "memory / compute pair 705 / 706" of code and / or data memory 705 and computational hardware 706 to reflect the conceptual organization of a neural network. In at least one embodiment, each of the memory / compute pairs 701 / 702 and 705 / 706 may correspond to more than one neural network layer. In at least one embodiment, additional memory / compute pairs (not shown) may be included subsequent to or in parallel with memory / compute pairs 701 / 702 and 705 / 706 in inference and / or training logic 715. DATA CENTER

[0087] Fig. Figure 8 shows an example of a data center 800 in which at least one embodiment may be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0088] In at least one embodiment, as in Fig. 8, the data center infrastructure layer 810 may include a resource orchestrator 812, clustered computing resources 814, and node computing resources (“node CRs”) 816(1)-816(N), where “N” represents any positive integer. In at least one embodiment, the node CRs 816(1)-816(N) may include, but are not limited to, any number of CPUs or other processors (including accelerators, FPGAs, graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or hard disk drives), network input / output devices (“NW I / O”), network switches, VMs, power modules and cooling modules, etc. In at least one embodiment, one or more node CRs among the node CRs 816(1)-816(N) may be a server having one or more of the above-mentioned computing resources.

[0089] In at least one embodiment, the grouped computing resources 814 may include separate groupings of node CRs housed in one or more racks (not shown) or in many racks in data centers in different geographic locations (also not shown). Separate groupings of node CRs within the grouped computing resources 814 may include grouped compute, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs, including CPUs or processors, may be grouped within one or more racks to provide compute resources to support one or more workloads.In at least one embodiment, one or more racks may also contain any number of power modules, cooling modules, and network switches in any combination.

[0090] In at least one embodiment, resource orchestrator 812 may configure or otherwise control one or more node CRs 816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, resource orchestrator 812 may include a software design infrastructure ("SDI") management entity for data center 800. In at least one embodiment, resource orchestrator 812 may include hardware, software, or a combination thereof.

[0091] In at least one embodiment, as in Fig. 8, the framework layer 820 includes a job scheduler 822, a configuration manager 824, a resource manager 826, and a distributed file system 828. In at least one embodiment, the framework layer 820 may include a framework for supporting software 832 of the software layer 830 and / or one or more applications 842 of the application layer 840. In at least one embodiment, the software 832 or the one or more applications 842 may each include web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 820 may be some type of free and open source software web application framework such as, but not limited to, Apache Spark™ (hereinafter "Spark"), which may utilize a distributed file system 828 for processing large amounts of data (e.g., "Big Data").In at least one embodiment, the job scheduler 822 may include a Spark driver to facilitate the scheduling of workloads supported by various layers of the data center 800. In at least one embodiment, the configuration manager 824 may be capable of configuring various layers, such as the software layer 830 and the framework layer 820, including Spark and the distributed file system 828 to support processing large amounts of data. In at least one embodiment, the resource manager 826 may be capable of managing clustered or grouped computing resources allocated or to be allocated to support the distributed file system 828 and the job scheduler 822. In at least one embodiment, clustered or grouped computing resources may include a clustered computing resource 814 at the infrastructure layer 810 of the data center.In at least one embodiment, the resource manager 826 may be coordinated with the resource orchestrator 812 to manage these allocated or assigned computing resources.

[0092] In at least one embodiment, the software 832 included in software layer 830 may include software used by at least portions of node CRs 816(1)-816(N), clustered computing resources 814, and / or distributed file system 828 of framework layer 820. The one or more types of software may include, among others, web page searching software, email virus scanning software, database software, and streaming video content software.

[0093] In at least one embodiment, the applications 842 included in the application layer 840 may include one or more types of applications used by at least portions of the node CRs 816(1)-816(N), the clustered computing resources 814, and / or the distributed file system 828 of the framework layer 820. One or more types of applications may include, but are not limited to, any number of a genomic application, a cognitive computing application, and a machine learning application, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in connection with one or more embodiments.

[0094] In at least one embodiment, configuration manager 824, resource manager 826, and resource orchestrator 812 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically feasible manner. In at least one embodiment, self-modifying actions may relieve a data center operator of data center 800 from potentially making poor configuration decisions and potentially avoiding underutilized and / or malfunctioning portions of a data center.

[0095] In at least one embodiment, data center 800 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using software and computing resources described above with respect to data center 800.In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 800 by using weighting parameters calculated by one or more training techniques described herein.

[0096] In at least one embodiment, the data center may use CPUs, ASICs, GPUs, FPGAs, or other hardware to perform training and / or inference with the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as a service to enable users to train or infer information, such as image capture, speech capture, or other artificial intelligence services ("KIT").

[0097] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are described below in connection with Fig. 7A and / or 7B. In at least one embodiment, the inference and / or training logic 715 in the system of Fig. 8 for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0098] Such components can be used to perform location-based tasks on captured content when the physical location information is not available for one or more capture devices. COMPUTER SYSTEMS

[0099] Fig. 9 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip ("SOC"), or a combination thereof 900, formed with a processor that may include execution units for executing an instruction in accordance with at least one embodiment. In at least one embodiment, the computer system 900 may include, without limitation, a component, such as a processor 902, for employing execution units including logic for performing algorithms for processing data in accordance with the present disclosure, as in the embodiment described herein. In at least one embodiment, the computer system 900 may include processors, such asthe PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors available from Intel Corporation of Santa Clara, California, USA, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, and the like) may be used. In at least one embodiment, computer system 900 may run a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Washington, USA, although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0100] Embodiments may be used in other devices such as handheld devices and embedded applications. Some examples of handheld devices are cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, embedded applications may include a microcontroller, a digital signal processor ("DSP"), a SOC, network computers ("NetPCs"), set-top boxes, network hubs, wide-area network ("WAN") switches, or any other system capable of executing one or more instructions according to at least one embodiment.

[0101] In at least one embodiment, computer system 900 may include, without limitation, a processor 902, which may include, without limitation, one or more execution units 908 for training and / or inferring a machine learning model according to the techniques described herein. In at least one embodiment, computer system 900 is a single-processor desktop or server system, but in another embodiment, computer system 900 may be a multiprocessor system.In at least one embodiment, processor 902 may include, without limitation, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word computing (VLIW) microprocessor, a processor implementing a combination of instruction sets, or any other processing device, such as a DSP. In at least one embodiment, processor 902 may be connected to a processor bus 910 that may communicate data signals between processor 902 and other components in computer system 900.

[0102] In at least one embodiment, processor 902 may include, without limitation, an internal level 1 ("L1") cache 904. In at least one embodiment, processor 902 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, cache 904 may be located external to processor 902. Other embodiments may also include a combination of both internal and external caches, depending on the particular implementation and needs. In at least one embodiment, a register file 906 may contain various types of data in various registers, including, without limitation, integer registers, floating-point registers, status registers, and instruction pointer registers.

[0103] In at least one embodiment, one or more execution units 908, including, but not limited to, logic for performing integer and floating-point operations, are also located in processor 902. In at least one embodiment, processor 902 may also include microcode ("ucode") read-only memory ("ROM") that stores microcode for certain macroinstructions. In at least one embodiment, the one or more execution units 908 may include logic for processing a packed instruction set 909. In at least one embodiment, by including a packed instruction set 909 in an instruction set of a general-purpose processor 902, along with associated instruction execution circuitry, operations used by many multimedia applications may be performed using packed data in a general-purpose processor 902.In one or more embodiments, many multimedia applications may be accelerated and executed more efficiently by utilizing the full width of a processor data bus 910 to perform operations on packed data, which may eliminate the need to transfer smaller units of data across the processor data bus 910 to perform one or more operations on one data item at a time.

[0104] In at least one embodiment, the one or more execution units 908 may also be used in microcontrollers, embedded processors, graphics devices, DSPs, and other types of logic circuits. In at least one embodiment, the computer system 900 may include, without limitation, a memory 920. In at least one embodiment, the memory 920 may be implemented as a DRAM device, an SRAM device, a flash memory device, or another memory device. In at least one embodiment, the memory 920 may store instructions 919 and / or data 921 represented by data signals that may be executed by the processor 902.

[0105] In at least one embodiment, a system logic chip may be connected to the processor bus 910 and the memory 920. In at least one embodiment, the system logic chip may include, without limitation, a memory controller hub ("MCH") 916, and the processor 902 may communicate with the MCH 916 via the processor bus 910. In at least one embodiment, the MCH 916 may provide a high-bandwidth memory path 918 to the memory 920 for instruction and data storage and for the storage of graphics instructions, data, and textures. In at least one embodiment, the MCH 916 may route data signals between the processor 902, the memory 920, and other components in the computer system 900, and to bridge data signals between the processor bus 910, the memory 920, and a system I / O 922.In at least one embodiment, a system logic chip may provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 916 may be coupled to the memory 920 via a high-bandwidth memory path 918, and the graphics / video card 912 may be coupled to the MCH 916 via an Accelerated Graphics Port ("AGP") connection 914.

[0106] In at least one embodiment, computer system 900 may use system I / O 922, which is a proprietary hub interface bus, to connect MCH 916 to I / O controller hub ("ICH") 930. In at least one embodiment, ICH 930 may provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, a local I / O bus may include, without limitation, a high-speed I / O bus for connecting peripheral devices to memory 920, chipset, and processor 902.Examples may include, without limitation, an audio controller 929, a firmware hub ("Flash BIOS") 928, a wireless transceiver 926, a data storage 924, a legacy I / O controller 923 including a user input and one or more user input interfaces 925, a serial expansion port 927, such as the Universal Serial Bus ("USB"), and a network controller 934. The data storage 924 may include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage devices.

[0107] In at least one embodiment, Fig. 9 a system containing interconnected hardware devices or “chips”, while in other embodiments Fig. 9 may illustrate an exemplary SOC. In at least one embodiment, the devices may be interconnected using proprietary interconnects, standardized interconnects (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of computer system 900 are interconnected via Compute Express Link ("CXL") interconnects.

[0108] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are described below in connection with Fig. 7A and / or 7B. In at least one embodiment, the inference and / or training logic 715 in the system of Fig. 9 be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0109] Such components can be used to perform location-based tasks on captured content when the physical location information is not available for one or more capture devices.

[0110] Fig. 10 is a block diagram illustrating an electronic device 1000 for using a processor 1010 according to at least one embodiment. In at least one embodiment, the electronic device 1000 may be, for example and without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0111] In at least one embodiment, the electronic device 1000 may include, without limitation, a processor 1010 communicatively coupled to any number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 1010 is coupled via a bus or interface, such as a 1°C bus, a System Management Bus (“SMBus”), a Low Pin Count (LPC) bus, a Serial Peripheral Interface (“SPI”), a High Definition Audio (“HDA”) bus, a Serial Advance Technology Attachment (“SATA”) bus, a USB (versions 1, 2, 3), or a Universal Asynchronous Receiver / Transmitter (“UART”) bus. In at least one embodiment, Fig. 10 an electronic device 1000 containing interconnected hardware devices or “chips”, while in other embodiments Fig. 10 may illustrate an exemplary SOC. In at least one embodiment, the Fig. 10 may be interconnected with proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components are Fig. 10 connected via CXL connections.

[0112] In at least one embodiment, Fig. 10 may include a display 1024, a touchscreen 1025, a touchpad 1030, a near-field communications (“NFC”) unit 1045, a sensor hub 1040, a thermal sensor 1046, an Express Chipset (“EC”) 1035, a Trusted Platform Module (“TPM”) 1038, a BIOS / firmware / flash memory (“BIOS, FW Flash”) 1022, a DSP 1060, a drive 1020 such as a solid-state disk (“SSD”) or a hard disk (“HDD”), a wireless local area network unit (“WLAN”) 1050, a Bluetooth unit 1052, a wireless wide-area network unit (“WWAN”) 1056, a global positioning system (“GPS”) 1055, a camera (“USB 3.0 camera”) 1054, such as a USB 3.0 camera, and / or a Low Power Double Data Rate ("LPDDR") memory unit ("LPDDR3") 1015, such as one implemented in the LPDDR3 standard. These components may each be implemented in any suitable manner.

[0113] In at least one embodiment, other components may be communicatively coupled to processor 1010 via the above-mentioned components. In at least one embodiment, an accelerometer 1041, an ambient light sensor (ALS) 1042, a compass 1043, and a gyroscope 1044 may be communicatively coupled to sensor hub 1040. In at least one embodiment, thermal sensor 1039, a fan 1037, a keyboard 1036, and a touchpad 1030 may be communicatively coupled to EC 1035. In at least one embodiment, speakers 1063, headphones 1064, and microphone (“mic”) 1065 may be communicatively coupled to an audio unit (“audio codec and Class-D amplifier”) 1062, which in turn may be communicatively coupled to DSP 1060. In at least one embodiment, the audio unit 1062 may include, for example and without limitation, an audio encoder / decoder ("codec") and a Class D amplifier.In at least one embodiment, the SIM card ("SIM") 1057 may be communicatively coupled to the WWAN unit 1056. In at least one embodiment, components such as the WLAN unit 1050 and the Bluetooth unit 1052, as well as the WWAN unit 1056, may be implemented in a Next Generation Form Factor ("NGFF").

[0114] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are described below in connection with Fig. 7A and / or 7B. In at least one embodiment, the inference and / or training logic 715 in the system of Fig. 10 be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0115] Such components can be used to perform location-based tasks on captured content when the physical location information is not available for one or more capture devices.

[0116] Fig. 11 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1100 includes one or more processors 1102 and one or more graphics processors 1108, and may be a single-processor desktop system, a multiprocessor workstation system, or a server system with a large number of one or more processors 1102 or one or more processor cores 1107. In at least one embodiment, system 1100 is a processing platform integrated into an integrated circuit (SoC) for use in mobile, wearable, or embedded devices.

[0117] In at least one embodiment, system 1100 may include or be integrated with a server-based gaming platform, a gaming console, including a gaming and media console, a mobile gaming console, a handheld gaming console, or an online gaming console. In at least one embodiment, system 1100 is a mobile phone, a smartphone, a tablet computing device, or a mobile internet device. In at least one embodiment, processing system 1100 may also include, be coupled to, or integrated with a wearable device, such as a wearable device with a smart watch, smart glasses, an AR device, or a VR device.In at least one embodiment, processing system 1100 is a television device or set-top box having one or more processors 1102 and a graphical interface generated by one or more graphics processors 1108.

[0118] In at least one embodiment, one or more processors 1102 each include one or more processor cores 1107 for processing instructions that, when executed, perform operations for system and application software. In at least one embodiment, each of one or more processor cores 1107 is configured to process a particular instruction set 1109. In at least one embodiment, the instruction set 1109 may enable CISC, RISC, or processing via VLIW. In at least one embodiment, the one or more processor cores 1107 may each process a different instruction set 1109, which may include instructions to facilitate emulation of other instruction sets. In at least one embodiment, the one or more processor cores 1107 may also include other processing devices, e.g., a DSP.

[0119] In at least one embodiment, the one or more processors 1102 include a cache memory ("cache") 1104. In at least one embodiment, the one or more processors 1102 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory 1104 is shared by various components of the one or more processors 1102. In at least one embodiment, the one or more processors 1102 also utilizes an external cache (e.g., a Level 3 (L3) cache or Last Level Cache (LLC)) (not shown) that may be shared by the one or more processor cores 1107 using known cache coherence techniques.In at least one embodiment, register file 1106 is additionally included in the one or more processors 1102, which may include various types of registers for storing various data types (e.g., integer registers, floating-point registers, status registers, and an instruction pointer register). In at least one embodiment, register file 1106 may include general-purpose registers or other registers.

[0120] In at least one embodiment, one or more processors 1102 are coupled to one or more interface buses 1110 for communicating communication signals, such as address, data, or control signals, between one or more processors 1102 and other components in system 1100. In at least one embodiment, the interface bus(es) 1110 may be a processor bus, such as a version of a Direct Media Interface ("DMI") bus. In at least one embodiment, the one or more interface buses 1110 are not limited to a DMI bus and may include one or more Peripheral Component Interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, the one or more processors 1102 include an integrated memory controller 1116 and a Platform Controller Hub ("PCH") 1130.In at least one embodiment, memory controller 1116 facilitates communication between a memory device 1120 and other components of system 1100, while PCH 1130 provides connections to I / O devices via a local I / O bus.

[0121] In at least one embodiment, memory device 1120 may be a DRAM device, an SRAM device, a flash memory device, a phase-change memory device, or other memory device with suitable performance to serve as process memory. In at least one embodiment, memory device 1120 may operate as system memory for processing system 1100 to store data 1122 and instructions 1121 for use when one or more processors 1102 are executing an application or process. In at least one embodiment, memory controller 1116 is also coupled to an optional external graphics processor 1112 that can communicate with one or more graphics processors 1108 in the one or more processors 1102 to perform graphics and media operations.In at least one embodiment, a display device 1111 may be connected to the one or more processors 1102. In at least one embodiment, a display device 1111 may include one or more of an internal display device, such as in a mobile electronic device or a laptop device, or an external display device connected via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1111 may include a head-mounted display ("HMD"), such as a stereoscopic display device for use in VR applications or AR applications.

[0122] In at least one embodiment, the PCH 1130 enables the connection of peripheral devices to the memory device 1120 and the one or more processors 1102 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, among others, an audio controller 1146, a network controller 1134, a firmware interface 1128, a wireless transceiver 1126, touch sensors 1125, and a data storage device 1124 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 1124 can be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a Peripheral Component Interconnect Bus (e.g., PCI, PCI Express).In at least one embodiment, touch sensors 1125 may include touchscreen sensors, pressure sensors, or fingerprint sensors. In at least one embodiment, wireless transceiver 1126 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a cellular network transceiver such as a 3G, 4G, or Long Term Evolution ("LTE") transceiver. In at least one embodiment, firmware interface 1128 allows communication with system firmware and may be, for example, a Unified Extensible Firmware Interface ("UEFI"). In at least one embodiment, network controller 1134 may allow network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the one or more interface buses 1110.In at least one embodiment, audio controller 1146 is a multi-channel high-definition audio controller. In at least one embodiment, system 1100 includes an optional legacy I / O controller 1140 for coupling legacy devices (e.g., Personal System 2 ("PS / 2")) to the system. In at least one embodiment, PCH 1130 may also be connected to one or more USB controllers 1142 that connect input devices such as keyboard and mouse combinations 1143, a camera 1144, or other USB input devices.

[0123] In at least one embodiment, an instance of memory controller 1116 and PCH 1130 may be integrated into a discrete external graphics processor, such as external graphics processor 1112. In at least one embodiment, PCH 1130 and / or memory controller 1116 may be external to one or more processors 1102. For example, in at least one embodiment, system 1100 may include an external memory controller 1116 and PCH 1130 configured as an MCH and peripheral controller hub within a system chipset that communicates with the one or more processors 1102.

[0124] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are described below in connection with Fig. 7A and / or 7B. In at least one embodiment, parts or all of the inference and / or training logic 715 may be integrated into the graphics processor 1500. For example, in at least one embodiment, the training and / or inference techniques described herein may utilize one or more ALUs included in a graphics processor. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed with a different logic than that described in Fig. 7A and / or 7B. In at least one embodiment, weighting parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALU(s) of a graphics processor to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0125] Such components can be used to perform location-based tasks on captured content when the physical location information is not available for one or more capture devices.

[0126] Fig. 12 is a block diagram of a processor 1200 having one or more processor cores 1202A-1202N, an integrated memory controller 1214, and an integrated graphics processor 1208, according to at least one embodiment. In at least one embodiment, the processor 1200 may include additional cores up to and including the additional core 1202N, represented by dashed, lined boxes. In at least one embodiment, each of the processor cores 1202A-1202N includes one or more internal cache units 1204A-1204N. In at least one embodiment, each processor core also has access to one or more shared cache units 1206.

[0127] In at least one embodiment, the one or more internal cache units 1204A-1204N and the one or more shared cache units 1206 represent a cache memory hierarchy within the processor 1200. In at least one embodiment, one or more cache memory units 1204A-1204N may include at least one level of instruction and data cache within each processor core and one or more levels of shared mid-level cache, such as a Level 2 (L2), Level 3 (L3), Level 4 (L4), or other levels of cache, with a highest level of cache prior to external memory being classified as LLC. In at least one embodiment, the cache coherence logic maintains coherence between the various cache units 1206 and 1204A-1204N.

[0128] In at least one embodiment, the processor 1200 may also include a set of one or more bus controller units 1216 and a system agent core 1210. In at least one embodiment, one or more bus controller units 1216 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1210 provides management functions for various processor components. In at least one embodiment, the system agent core 1210 includes one or more integrated memory controllers 1214 to manage access to various external memory devices (not shown).

[0129] In at least one embodiment, one or more of the processor cores 1202A-1202N include support for concurrent multithreading. In at least one embodiment, the system agent core 1210 includes components for coordinating the one or more processor cores 1202A-1202N during multithreaded processing. In at least one embodiment, the system agent core 1210 may additionally include a power control unit ("PCU") that includes logic and components for regulating one or more power states of one or more processor cores 1202A-1202N and the graphics processor 1208.

[0130] In at least one embodiment, processor 1200 additionally includes graphics processor 1208 for performing graphics processing operations. In at least one embodiment, graphics processor 1208 is coupled to shared cache units 1206 and a system agent core 1210 that includes one or more integrated memory controllers 1214. In at least one embodiment, system agent core 1210 also includes a display controller 1211 for controlling the output of the graphics processor to one or more coupled displays. In at least one embodiment, display controller 1211 may also be a separate module connected to graphics processor 1208 via at least one interconnect, or it may be integrated into graphics processor 1208.

[0131] In at least one embodiment, a ring-based interconnect 1212 is used to connect internal components of processor 1200. In at least one embodiment, an alternative interconnect may be used, such as a point-to-point connection, a switched connection, or other techniques. In at least one embodiment, graphics processor 1208 is connected to ring-based interconnect 1212 via an I / O connection 1213.

[0132] In at least one embodiment, I / O connection 1213 represents at least one of several types of I / O connections, including an on-package I / O connection that facilitates communication between various processor components and an embedded memory module 1218, such as an eDRAM module. In at least one embodiment, each of the processor cores 1202A-1202N and the graphics processor 1208 use the embedded memory module 1218 as a shared last-level cache.

[0133] In at least one embodiment, the one or more processor cores 1202A-1202N are homogeneous cores executing a common instruction set architecture. In at least one embodiment, the one or more processor cores 1202A-1202N are heterogeneous with respect to instruction set architecture (“ISA”), where one or more of the processor cores 1202A-1202N execute a common instruction set, while one or more other cores of the processor cores 1202A-1202N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, the one or more processor cores 1202A-1202N are microarchitecturally heterogeneous, where one or more relatively higher power cores couple with one or more lower power cores.In at least one embodiment, the processor 1200 may be implemented on one or more chips or as an integrated SOC circuit.

[0134] The inference and / or training logic 715 is used to perform inference and / or training operations associated with one or more embodiments. Details of the inference and / or training logic 715 are described below in connection with Fig. 7A and / or 7B. In at least one embodiment, parts or all of the inference and / or training logic 715 may be integrated into the processor 1200. In at least one embodiment, the training and / or inference techniques described herein may, for example, utilize one or more of the ALUs included in the graphics processor 1208, the graphics cores 1202A-1202N, or other components in Fig. 12. Furthermore, in at least one embodiment, the inference and / or training operations described herein may be performed with a method other than that described in Fig. 7A and / or 7B. In at least one embodiment, weighting parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) that configure the ALU(s) of a graphics processor 1200 to perform one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0135] Such components can be used to perform location-based tasks on captured content when the physical location information is not available for one or more capture devices. VIRTUALIZED COMPUTING PLATFORM

[0136] Fig. 13 is an example dataflow diagram for a process 1300 of generating and deploying a data processing and inference pipeline, according to at least one embodiment. In at least one embodiment, the process 1300 may be configured for use with imaging devices, processing devices, and / or other device types in one or more facilities 130. The process 1300 may be executed in a training system 1304 and / or a deployment system 1306. In at least one embodiment, the training system 1304 may be used to perform the training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use in the deployment system 1306.In at least one embodiment, the delivery system 1306 may be configured to offload processing and computing resources in a distributed computing environment to reduce the infrastructure requirements of the one or more devices 1302. In at least one embodiment, one or more applications in a pipeline may utilize or invoke services (e.g., inference, visualization, computation, AI, etc.) of the delivery system 1306 during application execution.

[0137] In at least one embodiment, some of the applications used in advanced processing and inference pipelines may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, machine learning models may be trained in the one or more facilities 1302 using data 1308 (such as imaging data) generated in the one or more facilities 1302 (and stored on one or more PACS servers in the one or more facilities 1302), they may be trained using imaging or sequencing data 1308 from one or more other facilities 1302, or a combination thereof.In at least one embodiment, the training system 1304 may be used to provide applications, services, and / or other resources for generating working, deployable machine learning models to the deployment system 1306.

[0138] In at least one embodiment, the model registry 1324 may be supported by an object store that may support versioning and object metadata. In at least one embodiment, the object store may be accessible from a cloud platform, for example, via a cloud storage compatible application programming interface (“APT”). In at least one embodiment, machine learning models in the model registry 1324 may be uploaded, listed, modified, or deleted by developers or partners of a system that interacts with an API. In at least one embodiment, an API may provide access to methods that enable users with appropriate credentials to associate models with applications so that models may be executed as part of the execution of containerized instantiations of applications.

[0139] In at least one embodiment, the training pipeline 1304 ( Fig. 13) include a scenario in which the one or more devices 1302 are training their own machine learning model or have an existing machine learning model that needs to be optimized or updated. In at least one embodiment, image data 1308 generated by imaging devices, sequencing devices, and / or other types of devices may be received. In at least one embodiment, after receiving the image data 1308, the AI-assisted annotation 1310 may be used to assist in generating annotations corresponding to the image data 1308 to be used as ground truth data for a machine learning model. In at least one embodiment, the AI-assisted annotation 1310 may include one or more machine learning models (e.g., convolutional neural networks ("CNNs")) that can be trained to generate annotations on particular types of image data 1308 (e.g.,of certain devices). In at least one embodiment, the AI-assisted annotation 1310 may then be used directly or adapted or fine-tuned using an annotation tool to generate ground truth data. In at least one embodiment, AI-assisted annotations 1310, labeled data 1312, or a combination thereof may be used as ground truth data for training a machine learning model. In at least one embodiment, a trained machine learning model may be referred to as output model(s) 1316 and may be used by the delivery system 1306 as described herein.

[0140] In at least one embodiment, a training pipeline may include a scenario in which the one or more devices 1302 require a machine learning model for use in performing one or more processing tasks for one or more applications in the delivery system 1306, but the one or more devices 1302 may not currently have such a machine learning model (or may not have had a model optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model may be selected from a model registry 1324. In at least one embodiment, the model registry 1324 may include machine learning models trained to perform a variety of different inferences from image data.In at least one embodiment, the machine learning models in the model registry 1324 may have been trained on imaging data from facilities other than the one or more facilities 1302 (e.g., facilities located at other locations). In at least one embodiment, machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, if trained on imaging data from a particular location, the training may take place at that location or at least in a manner that protects the confidentiality of the imaging data or restricts the transfer of imaging data off-site. In at least one embodiment, once a machine learning model has been trained—or partially trained—at a location, it may be added to the model registry 1324.In at least one embodiment, a machine learning model may then be retrained or updated at any number of other facilities, and a retrained or updated model may be made available in the model registry 1324. In at least one embodiment, a machine learning model may then be selected from the model registry 1324—and referred to as one or more output models 1316—and used in the delivery system 1306 to perform one or more processing tasks for one or more applications of a delivery system.

[0141] In at least one embodiment, a scenario may include the one or more devices 1302 requiring a machine learning model to perform one or more processing tasks for one or more applications in the provisioning system 1306, but the one or more devices 1302 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for those purposes).In at least one embodiment, a machine learning model selected from the model registry 1324 may not be tuned or optimized for the imaging data 1308 generated at the one or more devices 1302 due to differences in populations, the robustness of the training data used to train a machine learning model, the variety of anomalies in the training data, and / or other issues with the training data. In at least one embodiment, AI-assisted annotation 1310 may be used to assist in creating annotations corresponding to the imaging data 1308 to be used as ground truth data for retraining or updating a machine learning model. In at least one embodiment, labeled data 1312 may be used as base data for training a machine learning model.In at least one embodiment, the new training or update of a machine learning model may be referred to as model training 1314. In at least one embodiment, the model training 1314 (e.g., AI-assisted annotations 1310, labeled data 1312, or a combination thereof) may be used as ground truth data for the new training or update of a machine learning model. In at least one embodiment, a trained machine learning model may be referred to as output model(s) 1316 and may be used by the provisioning system 1306 as described herein.

[0142] In at least one embodiment, the provisioning system 1306 may include software 1318, services 1320, hardware 1322, and / or other components, features, and functions. In at least one embodiment, the provisioning system 1306 may include a software "stack" such that the software 1318 is built on top of the services 1320 and may utilize the services 1320 to perform some or all of the processing tasks, and the services 1320 and software 1318 may be built on top of the hardware 1322 and utilize the hardware 1322 to perform processing, storage, and / or other computational tasks of the provisioning system 1306. In at least one embodiment, the software 1318 may include any number of different containers, where each container may execute an instantiation of an application.In at least one embodiment, each application may perform one or more processing tasks in an advanced processing and inference pipeline (e.g., inference, audio acquisition, feature detection, segmentation, audio enhancement, calibration, etc.). In at least one embodiment, an advanced processing and inference pipeline may be defined based on the selection of various containers desired or required for processing image data 1308, in addition to containers that receive and configure image data for use by the individual containers and / or for use by the one or more devices 1302 after processing by a pipeline (e.g., to convert outputs back into a usable data type). In at least one embodiment, a combination of containers within the software 1318 (e.g.,form a pipeline) may be referred to as a virtual instrument (as further described herein), and a virtual instrument may utilize services 1320 and hardware 1322 to perform some or all of the processing tasks of applications instantiated in containers.

[0143] In at least one embodiment, a data processing pipeline may receive input data (e.g., image data 1308) in a particular format in response to an inference request (e.g., a request from a user of the provisioning system 1306). In at least one embodiment, the input data may represent one or more audio, video, and / or other data representations generated by one or more devices. In at least one embodiment, the data may be preprocessed as part of the data processing pipeline to prepare the data for processing by one or more applications.In at least one embodiment, post-processing may be performed on an output of one or more inference tasks or other processing tasks of a pipeline to prepare output data for a subsequent application and / or to prepare output data for transmission and / or use by a user (e.g., in response to an inference query). In at least one embodiment, inference tasks may be performed by one or more machine learning models, e.g., trained or deployed neural networks, which may include the one or more output models 1316 of the training system 1304.

[0144] In at least one embodiment, the tasks of the data processing pipeline may be encapsulated in one or more containers, each representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications may be published in a private (e.g., restricted access) area of ​​a container registry (described in more detail herein), and trained or deployed models may be stored in the model registry 1324 and associated with one or more applications. In at least one embodiment, images of applications (e.g.,Container images) must be available in a container registry, and once a user has selected them from a container registry for deployment in a pipeline, an image can be used to create a container for an instantiation of an application for use by a user's system.

[0145] In at least one embodiment, developers (e.g., software developers, clinicians, physicians, etc.) may develop, publish, and store applications (e.g., as containers) for performing audio processing and / or inferencing on provided data. In at least one embodiment, the development, publishing, and / or storage may be performed using a software development kit (“SDK”) associated with a system (e.g., to ensure that a developed application and / or container is compatible with a system). In at least one embodiment, a developed application may be tested locally (e.g., at a first facility, with data from a first facility) using an SDK that may support at least some of the services 1320 as a system (e.g., the system 1200 of Fig. 12). Because DICOM objects can contain between one and hundreds of audio or other data types, and due to the diversity of the data, in at least one embodiment, a developer may be responsible for managing the extraction and preparation of the incoming data (e.g., defining constructs, incorporating preprocessing into an application, etc.). In at least one embodiment, after being validated by system 1300 (e.g., for accuracy), an application may be available in a container registry for selection and / or implementation by a user to perform one or more processing tasks on data at a user's device (e.g., a second device).

[0146] In at least one embodiment, developers can then deploy applications or containers over a network for access and use by users of a system (e.g., System 1300 of Fig. 13). In at least one embodiment, completed and validated applications or containers may be stored in a container registry, and associated machine learning models may be stored in the model registry 1324. In at least one embodiment, a requesting entity—making an inference or audio processing request—may search a container registry and / or model registry 1324 for an application, container, dataset, machine learning model, etc., select a desired combination of elements for inclusion in the data processing pipeline, and submit an image processing request.In at least one embodiment, a request may include input data (and, in some examples, associated patient data) required to execute a request and / or a selection of applications and / or machine learning models to be executed in processing a request. In at least one embodiment, a request may then be forwarded to one or more components of the delivery system 1306 (e.g., a cloud) to perform the processing of the data processing pipeline. In at least one embodiment, the processing by the delivery system 1306 may include referencing selected items (e.g., applications, containers, models, etc.) from a container registry and / or model registry 1324. In at least one embodiment, the results, once generated by a pipeline, may be returned to a user for reference (e.g.,for display in a viewing application suite running on a local workstation or terminal).

[0147] In at least one embodiment, services 1320 may be used to support the processing or execution of applications or containers in pipelines. In at least one embodiment, services 1320 may include compute services, AI services, visualization services, and / or other types of services. In at least one embodiment, services 1320 may provide functionality common to one or more applications in software 1318 so that the functionality may be abstracted into a service that can be invoked or consumed by the applications. In at least one embodiment, the functionality provided by services 1320 may run dynamically and more efficiently while scaling well by enabling applications to process data in parallel (e.g., via a parallel computing platform).In at least one embodiment, not every application that uses the same functionality offered by services 1320 needs to have its own instance of services 1320, but rather services 1320 may be shared between different applications. In at least one embodiment, services 1320 may include an inference server or inference engine that may be used to perform detection or segmentation tasks, as non-limiting examples. In at least one embodiment, a model training service may be included that provides the ability to train and / or retrain machine learning models. In at least one embodiment, a data augmentation service may also be included that provides GPU-accelerated data extraction (e.g., DICOM, RIS, CIS, RESTful, RPC, raw data, etc.), resizing, scaling, and / or other augmentations.In at least one embodiment, a visualization service may be used that can add image rendering effects—such as ray tracing, rasterization, noise reduction, sharpening, etc.—to add realism to two-dimensional ("2D") and / or 3D models. In at least one embodiment, virtual instrument services may be included that provide beamforming, segmentation, inferencing, imaging, and / or support for other applications within virtual instrument pipelines.

[0148] In at least one embodiment where services 1320 include an AI service (e.g., an inference service), one or more machine learning models may be executed by invoking (e.g., as an API call) an inference service (e.g., an inference server) to execute machine learning models or their processing as part of application execution. In at least one embodiment where another application includes one or more machine learning models for segmentation tasks, an application may invoke an inference service to execute machine learning models to perform one or more processing operations associated with the segmentation tasks.In at least one embodiment, software 1318 implementing an advanced processing and inference pipeline including a segmentation application and an anomaly detection application may be streamlined because each application may invoke the same inference service to perform one or more inference tasks.

[0149] In at least one embodiment, hardware 1322 may include GPUs, CPUs, graphics cards, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX Systems), a cloud platform, or a combination thereof. In at least one embodiment, various types of hardware 1322 may be used to provide efficient, expedient support for software 1318 and services 1320 in delivery system 1306. In at least one embodiment, GPU processing may be implemented for on-premises processing (e.g., at one or more facilities 1302), in an AI / deep learning system, in a cloud system, and / or in other processing components of delivery system 1306 to improve the efficiency, accuracy, and effectiveness of audio processing and generation.In at least one embodiment, the software 1318 and / or the services 1320 may be optimized for GPU processing for deep learning, machine learning, and / or high-performance computing, as non-limiting examples. In at least one embodiment, at least a portion of the computing environment of the provisioning system 1306 and / or the training system 1304 may be executed in a data center on one or more supercomputers or high-performance computers with GPU-optimized software (e.g., a combination of hardware and software from NVIDIA's DGX systems). In at least one embodiment, the hardware 1322 may include any number of GPUs that may be used to process data in parallel, as described herein.In at least one embodiment, the cloud platform may further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other compute tasks. In at least one embodiment, the cloud platform (e.g., NVIDIA's NGC) may be executed using one or more AI / deep learning supercomputers and / or GPU-optimized software (e.g., as on NVIDIA's DGX systems) as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform may integrate an application container clustering system or an orchestration system (e.g., KUBERNETES) across multiple GPUs to enable seamless scaling and load balancing.

[0150] Fig. 14 is a system diagram for an example system 1400 for generating and deploying an image distribution pipeline according to at least one embodiment. In at least one embodiment, the system 1400 may be used to implement the process 1300 of Fig. 13 and / or other processes that include advanced processing and inference pipelines. In at least one embodiment, system 1400 may include a training system 1304 and a deployment system 1306. In at least one embodiment, training system 1304 and deployment system 1306 may be implemented using software 1318, services 1320, and / or hardware 1322, as described herein.

[0151] In at least one embodiment, system 1400 (e.g., training system 1304 and / or provisioning system 1306) may be implemented in a cloud computing environment (e.g., using cloud 1426). In at least one embodiment, system 1400 may be implemented locally in a healthcare facility or as a combination of cloud and local computing resources. In at least one embodiment, access to the APIs in cloud 1426 may be restricted to authorized users through enacted security measures or protocols. In at least one embodiment, a security protocol may include web tokens that may be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and provided with appropriate authorization.In at least one embodiment, the APIs of virtual instruments (as described herein) or other instances of system 1400 may be restricted to a set of public IPs that have been verified or authorized for interaction.

[0152] In at least one embodiment, various components of system 1400 may communicate with each other over a variety of network types, including, but not limited to, LANs and / or WANs via wired and / or wireless communication protocols. In at least one embodiment, communication between devices and components of system 1400 (e.g., for sending inference queries, for receiving results of inference queries, etc.) may occur over data bus(es), wireless data protocols (Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0153] In at least one embodiment, the training system 1304 may execute one or more training pipelines 1404 similar to those described herein with respect to Fig. 13. In at least one embodiment, where one or more machine learning models are to be used in the one or more deployment pipelines 1410 by the deployment system 1306, the one or more training pipelines 1404 may be used to train or retrain one or more (e.g., pre-trained) models and / or implement one or more of the pre-trained models 1406 (e.g., without the need for retraining or updating). In at least one embodiment, one or more output models 1316 may be generated as a result of training the one or more training pipelines 1404. In at least one embodiment, the one or more training pipelines 1404 may include any number of processing steps, such as converting or interpreting image data (or other input data).In at least one embodiment, different training pipelines 1404 may be used for different machine learning models used by the provisioning system 1306. In at least one embodiment, one or more training pipelines 1404 similar to a first example described with reference to FIG. Fig. 13, one or more training pipelines 1404 corresponding to a second, with respect to Fig. 13 may be used for a second machine learning model, and one or more training pipelines 1404 corresponding to a third, with respect to Fig. 13 may be used for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 1304 may be used, depending on what is required for the particular machine learning model. In at least one embodiment, one or more of the machine learning models may already be trained and ready for use, such that the machine learning models do not receive any processing by the training system 1304 and may be deployed by the deployment system 1306.

[0154] In at least one embodiment, the one or more output models 1316 and / or the one or more pre-trained models 1406 may include any type of machine learning model, depending on the implementation or embodiment. In at least one embodiment, and without limitation, the machine learning models used by system 1400 may include machine learning models using linear regression, logistic regression, decision trees, support vector machines ("SVM"), naive Bayes, k-nearest neighbor ("knn"), k-means clustering, random forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., auto-encoders, convolutional, recurrent, perceptrons, long / short term memory ("LSTM"), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, etc.), and / or other types of machine learning models.

[0155] In at least one embodiment, the one or more training pipelines 1404 may include AI-assisted annotation as described herein with respect to at least Fig. 14. In at least one embodiment, labeled data 1312 (e.g., traditional annotations) may be generated by any number of techniques. In at least one embodiment, labels or other annotations may be generated in a drawing program (e.g., an annotation program), a CAD program, a labeling program, another type of program suitable for generating annotations or labels for the ground truth, and / or, in some examples, hand-drawn. In at least one embodiment, the ground truth data may be synthetically generated (e.g., from computer models or renderings), physically generated (e.g., designed and generated from real data), machine-generated (e.g., using feature analysis and learning to extract features from the data and then generate labels), human-labeled (e.g.,an annotation expert determines the position of the labels) and / or a combination thereof. In at least one embodiment, for each instance of image data 1308 (or other data types used by machine learning models), there may be corresponding ground truth data generated by the training system 1304. In at least one embodiment, the AI-assisted annotation 1310 may be performed as part of the deployment pipeline(s) 1410; either in addition to or instead of the AI-assisted annotation 1310 included in the one or more training pipelines 1404. In at least one embodiment, the system 1400 may include a multi-tiered platform including a software layer (e.g., software 1318) with diagnostic applications (or other application types) capable of performing one or more medical diagnostic functions.In at least one embodiment, system 1400 may be communicatively coupled (e.g., via encrypted connections) to PACS server networks of one or more institutions. In at least one embodiment, system 1400 may be configured to access and reference data from PACS servers to perform operations such as training machine learning models, deploying machine learning models, audio processing, inferencing, and / or other operations.

[0156] In at least one embodiment, a software layer may be implemented as a secure, encrypted, and / or authenticated API through which applications or containers may be invoked from one or more external environments (e.g., device(s) 1302). In at least one embodiment, applications may then invoke or execute one or more services 1320 to perform computational, AI, or visualization tasks associated with the respective applications, and the software 1318 and / or services 1320 may utilize the hardware 1322 to perform processing tasks effectively and efficiently. In at least one embodiment, communication sent to or received from a training system 1304 and a provisioning system 1306 may occur using a pair of DICOM adapters 1402A, 1402B.

[0157] In at least one embodiment, the provisioning system 1306 may execute the provisioning pipeline(s) 1410. In at least one embodiment, the one or more provisioning pipelines 1410 may include any number of applications that may be applied sequentially, non-sequentially, or otherwise to imaging data (and / or other data types) generated by imaging devices, sequencing devices, genomics devices, etc. -- including AI-assisted annotation as described above. In at least one embodiment described herein, the one or more provisioning pipelines 1410 for a single device may be referred to as a virtual instrument for a device (e.g., a virtual ultrasound instrument, a virtual scanning instrument, a virtual sequencing instrument, etc.).In at least one embodiment, there may be more than one delivery pipeline 1410 for a single device, depending on what information is desired from the data generated by a device. In at least one embodiment, where anomaly detection from an audio device is desired, there may be one or more first delivery pipelines 1410, and where audio enhancement from the output of an audio device is desired, there may be one or more second delivery pipelines 1410.

[0158] In at least one embodiment, an audio generation application may include a processing task that includes the use of a machine learning model. In at least one embodiment, a user may use their own machine learning model or select a machine learning model from the model registry 1324. In at least one embodiment, a user may implement their own machine learning model or select a machine learning model for inclusion in an application to perform a processing task. In at least one embodiment, applications may be selectable and customizable, and by defining constructs of applications, the deployment and implementation of applications for a particular user is presented as a more seamless user experience.In at least one embodiment, the one or more deployment pipelines 1410 may be even more user-friendly, enable easier integration, and deliver more accurate, efficient, and timely results by leveraging other features of the system 1400, such as the services 1320 and the hardware 1322.

[0159] In at least one embodiment, the provisioning system 1306 may include a user interface (“UT”) 1414 (e.g., a graphical user interface, a web interface, etc.) that may be used to select applications for inclusion in the one or more provisioning pipelines 1410, arrange applications, modify or change applications or parameters or constructs thereof, use and interact with the provisioning pipeline(s) 1410 during setup and / or deployment, and / or otherwise interact with the provisioning system 1306.In at least one embodiment not illustrated with respect to training system 1304, UI 1414 (or other user interface) may be used to select models for use in provisioning system 1306, to select models for training or new training in training system 1304, and / or to otherwise interact with training system 1304.

[0160] In at least one embodiment, the pipeline manager 1412 may be used in addition to an application orchestration system 1428 to manage the interaction between applications or containers of the deployment pipeline(s) 1410 and services 1320 and / or hardware 1322. In at least one embodiment, the pipeline manager 1412 may be configured to enable application-to-application, application-to-services 1320, and / or application or service-to-hardware 1322 interaction. In at least one embodiment, the pipeline manager 1412 is included in the services 1320, although it is illustrated as being included in the software 1318, but this is not meant to be limiting. In at least one embodiment, the application orchestration system 1428 (e.g., Kubernetes, DOCKER, etc.)) may include a container orchestration system that can aggregate applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the one or more deployment pipelines 1410 (e.g., a reconstruction application, a segmentation application, etc.) with individual containers, each application can be executed in a self-contained environment (e.g., at the kernel level) to increase speed and efficiency.

[0161] In at least one embodiment, each application and / or container (or an image thereof) may be individually developed, modified, and deployed (e.g., a first user or developer may develop, modify, and deploy a first application, and a second user or developer may develop, modify, and deploy a second application independently of a first user or developer), thereby allowing focus on a task of a single application and / or container without being hindered by tasks of another application or container. In at least one embodiment, communication and collaboration between different containers or applications may be supported by the pipeline manager 1412 and the application orchestration system 1428.In at least one embodiment, if an expected input and / or output of each container or application is known by a system (e.g., based on application or container constructs), the application orchestration system 1428 and / or the pipeline manager 1412 may facilitate communication between the individual applications or containers and resource sharing between them. In at least one embodiment, since one or more applications or containers in the one or more deployment pipelines 1410 may share the same services and resources, the application orchestration system 1428 may orchestrate, consume, and determine the sharing of services or resources between and among the various applications or containers.In at least one embodiment, a scheduler may be used to track the resource requirements of applications or containers, the current or planned usage of those resources, and the availability of resources. In at least one embodiment, a scheduler may thus allocate resources to different applications and distribute resources between and among applications with regard to the needs and availability of a system. In some examples, a scheduler (and / or another component of the application orchestration system 1428) may determine resource availability and distribution based on constraints imposed on a system (e.g., user constraints), such as quality of service (QoS), urgency of the need for data output (e.g., to determine whether processing should be performed in real time or with a delay), etc.

[0162] In at least one embodiment, the services 1320 used by applications or containers in the deployment system 1306 may include compute service(s) 1416, AI service(s) 1418, visualization service(s) 1420, and / or other service types. In at least one embodiment, applications may invoke (e.g., execute) one or more of the services 1320 to perform processing operations for an application. In at least one embodiment, the one or more compute services 1416 may be used by applications to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, the one or more compute services 1416 may be used to perform parallel processing (e.g., with a parallel computing platform 1430) to process data by one or more applications and / or one or more tasks of a single application substantially concurrently.In at least one embodiment, parallel computing platform 1430 (e.g., NVIDIA's CUDA) may enable general-purpose computing on GPUs ("GPGPU") (e.g., GPUs / graphics 1422). In at least one embodiment, a software layer of parallel computing platform 1430 may enable access to virtual instruction sets and parallel compute elements of GPUs to execute compute cores. In at least one embodiment, parallel computing platform 1430 may include memory, and in some embodiments, memory may be shared between multiple containers and / or between different processing tasks within a single container.In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or for multiple processes within a container to use the same data from a shared memory segment of the parallel computing platform 1430 (e.g., when multiple different stages of an application or multiple applications process the same information). In at least one embodiment, the same data may be used in the same memory location for any number of processing tasks (e.g., at the same time, at different times, etc.), rather than making a copy of the data and moving the data to different locations in memory (e.g., a read / write operation).In at least one embodiment, when data is used to generate new data as a result of processing, this information about a new data storage location can be stored and shared between different applications. In at least one embodiment, the storage location of data and the storage location of updated or modified data can be part of a definition of what constitutes a payload within containers.

[0163] In at least one embodiment, AI services 1418 may be utilized to perform inference services for executing machine learning models associated with applications (e.g., tasked with performing one or more processing tasks of an application). In at least one embodiment, the one or more AI services 1418 may utilize the AI ​​system 1424 to execute one or more machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, the applications of the one or more deployment pipelines 1410 may utilize one or more output models 1316 of the training system 1304 and / or other models of applications to draw inferences from image data.In at least one embodiment, two or more instances of inferencing using the application orchestration system 1428 (e.g., a scheduler) may be available. In at least one embodiment, a first category may include a high priority / low latency path that can achieve higher service level agreements, e.g., for performing inference on urgent requests during an emergency or for a radiologist during diagnosis. In at least one embodiment, a second category may include a default priority path that can be used for requests that are not urgent or where analysis can be performed at a later time. In at least one embodiment, the application orchestration system 1428 may distribute resources (e.g., services 1320 and / or hardware 1322) based on priority paths for different inference tasks of one or more AI services 1418.

[0164] In at least one embodiment, shared memory may be provided for the one or more AI services 1418 within system 1400. In at least one embodiment, the shared memory may act as a cache (or other type of memory device) and may be used to process inference requests from applications. In at least one embodiment, when an inference is submitted, it may be received by a number of API instances of provisioning system 1306, and one or more instances may be selected (e.g., for best fit, for load balancing, etc.) to process a request.In at least one embodiment, to service a request, a query may be entered into a database, a machine learning model may be found from the model registry 1324 if it is not already in a cache, a validation step may ensure that a suitable machine learning model is loaded into a cache (e.g., shared memory), and / or a copy of a model may be stored in a cache. In at least one embodiment, a scheduler (e.g., the pipeline manager 1412) may be used to launch an application referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, an inference server may be launched if no inference server has already been launched to execute a model.Any number of inference servers can be started per model. In at least one embodiment, models can be cached in a pull model where inference servers are clustered when load balancing is beneficial. In at least one embodiment, inference servers can be statically loaded into corresponding, distributed servers.

[0165] In at least one embodiment, inferencing may be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server may be associated with a model (and optionally, multiple versions of a model). In at least one embodiment, if an instance of an inference server does not exist, a new instance may be loaded when a request to perform inference on a model is received. In at least one embodiment, a model may be passed to an inference server when starting an inference server, so that the same container may be used to serve different models as long as the inference server runs as a different instance.

[0166] In at least one embodiment, during application execution, an inference request for a particular application may be received, and a container (e.g., hosting an instance of an inference server) may be loaded (if not already loaded), and a startup procedure may be invoked. In at least one embodiment, preprocessing logic in a container may load the incoming data, decode it, and / or perform additional preprocessing (e.g., using CPU(s) and / or GPU(s)). In at least one embodiment, once the data is prepared for inference, a container may perform the necessary inference on the data.In at least one embodiment, an application may summarize the results prior to completion, which may include, without limitation, a single confidence score, pixel-level segmentation, voxel-level segmentation, the creation of a visualization, or the creation of text summarizing the results. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have a real-time priority (TAT < 1 min), while others may have a lower priority (e.g., TAT < 10 min). In at least one embodiment, the model execution time may be measured from the requesting institution or entity and may include the time to traverse the partner network as well as execution in an inference service.

[0167] In at least one embodiment, the transfer of requests between services 1320 and the inference applications may be hidden behind an SDK, and robust transport may occur via a queue. In at least one embodiment, a request is queued via an API for an individual application / tenant ID combination, and an SDK pulls a request from a queue and passes a request to an application. In at least one embodiment, a queue name may be specified in an environment from which an SDK retrieves it. In at least one embodiment, asynchronous communication via a queue may be useful because it allows each instance of an application to begin work as soon as it is available. Results may be transferred back via a queue to ensure that no data is lost.In at least one embodiment, queues may also provide the ability to segment work, as the highest priority work may go to a queue with the most connected instances of an application, while the lowest priority work may go to a queue with a single connected instance that processes the tasks in the order they are received. In at least one embodiment, an application may run on a GPU-accelerated instance generated in the cloud 1426, and an inference service may perform inference on a GPU.

[0168] In at least one embodiment, visualization services 1420 may be used to generate visualizations for displaying the outputs of applications and / or the deployment pipeline(s) 1410. In at least one embodiment, GPUs / graphics 1422 may be used by one or more visualization services 1420 to generate visualizations. In at least one embodiment, rendering effects, such as ray tracing, may be implemented by one or more visualization services 1420 to generate higher quality visualizations. In at least one embodiment, the visualizations may include, without limitation, 2D image renderings, 3D volume renderings, 3D volume reconstructions, 2D tomography slices, VR displays, AR displays, etc. In at least one embodiment, virtualized environments may be used to create a virtual interactive display or environment (e.g.,a virtual environment) for interaction by users of a system (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, the one or more visualization services 1420 may include an internal visualizer, kinematics, and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0169] In at least one embodiment, hardware 1322 may include GPUs / graphics 1422, AI system 1424, cloud 1426, and / or any other hardware used to execute training system 1304 and / or deployment system 1306. In at least one embodiment, GPUs / graphics 1422 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that may be used to perform processing tasks of one or more of compute services 1416, one or more AI services 1418, one or more visualization services 1420, other services, and / or any features or functions of software 1318.With respect to the one or more AI services 1418, GPUs / graphics 1422 may be used, for example, to preprocess image data (or other data types used by machine learning models), postprocess machine learning model outputs, and / or perform inference (e.g., to run machine learning models). In at least one embodiment, the cloud 1426, the AI ​​system 1424, and / or other components of the system 1400 may use GPUs / graphics 1422. In at least one embodiment, the cloud 1426 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, the AI ​​system 1424 may use GPUs, and the cloud 1426—or at least a portion thereof tasked with deep learning or inferencing—may be executed using one or more AI systems 1424.Although the hardware 1322 is illustrated as discrete components, this is not to be construed as a limitation, and all components of the hardware 1322 may be combined with or utilized by other components of the hardware 1322.

[0170] In at least one embodiment, the AI ​​system 1424 may include a purpose-built computing system (e.g., a supercomputer or HPC) configured for inferencing, deep learning, machine learning, and / or other AI tasks. In at least one embodiment, the AI ​​system 1424 (e.g., NVIDIA's DGX systems) may include GPU-optimized software (e.g., a software stack) that can execute with a variety of GPUs / graphs 1422, in addition to CPUs, RAM, memory, and / or other components, features, or functions. In at least one embodiment, one or more AI systems 1424 may be implemented in the cloud 1426 (e.g., in a data center) to perform some or all of the AI-based processing tasks of the system 1400.

[0171] In at least one embodiment, cloud 1426 may include a GPU-accelerated infrastructure (e.g., NVIDIA's NGC) that provides a GPU-optimized platform for executing processing tasks of system 1400. In at least one embodiment, cloud 1426 may include one or more AI systems 1424 to execute one or more AI-based tasks of system 1400 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, cloud 1426 may be integrated with application orchestration system 1428, which utilizes multiple GPUs to enable seamless scaling and load balancing between and among applications and services 1320.In at least one embodiment, the cloud 1426 may be tasked with running at least some of the services 1320 of the system 1400, including the one or more compute services 1416, the one or more AI services 1418, and / or the one or more visualization services 1420, as described herein. In at least one embodiment, the cloud 1426 may perform small and large batch inference (e.g., NVIDIA's TENSOR RT), provide a parallel computing platform 1430 (e.g., NVIDIA's CUDA), run an application orchestration system 1428 (e.g., KUBERNETES), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher-quality cinematic movies), and / or provide other functionality for the system 1400.

[0172] Fig. 15A illustrates a dataflow diagram for a process 1500 for training, retraining, or updating a machine learning model according to at least one embodiment. In at least one embodiment, the process 1500 may be implemented, as a non-limiting example, with the system 1400 of Fig. 14. In at least one embodiment, process 1500 may utilize the services and / or hardware described herein. In at least one embodiment, the refined model 1512 generated by process 1500 may be executed by a deployment system for one or more containerized applications in deployment pipelines.

[0173] In at least one embodiment, model training 1514 may include retraining or updating an initial model 1504 (e.g., a pre-trained model) using new training data (e.g., new input data, such as the customer record 1506, and / or new truth data associated with the input data). In at least one embodiment, to retrain or update the initial model 1504, the one or more output or loss layers of the initial model 1504 may be reset, deleted, and / or replaced with one or more updated or new output or loss layers. In at least one embodiment, the initial model 1504 may already have fine-tuned parameters (e.g., weights and / or biases) left over from a previous training so that the training or retraining does not take as long or require as much computation as training a new model.In at least one embodiment, during model training 1514, by resetting or replacing the one or more seed or loss layers of the original seed model 1504, the parameters may be updated and retuned for a new dataset based on loss calculations associated with the accuracy of the one or more seed or loss layers in making predictions for a new customer dataset 1506.

[0174] In at least one embodiment, the one or more pre-trained models 1406 may be stored in a data store or registry. In at least one embodiment, the one or more pre-trained models 1406 may have been trained at least partially in one or more facilities other than the facility executing the process 1500. To protect the privacy and rights of patients, subjects, or customers of various facilities, in at least one embodiment, the one or more pre-trained models 1406 may have been trained on-site using on-site generated customer or patient data.In at least one embodiment, the one or more pre-trained models 1406 may be trained using a cloud and / or other hardware, but confidential, privacy-protected patient data may not be transmitted to, used by, or accessible by components of a cloud (or other off-premises hardware). In at least one embodiment where the one or more pre-trained models 1406 are trained with patient data from more than one facility, the one or more pre-trained models 1406 may have been individually trained for each facility before being trained with patient or customer data from another facility. In at least one embodiment, e.g., where customer or patient data has been released for privacy reasons (e.g., through a waiver, for experimental purposes, etc.), or if customer or patient data is contained in a public dataset, customer or patient data from any number of facilities may be used to train pre-trained models 1406 on-site and / or off-site, for example, in a data center or other cloud computing infrastructure.

[0175] In at least one embodiment, when selecting applications for use in deployment pipelines, a user may also select machine learning models to use for specific applications. In at least one embodiment, a user may not have a model to use, so a user may select one or more pre-trained models 1406 for use with an application. In at least one embodiment, the pre-trained model may not be optimized to produce accurate results for a user's facility's customer record 1506 (e.g., based on patient diversity, demographics, type of medical devices used, etc.).In at least one embodiment, prior to deploying a pre-trained model in a deployment pipeline for use with one or more applications, the one or more pre-trained models 1406 may be updated, retrained, and / or fine-tuned for use in a corresponding device.

[0176] In at least one embodiment, a user may select the one or more pre-trained models 1406 to be updated, retrained, and / or fine-tuned, and this pre-trained model may be referred to as an initial model 1504 for a training system in process 1500. In at least one embodiment, a customer data set 1506 (e.g., imaging data, genomics data, sequencing data, or other data types generated by devices in a facility) may be used to perform model training (which may include, without limitation, transfer learning) on ​​the initial model 1504 to generate a refined model 1512. In at least one embodiment, ground truth data corresponding to the customer data set 1506 may be generated by the model training system 1504.In at least one embodiment, the ground truth data may be generated at least in part by clinicians, scientists, physicians, practitioners in an institution.

[0177] In at least one embodiment, in some examples, AI-assisted annotation 1310 may be used to generate ground truth data. In at least one embodiment, AI-assisted annotation 1310 (e.g., implemented with an AI-assisted annotation SDK) may utilize machine learning models (e.g., neural networks) to generate suggested or predicted ground truth data for a customer record. In at least one embodiment, a user may use annotation tools within a user interface (GUI) on a computing device.

[0178] In at least one embodiment, user 1510 may interact with a graphical user interface via computing device 1508 to edit or fine-tune (auto-)annotations. In at least one embodiment, a polygon editing function may be used to move vertices of a polygon to more precise or finer positions.

[0179] In at least one embodiment, once ground truth data is associated with the customer record 1506, ground truth data (e.g., from AI-assisted annotation 1310, manual labeling, etc.) may be used during model training to generate a refined model 1512. In at least one embodiment, the customer record 1506 may be applied to the initial model 1504 any number of times, and the ground truth data may be used to update the parameters of the initial model 1504 until an acceptable level of accuracy is achieved for the refined model 1512. In at least one embodiment, the refined model 1512, once generated, may be deployed in one or more deployment pipelines in a device for performing one or more processing tasks with respect to medical data.

[0180] In at least one embodiment, the refined model 1512 may be uploaded to the one or more pre-trained models 1542 in a model registry for selection by another facility. In at least one embodiment, this process may be performed at any number of facilities, allowing the refined model 1512 to be further refined on new datasets any number of times to produce a more universal model.

[0181] Fig. 15B is an exemplary illustration of a client-server architecture 1532 for enhancing annotation tools with pre-trained models 1542, according to at least one embodiment. In at least one embodiment, the AI-assisted annotation tool 1536 may be instantiated based on a client-server architecture 1532. In at least one embodiment, AI-assisted annotation tools 1536 may assist in imaging applications, for example, in identifying anomalies. In at least one embodiment, imaging applications may include software tools that assist the user 1510 in identifying, as a non-limiting example, a few extreme points on a particular organ of interest in the raw audio data 1534 and obtaining automatically annotated results.In at least one embodiment, the results may be stored in a data store as training data 1538 and used (for example, and without limitation) as ground truth data for training. In at least one embodiment, when computing device 1508 sends extreme points for AI-assisted annotation, a deep learning model may receive this data as input and return inference results of a segmented organ or abnormality. In at least one embodiment, pre-instantiated annotation tools, such as AI-assisted annotation tool 1536 in FIG. Fig.15B, through API calls (e.g., API call 1544) to a server, such as an annotation assistant server 1540, which may contain a set of pre-trained models 1542 stored, for example, in an annotation model registry. In at least one embodiment, an annotation model registry may store one or more pre-trained models 1542 (e.g., machine learning models such as deep learning models) pre-trained to perform the AI-assisted annotation 1310 for a particular organ or anomaly. These models may be further updated using training pipelines. In at least one embodiment, the pre-installed annotation tools may be improved over time as new labeled data is added.

[0182] Different embodiments can be described by the following clauses: Clause 1. Computer-implemented procedure comprising: Extracting a plurality of audio features from a plurality of audio signals recorded using a plurality of microphones; Processing the plurality of audio features using a neural network to generate a feature vector representing a spatial location of an audio source with respect to the plurality of microphones; and Providing the feature vector to a task-specific model to perform at least one audio-related task corresponding to the plurality of audio signals based at least on the spatial location. Clause 2. The computer-implemented method of Clause 1, wherein the audio-related task includes at least one of the following: extraction of speech corresponding to the spatial location, echo cancellation, enhancement of audio signals received from the spatial location associated with the audio source, or suppression of audio signals received from locations other than the spatial location associated with the audio source. Clause 3. A computer-implemented method according to clause 1 or clause 2, wherein the plurality of audio features represent differences in the plurality of audio signals corresponding to at least one of the real or imaginary parts of a complex audio spectrum, signal magnitude, signal phase, level and arrival time, or direct sound to reverberation ratio. Clause 4. The computer-implemented method of any preceding clause, wherein the feature vector includes at least a minimum number of elements, and wherein none of the elements need relate to a physical direction or physical location with respect to one or more of the plurality of microphones. Clause 5. A computer-implemented method according to any of the preceding clauses, further comprising: Combining the plurality of audio signals into a single audio signal before performing the at least one audio-related task. Clause 6. A computer-implemented method according to any one of the preceding clauses, wherein the feature vector represents a location-based embedding of the audio source in a latent space. Clause 7. A computer-implemented method according to any preceding clause, wherein the plurality of audio features comprises at least one of cross-channel features, temporal features, or spectral features. Clause 8. A computer-implemented method according to any of the preceding clauses, further comprising: Analyzing audio data indicated as being generated by the sound source to identify audio data from the sound source represented in the plurality of audio signals. Clause 9. A computer-implemented method according to any preceding clause, wherein the processing of the plurality of audio features is performed independently of the availability of information indicative of a physical configuration of the plurality of microphones. Clause 10. Processor, which includes: one or more circuits for: Extracting a plurality of audio features from a plurality of audio signals recorded using a plurality of microphones, Processing the plurality of audio features using a neural network to generate a feature vector representing a spatial location of an audio source with respect to the plurality of microphones; and Providing the feature vector to a task-specific model to perform at least one audio-related task based at least in part on the spatial location. Clause 11. The processor of Clause 10, wherein the audio-related task includes at least one of the following: extraction of speech corresponding to the spatial location, echo cancellation, enhancement of audio signals received from the spatial location associated with the audio source, or suppression of audio signals received from locations other than the spatial location associated with the audio source. Clause 12. A processor according to Clause 10 or Clause 11, wherein the plurality of audio characteristics represent differences in the plurality of audio signals corresponding to at least one of the real or imaginary parts of a complex audio spectrum, signal magnitude, signal phase, level and arrival time, or direct sound to reverberation ratio. Clause 13. The processor of any of clauses 10-12, wherein the feature vector includes at least a minimum number of elements, and wherein none of the elements need relate to a physical direction or location with respect to one or more of the plurality of microphones. Clause 14. The processor of any of clauses 10-13, wherein the feature vector represents a location-based embedding of the audio source in a latent space. Clause 15. A processor according to any one of clauses 10-14, wherein the plurality of audio features includes at least one of cross-channel features, temporal features, or spectral features. Clause 16. A processor according to any of Clauses 10-15, wherein the processor comprises at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed reality (MR) content; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system for performing hardware tests using simulations; a system for performing generative content operations using a language model; a system for generating synthetic data; a system for performing generative AI operations using a large language model (LLM); a collaborative content creation platform for 3D assets; or a system implemented at least in part using cloud computing resources. Clause 17. System comprising: one or more processors to provide a feature vector to a task-specific model to perform at least one audio-related task, wherein the feature vector is generated from a plurality of audio features using a neural network and represents a spatial location of an audio source with respect to the plurality of microphones, wherein the plurality of audio features are derived from a plurality of audio signals acquired using the plurality of microphones. Clause 18. The system of Clause 17, wherein the audio-related task includes at least one of the following: extraction of speech corresponding to the spatial location, echo cancellation, enhancement of audio signals received from the spatial location associated with the audio source, or suppression of audio signals received from locations other than the spatial location associated with the audio source. Clause 19. A system according to Clause 17 or Clause 18, wherein the plurality of audio characteristics represent differences in the plurality of audio signals corresponding to at least one of the real or imaginary parts of a complex audio spectrum, signal magnitude, signal phase, level and arrival time, or direct sound to reverberation ratio. Clause 20. A system as defined in any of Clauses 17 to 19, which system includes at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system for performing generative AI operations using a large language model (LLM); a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed reality (MR) content; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system for performing hardware tests using simulations; a system for performing generative content operations using a language model; a system for generating synthetic data; a collaborative content creation platform for 3D assets; or a system implemented at least in part using cloud computing resources.

[0183] Other variations are within the spirit of the present disclosure. While the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof have been shown in the drawings and described in detail above. It should be understood, however, that the intention is not to limit the disclosure to any particular form or forms, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure as defined by the appended claims.

[0184] The use of the terms "a" and "an" and "the," "the," "which," and similar designations in connection with the description of disclosed embodiments (particularly in connection with the following claims) is to be construed to include both the singular and the plural, unless otherwise stated herein or clearly contradicted by context, and not as a definition of any term. The terms "comprising," "having," "containing," and "including" are to be construed as open-ended terms (in the sense of "including, but not limited to") unless otherwise stated. "Connected," when unmodified and referring to physical connections, is to be construed as partially or wholly contained, attached, or joined, even if something in between.The enumeration of ranges of values ​​is intended only as a shorthand way to refer to each individual value that falls within the range, unless otherwise noted here, and each individual value is included in the specification as if it were listed individually here. The use of the term "set" (for example, "a set of items") or "subset" is, unless otherwise noted or contradicted by the context, to mean a non-empty collection comprising one or more members. Unless otherwise noted or contradicted by the context, the term "subset" of a corresponding set does not necessarily denote a proper subset of the corresponding set; rather, subset and corresponding set may be the same.

[0185] Subjunctive expressions, such as sentences of the form "at least one of A, B, and C" or "at least one of A, B, and C," are, unless explicitly stated otherwise or clearly contradicted by the context, taken with the context as they are generally used to represent that an item, term, etc., can be either A or B or C, or any non-empty subset of the set of A and B and C. In an illustrative example for a set with three members, the conjunctive expressions "at least one of A, B, and C" and "at least one of A, B, and C" refer to one of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Such a subjunctive formulation is therefore not intended to mean in general that in certain embodiments at least one of A, at least one of B and at least one of C must be present.Unless otherwise noted or contradicted by the context, the term "multiplicity" denotes a state of plurality (for example, "a plurality of items" denotes multiple items). A plurality consists of at least two items, but may be more if indicated either explicitly or by the context. Unless otherwise noted or evident from the context, the phrase "based on" means "at least partly based on" and not "exclusively based on."

[0186] The operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. In at least one embodiment, a process such as the processes described herein (or variations and / or combinations thereof) is performed under the control of one or more computer systems configured with executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that are collectively executed on one or more processors, by hardware, or combinations thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, e.g., in the form of a computer program comprising a plurality of instructions executable by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagated transient electrical or electromagnetic transmission) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues) in transient signal transceivers. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media having executable instructions stored thereon (or other memory for storing executable instructions) that, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform operations described herein.A set of non-transitory computer-readable storage media, in at least one embodiment, comprises a plurality of non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media of the plurality of non-transitory computer-readable storage media lacks all of the code, while a plurality of non-transitory computer-readable storage media collectively store all of the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors - for example, a non-transitory computer-readable storage medium stores instructions, and a main CPU executes some of the instructions, while a GPU executes other instructions. In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0187] Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or collectively perform operations of the processes described herein, and such computer systems are configured with applicable hardware and / or software that enable operations to be performed. Further, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system comprising multiple devices that operate differently such that the distributed computer system performs operations described herein and a single device does not perform all operations.

[0188] The use of examples or exemplary language (e.g., "such as") is intended only to better illustrate embodiments of the disclosure and does not limit the scope of the disclosure unless otherwise stated. No language in the specification should be construed to infer any unclaimed element as essential to the practice of the disclosure.

[0189] All references cited herein, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference were individually and expressly indicated to be incorporated by reference and reproduced herein in its entirety.

[0190] The terms "coupled" and "connected," and their derivatives, may be used in the description and claims. It should be understood that these terms are not intended to be synonymous. Rather, in certain examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" can also mean that two or more elements are not in direct contact with each other, but nevertheless cooperate or interact with each other.

[0191] Unless explicitly stated otherwise, terms such as "processing", "computing", "calculating", "determining" or similar terms used throughout the specification refer to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or transform data represented as physically, such as electronic quantities in the registers and / or memory of the computer system into other data similarly represented as physical quantities in the memory, registers or other devices for storing, transmitting or displaying information.

[0192] Similarly, the term "processor" may refer to a device or a portion of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As non-limiting examples, "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, the term "software" processes may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, as well as intelligent agents. Each process may also refer to multiple processes to execute instructions sequentially or in parallel, continuously or intermittently.The terms “system” and “process” are used interchangeably here in the sense that the system may embody one or more processes and processes may be considered as a system.

[0193] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data to a subsystem, a computer system, or a computer-implemented calculator. Obtaining, acquiring, receiving, or inputting analog and digital data may be performed in a variety of ways, such as receiving data as a parameter to a function call or an application programming interface call. In some implementations, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be performed by transmitting data over a serial or parallel interface. In another implementation, the process of obtaining, acquiring, receiving, or inputting analog or digital data may be performed by transmitting data over a computer network from the providing entity to the acquiring entity.It may also refer to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, a process of providing, outputting, transmitting, sending, or presenting analog or digital data may be performed by passing data as an input or output parameter of a function call, a parameter of an API, or an interprocess communication mechanism.

[0194] Although the above discussion sets forth example implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities are defined above for discussion purposes, various functions and responsibilities may be distributed and allocated in different ways depending on the circumstances.

[0195] Furthermore, although the subject matter has been described in language referring to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the particular features or acts described. Rather, particular features and acts are disclosed as exemplary forms of implementing the claims.

[0196] It is to be understood that the aspects and embodiments described above are purely exemplary and that changes in detail may be made within the scope of the claims.

[0197] Each device, method, and feature disclosed in the description and (optionally) in the claims and drawings may be presented independently or in any suitable combination.

[0198] Reference signs in the claims are for illustrative purposes only and do not limit the scope of the claims.

Claims

[1] Computer-implemented method comprising: Extracting a plurality of audio features from a plurality of audio signals recorded using a plurality of microphones; Processing the plurality of audio features using a neural network to generate a feature vector representing a spatial location of an audio source with respect to the plurality of microphones; and Providing the feature vector to a task-specific model to perform at least one audio-related task corresponding to the plurality of audio signals based at least on the spatial location. [2] The computer-implemented method of claim 1, wherein the audio-related task includes at least one of the following: extraction of speech corresponding to the spatial location, echo cancellation, enhancement of audio signals received from the spatial location associated with the audio source, or suppression of audio signals received from locations other than the spatial location associated with the audio source. [3] A computer-implemented method according to claim 1 or claim 2, wherein the plurality of audio features represent differences in the plurality of audio signals corresponding to at least one of the real or imaginary parts of a complex audio spectrum, signal magnitude, signal phase, level and arrival time, or direct sound to reverberation ratio. [4] A computer-implemented method according to any one of the preceding claims, wherein the feature vector includes at least a minimum number of elements and wherein none of the elements need relate to a physical direction or physical location with respect to one or more of the plurality of microphones. [5] A computer-implemented method according to any one of the preceding claims, further comprising: Combining the plurality of audio signals into a single audio signal before performing the at least one audio-related task. [6] Computer-implemented method according to one of the preceding claims, wherein the feature vector represents a location-related embedding of the audio source in a latent space. [7] A computer-implemented method according to any one of the preceding claims, wherein the plurality of audio features comprises at least one of cross-channel features, temporal features, or spectral features. [8] A computer-implemented method according to any one of the preceding claims, further comprising: Analyzing audio data indicated as being generated by the sound source to identify audio data from the sound source represented in the plurality of audio signals. [9] A computer-implemented method according to any one of the preceding claims, wherein the processing of the plurality of audio features is performed independently of the availability of information indicative of a physical configuration of the plurality of microphones. [10] Processor comprising one or more circuits for: Extracting a plurality of audio features from a plurality of audio signals recorded using a plurality of microphones, Processing the plurality of audio features using a neural network to generate a feature vector representing a spatial location of an audio source with respect to the plurality of microphones; and Providing the feature vector to a task-specific model to perform at least one audio-related task based at least in part on the spatial location. [11] The processor of claim 10, wherein the audio-related task includes at least one of the following: extraction of speech corresponding to the spatial location, echo cancellation, enhancement of audio signals received from the spatial location associated with the audio source, or suppression of audio signals received from locations other than the spatial location associated with the audio source. [12] A processor according to claim 10 or claim 11, wherein the plurality of audio features represent differences in the plurality of audio signals corresponding to at least one of the real or imaginary parts of a complex audio spectrum, signal magnitude, signal phase, level and arrival time, or direct sound to reverberation ratio. [13] The processor of any of claims 10-12, wherein the feature vector includes at least a minimum number of elements, and wherein none of the elements need relate to a physical direction or physical location with respect to one or more of the plurality of microphones. [14] Processor according to any one of claims 10-13, wherein the feature vector represents a location-related embedding of the audio source in a latent space. [15] The processor of any of claims 10-14, wherein the plurality of audio features includes at least one of cross-channel features, temporal features, or spectral features. [16] Processor according to any one of claims 10-15, wherein the processor comprises at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed reality (MR) content; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system for performing hardware tests using simulations; a system for performing generative content operations using a language model; a system for generating synthetic data; a system for performing generative AI operations using a large language model (LLM); a platform for collaborative content creation for 3D assets; or a system implemented at least in part using cloud computing resources. [17] System comprising: one or more processors to provide a feature vector to a task-specific model to perform at least one audio-related task, wherein the feature vector is generated from a plurality of audio features using a neural network and represents a spatial location of an audio source with respect to the plurality of microphones, wherein the plurality of audio features are derived from a plurality of audio signals acquired using the plurality of microphones. [18] The system of claim 17, wherein the audio-related task includes at least one of the following: extraction of speech corresponding to the spatial location, echo cancellation, enhancement of audio signals received from the spatial location associated with the audio source, or suppression of audio signals received from locations other than the spatial location associated with the audio source. [19] The system of claim 17 or claim 18, wherein the plurality of audio features represent differences in the plurality of audio signals corresponding to at least one of the real or imaginary parts of a complex audio spectrum, signal magnitude, signal phase, level and arrival time, or direct sound to reverberation ratio. [20] A system according to any one of claims 17-19, wherein the system comprises at least one of the following: a system for performing simulation operations; a system for performing simulation operations to test or validate autonomous machine applications; a system for performing digital twin operations; a system for performing light transport simulations; a system for rendering graphical output; a system for performing deep learning operations; a system for performing generative AI operations using a large language model (LLM); a system implemented using an edge device; a system for generating or displaying virtual reality (VR) content; a system for generating or displaying augmented reality (AR) content; a system for generating or displaying mixed reality (MR) content; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system for performing hardware tests using simulations; a system for performing generative content operations using a language model; a system for generating synthetic data; a platform for collaborative content creation for 3D assets; or a system implemented at least in part using cloud computing resources.

Images