OBJECT IMAGE FEATURE GENERATION FROM AUDIO SIGNALS USING NEURAL NETWORKS

Abstract

Devices, systems, and techniques for using one or more neural networks to map object features are described. In at least one embodiment, one or more neural networks are used to generate one or more image object features, based at least partially on a variable sampling rate of one or more audio signals.

Description

TECHNICAL AREA

[0001] At least one embodiment relates to processing resources used to perform and facilitate artificial intelligence for various tasks. For example, at least one embodiment relates to processors or computing systems that use neural networks to generate object image features. BACKGROUND

[0002] Artificial intelligence techniques are used to implement various tasks. The architecture and size of parameters in neural networks, for example, can consume significant memory, time, or computing resources to perform different tasks. The amount of memory, time, or computing resources used to perform different tasks can be improved through training or inference using a neural network. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates a block diagram of an image object feature generation using neural networks and a variable sampling rate of audio signals according to at least one embodiment; Fig. 2 illustrates a block diagram of an audio-based facial motion generation system that uses neural networks and a variable sampling rate of audio signals, according to at least one embodiment; Fig. 3 illustrates a block diagram of an audio encoder according to at least one embodiment; Fig. 4 illustrates a block diagram of a motion decoder according to at least one embodiment; Fig. Figure 5 is a flowchart illustrating techniques that implement image object feature generation using neural networks and a variable sampling rate of audio signals according to at least one embodiment; Fig. Figure 6 illustrates an exemplary data center system according to at least one embodiment; Fig. Figure 7 illustrates a system-on-a-chip (SOC) according to at least one embodiment; Fig. Figure 8A illustrates a parallel processor according to at least one embodiment; Fig. Figure 8B illustrates a processing cluster according to at least one embodiment; Fig. Figure 8C illustrates a graphics multiprocessor according to at least one embodiment. Fig. Figure 9 illustrates an accelerator processor according to at least one embodiment; Fig. Figure 10A illustrates a central processing unit and a core of the central processing unit according to at least one embodiment; Fig. Figure 10B illustrates a core of the central processing unit in Fig. 10A according to at least one embodiment; Fig. Figure 11 illustrates another accelerator processor according to at least one embodiment; Fig. Figure 12 illustrates a neuromorphic processor according to at least one embodiment; Fig. 13 illustrates a supercomputer according to at least one embodiment; Fig. Figure 14 illustrates another accelerator processor according to at least one embodiment; Fig. 15 illustrates another processor according to at least one embodiment; Fig. Figure 16 illustrates another accelerator processor according to at least one embodiment; Fig. Figure 17 illustrates a tensor processing unit according to at least one embodiment; Fig. Figure 18 illustrates a RISC-V compatible processor according to at least one embodiment; Fig. 19A and Fig. Figure 19B illustrates a speech processing unit according to at least one embodiment; Fig. 20 illustrates a software stack of a programming platform according to at least one embodiment; Fig. 21 illustrates software supported by a programming platform according to at least one embodiment; Fig. 22 illustrates compiling code for execution on programming platforms of Fig. 18 according to at least one embodiment; Fig. Figure 23 illustrates an example of an autonomous vehicle and its system architecture according to at least one embodiment; Fig. 24A illustrates inference and / or training logic according to at least one embodiment; Fig. 24B illustrates inference and / or training logic according to at least one embodiment; Fig. 24C illustrates the training and use of a neural network according to at least one embodiment. DETAILED DESCRIPTION

[0003] Fig. Figure 1 illustrates a block diagram of image object feature generation using neural networks and a variable sampling rate of audio signals according to at least one embodiment. In at least one embodiment, image object feature generation 110 can implement different techniques to generate features of objects in image data according to audio signal(s) 102. In at least one embodiment, changing object features 104 can be used to render a sequence of image frames included in a video to show object motion corresponding to audio signal(s) 102. In at least one embodiment, as described below with respect to Fig. 2. As discussed in detail, image object features can be facial movements corresponding to speech audio. In at least one embodiment, other image objects (e.g., vehicles, animals, or other objects capable of movement, such as projectiles) corresponding to audio signals can be generated (e.g., train noises can be used in audio signal(s) 102 to generate train image features that change over time).

[0004] In at least one embodiment, image object feature generation can implement 110 different techniques, which are described below with respect to Fig. 2, Fig. 4 and Fig. 5 will be discussed in detail to generate image object features using one or more neural networks 130 based on a variable sampling rate of audio frames 104. In at least one embodiment, the use of a variable sampling rate can allow varying sizes, proportions, or attentions of audio data to be used to improve the matching of object image feature movements with corresponding sounds (e.g., facial movements matching speech audio), thus allowing optimal proportions of audio data to guide the generation of object image features. In at least one embodiment, for example, an image frame 104a can be input for further image feature generation, and audio signal(s) 102 can be sampled to obtain, use, or focus using attention techniques, such as bias matching as discussed below, to use four audio frames 122A.In at least one embodiment, an output image frame 140B can be generated, which can be provided as input for a subsequent image frame generation based on audio frame sampling 122B of five audio frames to generate image frame 140C, which can be provided as input for a subsequent image frame generation based on audio frame sampling 122C of three audio frames to generate another image frame, and so on, until audio frames 122N are used to predict a final image frame 140N from audio signal(s) 102.

[0005] Fig. Figure 2 illustrates a block diagram of an audio-based facial motion generation system that uses neural networks and a variable sampling rate of audio signals, according to at least one embodiment. In at least one embodiment, the audio-based facial motion generation system 210 can receive audio 202 (e.g., one or more raw audio signals). In at least one embodiment, the audio encoder 220 can employ various techniques, as described below with respect to Fig. 3 is discussed in detail to generate coded audio 204. In at least one embodiment, an autoregressive face motion decoder 230 can receive coded audio 204 and generate image data with face motions 206 corresponding to speech in audio data. In at least one embodiment, an autoregressive face motion decoder 230 can use previously generated face motions to predict new face motions, as indicated by arrow 206 returning to the autoregressive face motion decoder 230. For example, in at least one embodiment, audio-based face motion generation 210 can receive raw audio as input and autoregressively generate a sequence of animated three-dimensional face meshes that can be rendered to indicate face motion as surface meshes change over time.

[0006] Fig. Figure 3 illustrates a block diagram of an audio encoder according to at least one embodiment. In at least one embodiment, the audio encoder 220 can implement spectral extension 311, which can receive a raw audio signal or other audio data and perform spectral extension. In at least one embodiment, an audio signal can be a raw audio signal, such as pulse-code modulation (PCM) values ​​without metadata. In at least one embodiment, various different file formats or extensions can be used for raw audio signals, including, but not limited to, ".raw", ".pcm", or ".sam".In at least one embodiment, spectral extension 311 can employ techniques to extend audio signals using libraries such as SpecAugment and WavAugment, which can perform one or more extensions, such as pitch randomization, reverberation, additive noise, time dropout, band suppression and / or clipping.

[0007] In at least one embodiment, the output of spectral extension 311 can be passed to convolutional subsampling 312. In at least one embodiment, convolutional techniques are used to extract features from the extended audio data and subsample them. In at least one embodiment, the output of convolutional subsampling 312 can be passed to linear layer 313. In at least one embodiment, a linear layer 313 can perform a linear transformation on input data (e.g., a vector) to generate output data (e.g., a vector) using trained weights and biases. In at least one embodiment, the output of linear layer 313 can be passed to dropout 314. In at least one embodiment, dropout 314 can drop some data (e.g., scores or other values ​​determined from audio signals).

[0008] In at least one embodiment, output from dropout 314 can be passed on to a number (e.g., N) of deep sparse conformers 315. In at least one embodiment, deep sparse conformers 315 can include a feed-forward layer 321, the output of which can be passed on to multi-head self-awareness sparse conformers 322. In at least one embodiment, output from multi-head self-awareness sparse conformers 322 can be passed on to convolutional layer(s) 323. In at least one embodiment, output from convolutional layer(s) 323 can be passed on to feed-forward layer(s) 324. In at least one embodiment, output from feed-forward layer(s) 324 can be passed on to layer normalizer 325, which can output coded audio (e.g., sparsely populated coded audio frames).

[0009] In at least one embodiment, Multi-Head Self-Attentional Weakness 322 can be described using algorithm pseudocode as:

[0010] In at least one embodiment, Algorithm 1 describes pseudocode for a multi-head attention layer using ProbSparse (lines 12 to 19 and 21) and relative positional coding (lines 10, 11, and 19). In at least one embodiment, an additional linear layer with weight matrix W can be included. P ∈ R d×d They are introduced to parameterize positional information of storage tensors. In at least one embodiment, two learnable vectors U1, U2 ∈ R are used. d (d = h * d q ) in forms of (h, d q) in lines 10 and 11 to assist relative positional coding by mitigating attentive bias towards different words in different positions. In at least one embodiment, the number L' K Key vectors are used by sampling the entire L K (Lines 12, 13). In at least one embodiment, a hyperparameter c1 (e.g., = 5.0) determines the size of a key vector subset. In at least one embodiment, when a query sparsity measurement is calculated, Top-L' Q Queries where L' QA number is selected from the query vectors. In at least one embodiment, another hyperparameter c2 (e.g., = 5.0) can determine the size of the query vector subset. In at least one embodiment, as shown in line 19, "similarity scores" between a reduced query vector set and an original key and position tensors can be combined. In at least one embodiment, scores can be masked by the lengths of sequences in a batch, normalized to probability-style scores under softmax, and finally dropped out with a given probability p. dropout .

[0011] In at least one embodiment, other multi-head self-awareness techniques can be included in audio encoder 220 to encode audio as part of or instead of deep sparse conformers 315.

[0012] Fig. Figure 4 illustrates a block diagram of a motion decoder according to at least one embodiment. In at least one embodiment, autoregressive face motion decoder 230 can receive one or more past face motions, determined by autoregressive face motion decoder 230, such as past face motions 402A, 402B to 402N-1. In at least one embodiment, autoregressive face motion decoder 230 can implement motion encoder 411 to encode past face motions 402A, 402B to 402N-1 for subsequent processing and decoding. In at least one embodiment, style embedding 406 can be received and combined with encoded past face motions for input into periodic position encoding 412. In at least one embodiment, style embedding can correspond to a speaker style or the style of an audio signal producer (e.g.,Speaker age, speaker gender, or other speaker characteristics that can be provided by another neural network trained to classify or generate a style embedding).

[0013] In at least one embodiment, periodic position coding 412 can further encode past facial movements (and style embedding) for input into biased causal multi-head self-awareness 413. In at least one embodiment, periodic position coding 412 can support large amounts of input (e.g., large amounts of past facial movements, such as a large number of frames). In at least one embodiment, biased causal multi-head self-awareness 413 can apply or use encoded audio 404 obtained from audio encoder 220 to learn and apply temporal bias 421. In at least one embodiment, biased causal multi-head self-awareness 413 can apply self-awareness such that the importance of past facial frames at the current time can also be determined by an audio sequence.In at least one embodiment, a goal of gebiaster causal MH self-awareness can be to allow generalization to longer sequences in speech modeling and face frame sequence modeling. In at least one embodiment, starting from a temporally encoded face motion representation sequence: Ft=(f1,f2,…,ft)

[0014] Geiaste causal multi-head self-awareness first projects linear F t in queries Q F and key K F of latent dimension d k and values ​​V F of dimension d v In at least one embodiment, to learn dependencies under each frame in a context of the historical face movement sequence, a weighted context representation can be calculated by performing a scaled dot product attention, which can be described as: Att(QF,KF,VF,BF)=softmax(QF(KF)⊤dk+BF¯)VF where B F learnable temporal bias 421 can be added to an attention matrix to ensure causality and improve the ability to generalize to longer sequences by also referring to a current audio sequence.

[0015] In at least one embodiment, B F a t × t, where t can represent a number of frames, such that t × t is a square matrix encompassing negative infinity in an upper triangle (e.g., empty squares in temporal bias 421) to avoid considering future (e.g., right-handed) frames in order to make current predictions of the next frame. In at least one embodiment, learnable temporal biases can be appended to a lower (e.g., bottom-left corner) triangle of temporal bias 421 (e.g., B FIn at least one embodiment, temporal bias 421 can take into account various aspects, including: • Relative distance d(i, j) between current frame (i-th) and previous frames (e.g., j-th), where a greater distance reflects a smaller impact on a current frame • Attention score att F (i, j) between a current frame and previous frames, where a higher attention (e.g., cosine) similarity score reflects a greater impact on a current frame • (Self-)Attention Score att A(i, j') between a current audio frame (e.g., i'-ter) and previous audio frames (e.g., j'-ter), wherein during both training and inference an estimate of the match between a current face frame to a fragment of audio frames, as an entire output of a final layer of a densely sparsely populated transformer in an audio encoder may already have computed a self-awareness score matrix (e.g., of size T × T) of an audio sequence, and a part (e.g., t × t) can be used to determine learnable temporal bias 421.

[0016] In at least one embodiment, a linear combination of different aspects of learnable temporal bias 421 can be described as: BF(i,j)=w1×d(i,j)+w2×attF(i,j)+w3×attA(i',j') where w1, w2, w1 are learnable weights for the aspects described above.

[0017] In at least one embodiment, coded values ​​from biased multi-head self-awareness 412 can be passed to biased cross-modal multi-head self-awareness 414, which can apply matching bias 423 to apply a variable sampling rate of coded audio 404. In at least one embodiment, cross-modal multi-head self-awareness 414 can combine coded audio 404 (e.g., an audio tensor feature sequence) and a biased causal motion feature sequence to match audio and motion modalities. In at least one embodiment, to smoothly control the weights from the audio sequence, we add a learnable matching bias to a query key attention score (e.g., query = motion sequence and key = audio memory). In at least one embodiment, a cross-attention score can be computed as: Att(QF,KA,VA,BA)=softmax(QA(KA)⊤dk+BA¯)VA

[0018] In at least one embodiment, B A refer to a learnable matching bias 423 that can assign a weight to each pair of previously generated motion frames and each audio frame in an entire speech input sequence. In at least one embodiment, for an i-th motion frame, a range (e.g., 2α) of audio frames [i' - α, i' + α] can be roughly matched, but a precise range that contributes most to generating the next motion frame cannot be known in advance. In at least one embodiment, an encoded audio sequence can include a variety of speech styles (e.g., singing, shouting, or varying emotions). In at least one embodiment, a form of cross-attention memory can be described as (t, T) where t can be a motion frame sequence length and T can be an audio frame sequence length.

[0019] In at least one embodiment, learnable matching bias 423 can include higher bias weights for continuous audio frames whose motion frames are closer to a current motion frame (e.g., weighted by d(i, j) for the i-th and j-th motion frames). In at least one embodiment, an attention score can be guided to a self-awareness score between the i'-th motion frame and the j'-th audio frame. In at least one embodiment, a learnable matching bias can be defined as a linear combination of aspects such as those described above: BA(i,j')=w1×d(i,j)+w2×attA(i',j')

[0020] In at least one embodiment, determining matching bias 423 can involve finding a range of j (motion feature index set) from a given j' (speech frame index) and finding a range of i' (speech frame index set) from a given i (motion frame index).

[0021] In at least one embodiment, output from gebiaster cross-modal multi-head self-awareness 414 can be passed on to feed-forward 415.

[0022] In at least one embodiment, the output of feed-forward 415 can be passed on to motion decoder 416, which generates a current face movement 402N.

[0023] Fig. Figure 5 is a flowchart illustrating techniques that implement image feature generation using neural networks and a variable sampling rate of audio signals according to at least one embodiment. In at least one embodiment, one or more audio signals can be received, as shown in Figure 510. In at least one embodiment, audio signals can be received in a streaming manner (e.g., in real time) from a recording device or other sensor that captures audio data and provides the signal to a system, such as image feature generation (Figure 110) or audio-based facial motion generation (Figure 210). In at least one embodiment, audio signals can be received from a storage system and provided for image feature generation as part of an audio processing pipeline, an image generation system, or other offline processing system.In at least one embodiment, different audio signals can be obtained (e.g., different audio signals captured using different microphones) and made available for image object feature generation.

[0024] In at least one embodiment, one or more neural networks can be used to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as shown in Figure 520. In at least one embodiment, image object features can include two- or three-dimensional image data (e.g., frames, meshes, pixels, or other image data that can be used to render a sequence of images to indicate motion in two or more dimensions). In at least one embodiment, a variable sampling rate can include techniques that direct attention or weight some audio data (e.g., audio frames) more heavily than other audio data. In at least one embodiment, a variable sampling rate can be generated probabilistically, such as a learnable matching bias applied as part of cross-modal multi-head self-attention, as described above in relation to Figure 520. Fig. 4 discussed in detail. In at least one embodiment, other learnable techniques based on audio signals can be applied, such as causal multi-head self-awareness that applies a learnable temporal bias, as above in relation to Fig. 4 discussed. In at least one embodiment, image object feature generation can be performed to generate facial movements. In at least one embodiment, other types of objects can be generated. In at least one embodiment, image object features for multiple image objects can be generated using one or more input audio signals. DATA CENTER

[0025] Fig. Figure 6 illustrates an exemplary data center 600 in which at least one embodiment can be used. The data center 600 can include one or more rooms with racks 602 and auxiliary equipment used to house one or more racks 602 and one or more base plates 604. A rack 602 can contain one or more base plates 604. A rack 602 can include an enclosure that accommodates and supports individual base plates 604. Operational aspects of the rack 602 can be regulated at a rack level, according to a group of base plates 604, or at a base plate level, according to individual base plates 604, among other options. The rack 602 or base plates 604 can have individually selected maximum operating parameters, such as, but not limited to, power consumption, operating frequencies, and others.Data center 600 can be supported by various cooling systems, such as, but not limited to, cooling towers, cooling circuits, pumps, and other support systems. Cooling systems may include sensors and controllers for monitoring and managing cooling characteristics for racks 602. Base plates 604 within racks 602 can receive power from one or more power distribution units (PDUs; not shown). PDUs may be located within racks 602, for example, between racks 602 including base plates 604, or within racks 602 that also house base plates 604.

[0026] Racks 602 and base plates 604 can contain subsystems, modules, add-in cards, and other semiconductor components. Base plates 604 can contain one or more computing units 606, which can contain one or more processors 608, one or more memory units 610, and an interface controller 612. Computing units 606 can contain any number of processors, such as, but not limited to, central processing units (“CPUs”), graphics processing units (“GPUs”), or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors, etc.), including any processors described herein, such as, but not limited to, the processors in Fig. 7-19. Computing Units 606 can include one or more Storage Devices 610 (e.g., dynamic read-only memory, solid-state memory, or disk storage), as well as Network Input / Output Devices (“NW I / O”), Network Switches, Virtual Machines (“VMs”), Power Modules, and Cooling Modules, etc. One or more Computing Units 606 can constitute a server with one or more of the above-mentioned computing resources.

[0027] Compute Units 606 can include separate groupings of compute units located in one or more racks (not shown), or many racks located in data centers at different geographic locations (also not shown). Separate groupings of compute units can include grouped compute, network, short-term memory, or long-term storage resources that can be configured or allocated to support one or more workloads. Multiple compute units (e.g., including CPUs and / or other processors) can be grouped within one or more racks to provide compute resources to support one or more workloads. A Resource Orchestrator 614 can configure or otherwise control one or more Compute Units 606 or groups of compute units.Resource Orchestrator 614 can include a Software Design Infrastructure (“SDI”) management entity for Data Center 600. Resource Orchestrator 614 can include hardware, software, or any combination thereof.

[0028] Data Center 600 can include any one or any combination of a framework layer 620, a software layer 630, and an application layer 6340. As in Fig. As shown in Figure 6, framework layer 620 includes a job scheduler 622, a configuration manager 624, a resource manager 626, and a distributed file system 628. Framework layer 620 can include a framework to support software 632 from software layer 630 and / or one or more applications 642 from application layer 640. Software 632 or application(s) 642 can each include, but are not limited to, web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 620 can be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark™ (hereinafter "Spark"), which can use distributed file system 628 for large-scale data processing (e.g., "Big Data").Job Scheduler 622 can include a Spark driver to facilitate scheduling workloads supported by various layers of Data Center 600. Configuration Manager 624 can configure various layers, such as, but not limited to, Software Layer 630 and Framework Layer 620, including Spark and Distributed File System 628, to support large-scale computing. Resource Manager 626 can manage clustered or grouped compute units 606, mapped to or allocated to support Distributed File System 628 and Job Scheduler 622. Resource Manager 626 can coordinate with Resource Orchestrator 614 to manage these mapped or allocated compute resources.

[0029] Software 632 can be contained in software layer 630 and can include software used by at least portions of a computing unit 606, one or more computing units 606, groups of computing units 606, and / or a distributed file system 628 of framework layer 620. One or more types of software can include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0030] Application(s) 642 may be included in application layer 640 and may include one or more types of applications that utilize at least portions of a compute unit 606, one or more compute units 606, groups of compute units 606, and / or distributed file system 628 of framework layer 620. One or more types of applications may, but are not limited to, include any number of a genomics application, a cognitive computing application, and a machine learning application, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0031] Each of the Configuration Manager 624, Resource Manager 626, and Resource Orchestrator 614 can implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible way. Self-modifying actions can relieve a Data Center 600 operator of potentially making poor configuration decisions and potentially avoiding underutilized and / or poor-performing portions of a data center.

[0032] Data Center 600 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using software and computing resources described above in relation to Data Center 600. Trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above in relation to Data Center 600 by using weight parameters calculated by one or more training techniques described herein.

[0033] Data Center 600 can accommodate CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware (e.g., configurations in Fig. 7-19) to perform some or all of the processes and techniques described elsewhere herein, such as, but not limited to, training and / or inference using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as a service to allow users to train or infer information, such as image recognition, speech recognition, or other artificial intelligence services, but not limited to.

[0034] In at least one embodiment, processor 608 may include one of the processors listed below and / or have one or more circuits to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform one of the operations described above or elsewhere herein. In at least one embodiment, processor 608 is configured by software 632 to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. Data Center 600 can utilize logic, CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware (e.g., embodiments in Fig. 7-19) to perform any of the operations described above or elsewhere herein. PROCESSORS

[0035] The following figures represent, without limitation, exemplary processors and processing systems that can be used to employ one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform some or all of the processes, operations, and / or techniques described elsewhere herein. Exemplary processors and processing systems can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. Processors and processing systems may consist of logic, central processing units (CPUs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), field-programmable arrays (FPGAs), XPUs (i.e., any computing architecture best suited to the needs of an application), or other hardware (e.g., embodiments in Fig. 7-19) to perform any of the operations described above, below, or elsewhere herein. Processors and / or processing systems described herein may include one or more circuits that can be used to employ one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. As used herein, one or more circuits can be configured by software to use one or more neural networks to generate image object features based at least in part on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein. Fig. 24A and Fig. Section 24B illustrates logic 2415, which, as described elsewhere herein, can be used in one or more devices to perform operations such as, but not limited to, those discussed herein according to at least one embodiment. Logic may, for example, refer to any combination of software logic, hardware logic, and / or firmware logic to provide functionality and / or operations described herein, wherein logic, collectively or individually, may be embodied as switching technology that forms part of a larger system, such as an integrated circuit (IC), an application-specific integrated circuit (ASIC), a field-programmable array (FPGA), a system-on-a-chip (SoC), or one or more processors (e.g., CPU, GPU).

[0036] Fig. Figure 7 illustrates a processor that is a System-on-a-Chip (SOC) 700 (which may be referred to as a System-on-Chip, a Superchip, or by another name) according to at least one embodiment. SOC 700 may include processor complex 710 and processor complex 740. SOC 700 may include any number of processor complexes 710 and / or processor complexes 740, which may include any number of processors described herein, such as, but not limited to, those in Fig. 7-19, in any combination. For example, processor 710 may include a central processing unit (CPU) and processor 740 may include a graphics processing unit (GPU). Alternatively, processor 710 may include a GPU and processor 740 may include a GPU. SOC 700 may include any number of display controllers 792, any number of multimedia engines 794, any number of I / O interfaces 770, any number of memory controllers 780, and any number of fabrics 760 in any combination. For explanatory purposes, multiple instances of similar objects are referred to herein by reference numbers that identify the object and, where needed, by bracket numbers that identify the instance. SOC700 may include a processor made by Broadcom in Palo Alto, CA.

[0037] Processor complex 710 can include a CPU, processor complex 740 can include a GPU, and SOC 700 can be a processing unit that integrates the 710 and 740 on a single chip. Some tasks can be assigned to processor complex 710, and other tasks can be assigned to processor complex 740. Processor complex 710 can be configured to run main control software associated with SOC 700, such as, but not limited to, an operating system. Processor complex 710 can be the main processor of SOC 700, controlling and coordinating operations of other processors. Processor complex 710 can issue instructions that control the operation of processor complex 740 to perform some or all of the operations described herein. Processor complex 710 can be configured to execute host executable code provided by CUDA or other source code (e.g.,HIP source code) is derived, and processor complex 740 can be configured to execute device executable code derived from CUDA or other source code to perform any of the operations described herein.

[0038] A 710 processor complex can include 720(1)-720(4) cores and a 730 cache (e.g., L3 cache) to store information for performing the operations described herein. A 710 processor complex can include any number of 720 cores and any number and type of caches in any combination. 720 cores can be configured to execute instructions of a single instruction set architecture (“ISA”) to perform some or all of the operations described herein. Each 720 core can contain one CPU core. 720(1)-720(4) cores can be referred to as an arithmetic unit or arithmetic units. A 700 system-on-a-chip (SoC) can include any number of 710 processor complexes, 760 fabrics, 770 I / O interfaces, and 780 memory controllers.

[0039] Each Core 720 can include a Fetch / Decode Unit 722, an Integer Execution Engine 724, a Floating-Point Execution Engine 726, and an L2 Cache 728. The Fetch / Decode Unit 722 can fetch instructions to perform some or all of the operations described herein (such as, but not limited to, an API compiled into instructions) and decode such instructions, generate microoperations, and send separate microinstructions to the Integer Execution Engine 724 and / or the Floating-Point Execution Engine 726. The Fetch / Decode Unit 722 can simultaneously send one microinstruction to the Integer Execution Engine 724 and another microinstruction to the Floating-Point Execution Engine 726. The Integer Execution Engine 724 can perform integer and memory operations. The Floating-Point Engine 726 can perform floating-point and vector operations.The 722 call decoder unit can send micro-instructions to one or more execution engines, replacing both the 724 integer execution engine and the 726 floating-point execution engine.

[0040] Each core 720(i), where i is an integer representing a single instance of core 720, can access L2 cache 728(i), contained within core 720(i). Each core 720, contained within core complex 710(j), where j is an integer representing a single instance of core complex 710, can be connected to other cores 720, contained within core complex 710(j), via L3 cache 730(j), contained within core complex 710(j). Cores 720, contained within core complex 710(j), where j is an integer representing a single instance of core complex 710, can access all of L3 cache 730(j), contained within core complex 710(j). L3 cache 730 can contain any number of slices.

[0041] Processor Complex 740 can be a graphics complex that can be configured to perform computational operations (e.g., computational operations involved in the operations described herein) in a highly parallel manner. Processor Complex 740 can be configured to perform graphics pipeline operations, such as, but not limited to, drawing commands, pixel operations, geometric calculations, and other operations associated with rendering an image to a display. Processor Complex 740 can be configured to perform operations not related to graphics, such as, but not limited to, training and / or simulations of neural networks. Processor Complex 740 can be configured to perform both graphics-related and non-graphics-related operations.

[0042] A 740 processor complex can include any number of 750(1)-750(N) processing units, where N is any integer greater than 1, and an L2 cache 742. 750 processing units can share the L2 cache 742, which can store information to be used to perform some or all of the operations described herein. The L2 cache 742 can be partitioned. A 740 processor complex can include any number of 750 processing units and any number (including zero) and type of caches. A 740 processor complex can include any amount of dedicated graphics hardware.

[0043] Each compute unit 750 can contain any number of SIMD units 752(1)-752(N), where N is any integer greater than 1, and a shared memory 754. Each SIMD unit 752 can implement a SIMD architecture and can be configured in parallel for some or all of the operations described herein. Each compute unit 750 can execute any number of thread blocks, but each thread block can be executed on a single compute unit 750, although in some embodiments a thread block can be executed on multiple compute units. A thread block can contain any number of execution threads. A workgroup can be a thread block. Each SIMD unit 752 can execute a group of threads. A group of threads (e.g., 16 threads), which can also be called a warp, subgroup, or wavefront (e.g.,(as used by AMD and Intel), where each thread in the warp, wave, subgroup, or wavefront can belong to a single thread block and is configured to process a different data set based on a single set of instructions. Prediction can be used to disable one or more threads in a warp, subgroup, or wavefront. A lane can be a thread. A work item can be a thread, as with OpenCL, but is not limited to this. Different warps, subgroups, or wavefronts in a thread block can synchronize with each other and communicate via shared memory. Each compute unit can contain one or more thread block clusters, where a thread block cluster can allow programmatic control of locality with a granularity greater than that of a single thread block of a single streaming multiprocessor (SM).Thread block clusters (also referred to as "clusters") enable multiple thread blocks running concurrently across streaming multiprocessors to synchronize and collaboratively retrieve, exchange, or otherwise use data. In at least one embodiment, streaming multiprocessors ("SMs") can be referred to as streaming microprocessors, stream processors ("SPs"), stream processing units ("SPUs"), compute units ("CUs"), execution units ("EUs"), and / or slices, where a slice in this context can refer to a portion of processing resources within a processing unit (e.g., 16 cores, a ray tracing unit, a thread judge, or a scheduler).

[0044] Fabric 760 can be a system assembly that facilitates data and control transfers across Processor Complex 710, Processor Complex 740, I / O Interfaces 770, Memory Controller 780, Display Controller 792, and Multimedia Engine 794, for example, to perform some or all of the operations described herein. SOC 700 can include any quantity and type of system assembly in addition to or instead of Fabric 760, facilitating data and control transfers across any number and type of directly or indirectly linked components, which may be internal or external to SOC 700. I / O Interfaces 770 can represent any number and type of I / O interface (e.g., PCI, PCI-Extended (“PCI-X”), PCIe, Gigabit Ethernet (“GBE”), USB, etc.). Various types of peripheral devices can be coupled to I / O Interfaces 770.Peripheral devices that can be coupled with I / O interfaces 770 can include keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so on.

[0045] Display controller 792 can display images on one or more display devices, such as, but not limited to, a liquid crystal display (“LCD”) device. Multimedia engine 794 can include any quantity and type of switching technology related to multimedia, such as, but not limited to, a video decoder, a video encoder, a picture signal processor, etc. Memory controller 780 can facilitate data transfers between SOC 700 and a unified system memory 790. Processor complex 710 and processor complex 740 can share unified system memory 790. Unified system memory 790 can include various types of memory devices, including dynamic random-access memory (DRAM) or graphics random-access memory, such as, but not limited to, synchronous graphics random-access memory (SGRAM), including graphics double data rate (GDDR) memory.Unified System Memory 790 can include 3D stacked memory, including but not limited to High Bandwidth Memory (HBM), HBM2e or HDM3.

[0046] The SOC 700 can implement a memory subsystem that includes any number and type of memory controllers 780 and memory devices (e.g., shared memory 754) that can be dedicated to a component or shared by multiple components to perform any of the operations described herein. The SOC 700 can implement a cache subsystem that includes one or more cache memories (e.g., L2 caches 728, L3 cache 730, and L2 cache 742) that can each be private or shared by any number of components (e.g., cores 720, core complex 710, SIMD units 752, compute units 750, and processor complex 740).

[0047] In at least one embodiment, SOC 700 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0048] Fig. Figure 8A illustrates a Parallel Processor 800 according to at least one embodiment. Parallel Processor 800 can be implemented using one or more circuits and can be a programmable processor (e.g., a CPU and / or GPU), logic, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other hardware (e.g., embodiments in Fig. 7-19) to perform any of the operations described above or elsewhere herein.

[0049] The Parallel Processor 800 may include a Parallel Processing Unit 802 to perform any of the operations described above or elsewhere herein. The Parallel Processing Unit 802 may include an I / O Unit 804, which enables communication with other devices, including other instances of the Parallel Processing Unit 802. The I / O Unit 804 may be directly connected to other devices. The I / O Unit 804 may connect to other devices using a hub or switch interface, such as, but not limited to, a Memory Hub 805. Connections between the Memory Hub 805 and the I / O Unit 804 may form a Communication Link 813.The I / O unit 804 can connect to a host interface 806 and a memory crossbar 816, with the host interface 806 receiving commands aimed at performing processing operations and the memory crossbar 816 receiving commands aimed at performing memory operations.

[0050] When host interface 806 receives a command buffer via I / O unit 804, host interface 806 can direct work operations to a front end 808 to execute these commands. Front end 808 can couple with a scheduler 810 (which can be described as a sequencer) configured to distribute commands or other work items to a processing cluster array 812. Scheduler 810 can ensure that processing cluster array 812 is properly configured and in a valid state before tasks can be distributed to a cluster of processing cluster array 812. Scheduler 810 can be implemented via firmware logic running on a microcontroller.The microcontroller-implemented Scheduler 810 can be configured to perform complex scheduling and workload distribution operations with coarse and fine granularity, enabling fast preemption and context switching of threads running on Processing Array 812. Host software can designate workloads for scheduling on Processing Array 812 via one of several graphics processing paths. Workloads can then be automatically distributed across Processing Array 812 by Scheduler 810 logic within a microcontroller that incorporates Scheduler 810.

[0051] Processing cluster array 812 can perform any of the operations described above or elsewhere herein and can contain up to "N" processing clusters (for example, cluster 85A, cluster 85B, through cluster 85N), where "N" is a positive integer (which may be a different integer "N" than used in other figures). Each cluster 85A-85N of processing cluster array 812 can execute a large number of concurrent threads. Scheduler 810 can allocate work to clusters 814A-814N of processing cluster array 812 using various scheduling and / or workload allocation algorithms that can vary depending on the workload generated by each type of program or computation. Scheduling can be handled dynamically by Scheduler 810 or can be partially assisted by compiler logic during the compilation of program logic configured to be executed by processing cluster array 812.Different clusters 814A -814N of processing cluster array 812 can be allocated to process different types of programs or to perform different types of calculations.

[0052] Processing Cluster Array 812 can be configured to perform various types of parallel processing operations, such as, but not limited to, any of the operations described above or elsewhere herein. Processing Cluster Array 812 can be configured to perform general-purpose parallel computing operations. For example, Processing Cluster Array 812 can include logic to perform processing tasks, including filtering video and / or audio data, performing modeling operations, including physics operations, and performing data transformations.

[0053] Processing Cluster Array 812 can be configured to perform parallel graphics processing operations. Processing Cluster Array 812 can include additional logic to support the execution of such graphics processing operations, including, but not limited to, texture sampling logic for performing texture operations, as well as tessellation logic and other vertex processing logic. Processing Cluster Array 812 can be configured to execute graphics processing-related shader programs, such as, but not limited to, vertex shaders, tessellation shaders, geometry shaders, and pixel shaders. Parallel Processing Unit 802 can transfer data from system memory via I / O Unit 804 for processing. During processing, transferred data can be stored in on-chip memory (e.g., Parallel Processor Memory 822) and then written back to system memory.

[0054] When Parallel Processing Unit 802 is used to perform graphics processing, Scheduler 810 can be configured to divide a processing workload into approximately equal-sized tasks to better distribute graphics processing operations across multiple clusters 814A–814N of Processing Cluster Array 812. Portions of Processing Cluster Array 812 can be configured to perform different types of processing. For example, a first portion can be configured to perform vertex shading and topology generation, a second portion can be configured to perform tessellation and geometry shading, and a third portion can be configured to perform pixel shading or other screen-space operations to produce a rendered image for display.Intermediate data produced by one or more of clusters 814A-814N can be stored in buffers to allow intermediate data to be transferred between clusters 814A-814N for further processing.

[0055] Processing cluster array 812 can receive processing tasks to be executed by scheduler 810, which receives commands defining processing tasks from frontend 808. Processing tasks can include indices of data to be processed, such as surface (patch) data, primitive data, vertex data, and / or pixel data, as well as state parameters and commands that define how data is to be processed (e.g., which program to execute). Scheduler 810 can be configured to retrieve indices corresponding to tasks or can receive indices from frontend 808. Frontend 808 can be configured to ensure that processing cluster array 812 is in a valid state before initiating a workload specified by incoming command buffers (e.g., batch buffers, push buffers, etc.).

[0056] Each of one or more instances of Parallel Processing Unit 802 can couple to Parallel Processing Memory 822 to perform any of the operations described above or elsewhere herein. Parallel Processing Memory 822 can be accessed via Memory Crossbar 816, which can receive memory requests from Processing Cluster Array 812 and I / O Unit 804. Memory Crossbar 816 can access Parallel Processing Memory 822 via a Memory Interface 818. Memory Interface 818 can include multiple Partitioning Units (e.g., Partitioning Unit 820A, Partitioning Unit 820B, up to Partitioning Unit 820N), each of which can couple to a portion (e.g., a memory unit) of Parallel Processing Memory 822.A number of partitioning units 820A–820N can be configured to be equal to a number of storage units, such that a first partitioning unit 820A includes a corresponding first storage unit 824A, a second partitioning unit 820B includes a corresponding storage unit 824B, and an Nth partitioning unit 820N includes a corresponding Nth storage unit 824N. A number of partitioning units 820A–820N can also be different from a number of storage units.

[0057] Memory units 824A–824N can include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as, but not limited to, synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. Memory units 824A–824N can also include 3D stacked memory, including, but not limited to, high bandwidth memory (HBM), HBM2e, or HDM3. Render targets, such as, but not limited to, frame buffers or texture images, can be stored across memory units 824A–824N, allowing partitioning units 820A–820N to write portions of each render target in parallel to efficiently utilize available bandwidth of parallel processor memory 822.A local instance of parallel processor memory 822 can be excluded in favor of a unified memory design that uses system memory in conjunction with local cache memory.

[0058] Each of the 814A-814N clusters of the 812 processing cluster array can process data written to any of the 824A-824N storage units within the 822 parallel processor memory. The 816 memory crossbar can be configured to transfer an output from each 814A-814N cluster to any partitioning unit 820A-820N or to another 814A-814N cluster that can perform additional processing operations on the output. Each 814A-814N cluster can communicate with the 818 memory interface through the 816 memory crossbar to read from or write to various external storage devices.Memory crossbar 816 can have a connection to memory interface 818 to communicate with I / O unit 804, as well as a connection to a local instance of parallel processor memory 822, which allows processing units within different processing clusters 814A–814N to communicate with system memory or other memory that is not local to parallel processing unit 802. Memory crossbar 816 can use virtual channels to separate traffic flows between clusters 814A–814N and partitioning units 820A–820N.

[0059] Multiple instances of Parallel Processing Unit 802 can be deployed on a single add-in card, or multiple add-in cards can be interconnected. Different instances of Parallel Processing Unit 802 can be configured to work together compatiblely, even if different instances include different numbers of processing cores, different amounts of local parallel processor memory, and / or other configuration differences. For example, some instances of Parallel Processing Unit 802 can include higher-precision floating-point units relative to other instances.Systems incorporating one or more instances of Parallel Processing Unit 802 or Parallel Processor 800 can be implemented in a variety of configurations and form factors, including but not limited to desktop, laptop or handheld personal computers, servers, workstations, game consoles and / or embedded systems.

[0060] Fig. Figure 8A further includes a block diagram of a partitioning unit 820 according to at least one embodiment. Partitioning unit 820 is an instance of one of partitioning units 820A-820N of Fig. 8A. Partitioning Unit 820 can include an L2 cache 821, a frame buffer interface 825, and a ROP 826 (raster operation unit). L2 cache 821 can be a read / write cache configured to perform load and store operations received from memory crossbar 816 and ROP 826. Read failures and urgent write-back requests can be passed from L2 cache 821 to frame buffer interface 825 for processing. Updates can also be sent to a frame buffer via frame buffer interface 825 for processing. Frame buffer interface 825 can be connected to one of the memory units in parallel processor memory, such as, but not limited to, memory units 824A–824N. Fig. 8A (e.g., within parallel processor memory 822).

[0061] The ROP 826 can be a processing unit that performs raster operations, such as, but not limited to, stenciling, z-testing, blending, etc. The ROP 826 can then output processed graphics data stored in graphics memory. The ROP 826 can include compression logic to compress depth or color data written to memory and decompress depth or color data read from memory. This compression logic can be lossless, employing one or more of several compression algorithms. The type of compression performed by the ROP 826 can vary based on the statistical characteristics of the data being compressed. For example, delta color compression is performed on depth and color data on a per-tile basis.

[0062] ROP 826 can be used within any processing cluster (e.g., clusters 814A - 814N of Fig. 8A) instead of being contained within partitioning unit 820. Read and write requests for pixel data can be submitted via memory crossbar 816 instead of pixel fragment data. Processed graphics data can be displayed on a screen, which is then routed to processor(s) 1602 for further processing, or to one of the processing entities within parallel processor 800. Fig. 8A will be routed.

[0063] In at least one embodiment, the parallel processor 800 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1- Fig. 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0064] Fig. Figure 8B includes a block diagram of a processing cluster 814 within a parallel processing unit according to at least one embodiment. A processing cluster can be an instance of one of the processing clusters 814A to 814N of Fig. 8A can be used to perform any of the operations described above or elsewhere herein. Processing Cluster 814 can be configured to run many threads in parallel, where "thread" refers to an instance of a single program running on a single set of input data. Single-instruction-multiple-data (SIMD) instruction output techniques can be used to support the parallel execution of a large number of threads without providing multiple independent instruction units. Single-instruction-multiple-thread (SIMT) techniques can be used to support the parallel execution of a large number of generally synchronized threads, using a common instruction unit configured to issue instructions to a set of processing engines within each of the processing clusters.

[0065] The operation of processing cluster 814 can be controlled via a pipeline manager 832, which distributes processing tasks to parallel SIMT processors. Pipeline manager 832 can receive instructions from scheduler 810. Fig. 8A receives and manages the execution of these instructions via a graphics multiprocessor 834 and / or a texture unit 836. The graphics multiprocessor 834 can be an example instance of a parallel SIMT processor. However, different types of parallel SIMT processors of different architectures can be contained within the processing cluster 814. One or more instances of the graphics multiprocessor 834 can be contained within the processing cluster 814. The graphics multiprocessor 834 can process data, and a data crossbar 840 can be used to distribute processed data to one of several possible destinations, including other shader units. The pipeline manager 832 can facilitate the distribution of processed data by specifying destinations for processed data to be distributed via the data crossbar 840.

[0066] Each 834 graphics multiprocessor within an 814 processing cluster can include an identical set of functional execution logic (e.g., arithmetic logic units, load-store units, etc.) to perform computations for any of the operations described above or elsewhere herein. Functional execution logic can be configured in a pipelined manner, allowing new instructions to be issued before previous instructions have completed. Functional execution logic can support a wide variety of operations, including integer and floating-point arithmetic, comparison operations, Boolean operations, bit shifting, and the computation of various algebraic functions. The same hardware functional unit can be used to perform different operations, and any combination of functional units can be present.

[0067] Instructions sent to Processing Cluster 814 can constitute a thread, also known as a warp, subgroup, wave, or wavefront. A set of threads executing across a set of parallel processing engines can be called a thread group. A thread group can execute a common program on different input data. Each thread within a thread group can be assigned to a different processing engine within a Graphics Multiprocessor 834. A thread group can contain fewer threads than the number of processing engines within the Graphics Multiprocessor 834. If a thread group contains fewer threads than the number of processing engines, one or more of the processing engines may be idle during cycles in which that thread group is being processed.A thread group can contain more threads than the number of processing engines within the 834 graphics multiprocessor. If a thread group contains more threads than the number of processing engines within the 834 graphics multiprocessor, processing can be performed over successive clock cycles. Multiple thread groups can run concurrently on a single 834 graphics multiprocessor.

[0068] The 834 graphics multiprocessor includes an internal cache memory for performing load and store operations, such as, but not limited to, any of the operations described above or elsewhere herein. The 834 graphics multiprocessor may forgo an internal cache and use a cache memory (e.g., the 848 L1 cache) within processing clusters 814. Each 834 graphics multiprocessor may also access L2 caches within partitioning units (e.g., partitioning units 820A–820N of the 814 series). Fig. 8A) which are shared by all 814 processing clusters and can be used to transfer data between threads. The 834 graphics multiprocessor can also access off-chip global memory, which may include one or more local parallel processor memories and / or system memories. Any memory outside of the 802 parallel processing unit can be used as global memory. The 814 processing cluster can contain multiple instances of the 834 graphics multiprocessor and can share instructions and data that can be stored in the 848 L1 cache.

[0069] Each 814 processing cluster can contain an 845 MMU (memory management unit), which can be configured to map virtual addresses to physical addresses. One or more instances of an 845 MMU can be located within a 818 memory interface. Fig. MMU 845 can reside in 8A. MMU 845 can contain a set of page table entries (PTEs) used to map a virtual address to a physical address of a tile and, optionally, a cache row index. MMU 845 can include address translation lookaside buffers (TLBs) or caches that may reside within Graphics Multiprocessor 834 or L1 848 Cache or Processing Cluster 814. A physical address can be processed to distribute surface data access locally, allowing efficient query nesting between partitioning units. A cache row index can be used to determine whether a query for a cache row is a hit or a miss.

[0070] Processing Cluster 814 can be configured such that each Graphics Multiprocessor 834 is coupled to a Texture Unit 836 for performing texture mapping operations, such as determining texture sampling positions, reading texture data, and filtering texture data. Texture data can be read from an internal texture L1 cache (not shown) or from an L1 cache within Graphics Multiprocessor 834 and retrieved from an L2 cache, local parallel processor memory, or system memory as needed. Each Graphics Multiprocessor 834 can output processed tasks to Data Crossbar 840 to provide the processed task to another Processing Cluster 814 for further processing or to store the processed task in an L2 cache, local parallel processor memory, or system memory via Memory Crossbar 816.A preROP 842 (preraster operations unit) can be configured to receive data from the 834 graphics multiprocessor and to direct data to ROP units that may be located with partitioning units as described herein (e.g., 820A-820N partitioning units from ). Fig. 8A). The PreROP 842 unit can perform color blending optimizations, organize pixel color data, and perform address translations.

[0071] In at least one embodiment, processing cluster 814 can include one or more circuits to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0072] Fig. Figure 8C shows a graphics multiprocessor 834 according to at least one embodiment, for example, to perform one of the operations described above or elsewhere herein. The graphics multiprocessor 834 can be coupled to a pipeline manager 832 of a processing cluster 814. The graphics multiprocessor 834 can include an execution pipeline, including, but not limited to, an instruction cache 852 (which can, for example, store instructions such as compiled API instructions), an instruction unit 854, an address mapping unit 856, a register file 858, one or more general-purpose graphics processing unit (GPGPU) cores 862, and one or more load / store units 866, wherein one or more load / store units 866 can perform load / store operations to load / store instructions according to the execution of an operation.GPGPU cores 862 and load / store units 866 can be coupled via a memory and cache assembly 868 with cache memory 872 and shared memory 870. GPGPU cores 862 can be part of a SoC, such as, but not limited to, integrated circuit 700. Fig. 7.

[0073] Instruction Cache 852 can receive a stream of instructions (e.g., to perform any of the operations described above or elsewhere herein) for executing Pipeline Manager 832. Instructions can be cached in Instruction Cache 852 and dispatched for execution by an Instruction Unit 854. Instruction Unit 854 can dispatch instructions as thread groups (e.g., warps, subgroups, wavefronts, or waves), with each thread in a thread group being assigned to a different execution unit within GPGPU Cores 862. An instruction can access any address space from a local, shared, or global address space by specifying an address within a unified address space.The Address Mapping Unit 856 can be used to translate addresses in a unified address space into a different memory address that can be accessed by the Load / Storage Unit 866.

[0074] Register file 858 can provide a set of registers for functional units of the 834 graphics multiprocessor. Register file 858 can provide temporary storage for operands associated with data paths of functional units (e.g., 862 GPGPU cores, 866 load / store units) of the 834 graphics multiprocessor. Register file 858 can be subdivided among each of the functional units, so that each functional unit is allocated a dedicated portion of register file 858. Register file 858 can be subdivided among different warps (which may be referred to as wavefronts, subgroups, and / or waves or threads) executed by the 834 graphics multiprocessor.

[0075] GPGPU 862 cores can each contain floating-point units (FPUs) and / or integer arithmetic logic units (ALUs) that can be used to execute instructions from the 834 graphics multiprocessor. GPGPU 862 cores can be similar or different in architecture. One set of GPGPU 862 cores can contain a single-precision FPU and an integer ALU, while a second set of GPGPU cores can contain a double-precision FPU. FPUs can implement IEEE 754-2008 standard floating-point arithmetic or enable variable-precision floating-point arithmetic. The 834 graphics multiprocessor can additionally include one or more fixed-function or special-purpose function units to perform specific functions, such as, but not limited to, copy-rectangle or pixel blending operations. One or more of the GPGPU 862 cores can also contain fixed or special function logic.

[0076] GPGPU Cores 862 can include SIMD logic capable of executing a single instruction on multiple sets of data. GPGPU Cores 862 can physically execute SIMD4, SIMD8, and SIMD16 instructions and logically execute SIMD1, SIMD2, and SIMD32 instructions. SIMD instructions for GPGPU Cores can be generated at compile time by a shader compiler or automatically when running programs written and compiled for Single-Program-Multiple-Data (SPMD) or SIMT architectures. Multiple threads of a program can be configured for a SIMT execution model that can be executed via a single SIMD instruction. For example, eight SIMT threads performing the same or similar operations can run in parallel over a single SIMD8 logic unit.

[0077] Memory and cache interconnect 868 can include a interconnect network that connects each functional unit of the graphics multiprocessor 834 to register file 858 and shared memory 870. Memory and cache interconnect 868 can be a crossbar interconnect, allowing load / store unit 866 to perform load and store operations between shared memory 870 and register file 858. Register file 858 can operate at the same frequency as GPGPU cores 862, thus enabling very low-latency data transfer between GPGPU cores 862 and register file 858. Shared memory 870 can be used to enable communication between threads running on functional units within the graphics multiprocessor 834. Cache memory 872, for example, can be used as a data cache to temporarily store texture data communicated between functional units and texture unit 836.Shared memory 870 can also be used as a program-managed cache. Threads running on GPGPU cores 862 can programmatically store data within shared memory in addition to automatically cached data stored within cache memory 872.

[0078] A parallel processor or GPGPU, as described herein, can be communicatively coupled to host / processor cores to accelerate graphics operations, machine learning operations, pattern analysis operations, and various general-purpose GPU (GPGPU) functions. A GPU can be communicatively coupled to host processor / cores via a bus or other interconnect (e.g., a high-speed interconnect such as, but not limited to, PCIe or NVLink). A system-on-a-chip (SoC) can incorporate a parallel processor or GPGPU, as described herein, with the parallel processor or GPGPU operating on the SoC. A GPU can be integrated as cores on a package or chip and communicatively coupled to cores internally to a package or chip via an internal processor bus / interconnect.Regardless of how a GPU is connected, processor cores can allocate work to such a GPU in the form of sequences of instructions contained in a work descriptor. The GPU can then use dedicated switching technology / logic to efficiently process these instructions to perform any of the operations described above or elsewhere herein.

[0079] In at least one embodiment, the graphics multiprocessor 834 can include one or more circuits for using one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1- Fig. 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0080] Fig. Figure 9 shows a Processor 900 according to at least one embodiment. Processor 900 may include a hybrid-architecture processor (e.g., Lunar Lake or Meteor Lake) from Intel Corporation in Santa Clara, CA, or any other processor that shares at least some of the components described herein. Processor 700 may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more neural processing units (NPUs), which may, for example, be a dedicated AI accelerator that offloads artificial intelligence (AI) workloads from the CPU and GPU. Processor 700 may use instructions that, when executed, cause Processor 700 and / or any of its components to perform some or all of the processes and techniques described elsewhere herein.The Processor 900 can include any number of memory and cache units 910 to facilitate processing among the various components. Memory and cache 910 on the Processor 900 can include one or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination. With respect to the Processor 900 and any of its components described above or elsewhere herein, one or more of the APIs described herein can, for example, be compiled into instructions that are retrieved by instruction call logic or equivalent, decoded by a processor decoder or equivalent, and scheduled for execution by a scheduler or equivalent (e.g.,APIs (in order or out of order) can be executed by execution logic or equivalents, reordered, and then shut down by shutdown logic or equivalents. APIs (and / or compiled instructions containing APIs) can be stored in any memory outside or inside the processor (e.g., cache and / or memory). A result of APIs can then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory. One or more of the APIs described herein can contain a call.

[0081] The 900 series processor can incorporate compute engines as CPUs 902 and can include any number of cores, such as, but not limited to, up to 16 cores / 22 threads. Cores in CPU 902 can be P cores (Performance), E cores (Efficient), and LP-E cores (Low-Power Efficient). Performance cores can be used for single-threaded, computationally intensive workloads with low latency, while Efficient cores can be used for multi-threaded, less computationally intensive workloads. Low-Power Efficient cores can be used for scalable multi-threaded performance and offloading background tasks. P cores can be used for single- and limited-threaded performance, while E and LP-E cores can be used for multi-threaded throughput and power efficiency.

[0082] GPU 906 can include any number of graphics engines, such as, but not limited to, Intel® Arc™ Graphics Engines (Xe LPG) with 8 Xe cores (up to 128 execution units or EUs). As shown in Fig. As shown in Figure 9, the GPU 906 can include vector engines 910 and matrix engines 912, which can, for example, execute FP, INT, and matrix operations all simultaneously, separately, or in batches. The GPU 906 can include a load / store unit 914, as well as other memory, such as, but not limited to, an instruction cache (I$) 916 and L1 cache / subsystem local memory (SLM) 918, which can, for example, store instructions to perform any of the operations described above or elsewhere herein.

[0083] The NPU 904 can include one or more Intel® AI Boost built-in neural processing units (NPUs). The NPU 904 can be enumerated as an integrated PCIe device for the host processor. The NPU 904 can include one or more (e.g., two) neural computing engine (NCE) tiles 930. Each tile can be configured with any combination of, but not limited to, (e.g., 2000) Multiply Accumulate (MAC) engines 934, a post-processing engine (not shown), an AI DSP processor (not shown), and memory (2 MB dedicated SRAM) per tile, as shown in Fig. Figure 9 shows that for general computing needs, neural computing engines can include streaming hybrid architecture vector engines (SHAVE) for high-performance parallel computing, which can include DMA (Direct Memory Access) engines to shuttle data between system memory DRAM (Dynamic Random Access Memory) and a software-managed cache. Built-in device MMUs (Memory Management Units) plus IOMMUs (Input / Output Memory Management Units) (not shown) can support multiple concurrent hardware contexts and provide security isolation between execution contexts according to the MCDM (Microsoft Compute Driver Model) architecture.The Processor 900 may also include a media unit (not shown) that is included on or separate from the XCDs or other components of the processor to enable video playback and video processing of compressed or uncompressed data, such as using HEVC, AV1, VP9 and AVC-HW accelerated decoding support and HEVC, VP9 and AVC-HW accelerated encoding support.

[0084] An Intel® Thread Director, which includes firmware built into the processor, can prioritize and manage workload distribution by sending tasks to optimized cores. For example, Thread Director can group P cores, E cores, and / or LP-E cores (described above) together, with task scheduling capabilities and the ability to send less demanding tasks to the E cores or LP-E cores. Intel® Deep Learning Boost (Intel® DL Boost) (not shown) can provide built-in AI acceleration for training and inference workloads and can include VNNI (for CPU) and DP4a (for GPU) instruction set support. This instruction set can be optimized with OpenVINO™ Toolkit and oneAPI to accelerate INT8 inference. A software stack, such as... B. as described elsewhere herein, can be used to enable AI inference using OpenVINO™ Toolkit.The Processor 900 can be configured to run an application program, such as, but not limited to, a CUDA program.

[0085] In at least one embodiment, processor 900 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1- Fig. 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0086] Processor 900 may alternatively include a processor based on Qualcomm Corporation's AI Engine Direct architecture in Santa Clara, CA, or another processor that shares at least some of the components described herein. It may include any number of NPUs, GPUs, CPUs, and other related components, such as, but not limited to, NPU 904 as a Hexagon NPU, GPU 906 as an Adreno GPU, CPU 902 as a Kryo or Qualcomm Oryon CPU, as well as a Qualcomm Sensing Hub (not shown) and a Memory Subsystem 910 in any combination. Hexagon NPU 904 may include a power rail, a microtile inference unit, a hardware acceleration unit, a tensor unit, a scalar unit, and a vector unit (all not shown), which may include dedicated memory or share memory (e.g., cache or storage, such as HBM3) to, for example,Instructions for performing any of the operations described above or elsewhere herein can be stored. The Adreno GPU 906 can provide graphics and parallel processing for AI in formats such as, but not limited to, 32-bit floating point (FP32), 16-bit floating point (FP16), and 8-bit integer (INT8). Kryo or Qualcomm Oryon CPUs 902 can perform AI workloads and can handle contextualization for ubiquitous generative AI applications. The CPU 902 can also include an instruction fetch unit, a rename and shut-down unit, a memory management unit, a vector execution unit, an integer execution unit, and a load and store unit for processing and instruction management.With respect to Processor 900 and any of its components described above or elsewhere herein, one or more of the APIs described herein may, for example, be compiled into instructions that may be fetched by the Instruction Fetch Unit, decoded by a Processor Decoder or equivalent, scheduled for execution by a Scheduler or equivalent (e.g., in sequence or out of sequence), executed by Execution Logic or equivalent, reordered, and then shut down by the Rename and Shutdown Unit. API(s) (and / or compiled instructions containing API(s)) may be stored in any memory outside or inside the Processor (e.g., in cache and / or memory).Any number of CPU 902 cores can be contained in any number of CPU clusters, which can be coupled with memory and / or cache, such as, but not limited to, a shared L2 cache. Memory can be used separately or shared; for example, CPU clusters of CPU 902 cores can be coupled with Memory Subsystem 910, which can include fabric, system-level cache, and any number of memory management units capable of reading and writing memory (e.g., DRAM). The Qualcomm Sensing Hub (not shown) includes micro-NPUs, a power rail, and traditional sensors (a gyroscope, accelerometer, even a barometer) carrying voice and data streams. Memory Subsystem 910 can include memory and cache on Processor 900, which can include one or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g.,HBM2e or HBM3) in any combination, for example, to store information and / or instructions to perform any of the operations described above or elsewhere herein. All or part of the memory and / or cache in Memory Subsystem 910 can be shared or used individually by any component or combination of components (e.g., GPU 906, NPU 904, and CPU 902) on Processor 900.

[0087] The Qualcomm AI Engine 900 can be programmed and controlled with a software stack to perform some or all of the operations described herein. This stack includes, for example, a Qualcomm® Neural Processing SDK for inference, with versions for Android, Linux, and Windows. Developer libraries and services support the latest programming languages, virtual platforms, and compilers. At a lower level of the software stack, system software includes the basic real-time operating system (RTOS), system interfaces, and drivers. The software stack supports various operating systems, including Android, Windows, Linux, and QNX, and deployment and monitoring infrastructure such as Prometheus, Kubernetes, and Docker. OpenCL and DirectML are supported for direct cross-platform access to the GPU. For the CPU, LLVM compiler infrastructure optimizations enable accelerated and efficient AI inference.With respect to Qualcomm AI Engine 900 and any of its components described above or elsewhere herein, one or more of the APIs described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions containing API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory). A result of API(s) may then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory.

[0088] In at least one embodiment, Processor 900 or Qualcomm AI Engine 900 may include one or more circuits to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0089] Fig. Figure 10A illustrates a processor 1000 according to at least one embodiment. Processor 1000 may include a processor from Intel Corporation's Scalable Family of processors in Santa Clara, CA, or any other processor that shares at least some of the components described herein. Processor 1000 may include one or more cores 1012(1)-1012(N), where N is any integer greater than 1, capable of performing the operations described elsewhere herein. Cores 1012(1)-1012(N) may be interconnected using ring and / or mesh assemblies. The mesh assembly architecture allows an array of vertical and horizontal communication paths to permit traversal from one core to another 1012(1)-1012(N) via a shortest path (jumping along the vertical path to the correct row and jumping along the horizontal path to the correct column).For mesh clusters, a die can accommodate cores 1012(1)-1012(N) and can contain a grid of converged mesh stops (CMS) that can be associated with cores 1012(1)-1012(N) (e.g., 1:1). Each core can be associated with a lower-level cache (LLC) slice 1014(1)-1014(N), or cores 1012(1)-1012(N) can share cache, e.g., lower-level cache. LLCs 1014(1)-1014(N) can be inclusive, incorporating blocks into higher-level cache (e.g., L2 cache), or non-inclusive (with blocks that may not be present in higher-level cache). Each core and LLC slice can include a Caching and Home Agent (CHA) (not shown) that can maintain cache coherence by providing resource scalability across mesh networks for Intel® Ultra Path Interconnect (Intel® UPI 1016) cache coherence functionality.UPI 1016 can provide a coherent association for scalable systems and can allow multiple processors to share a single shared address space through links, such as, but not limited to, two or three UPI links per processor.

[0090] Processor 1000 may also include System Agent 1010, which can accommodate and / or perform various functionalities, such as, but not limited to, memory management, display functions, and / or input / output (I / O) functions. For example, Processor 1000 may include one or more integrated memory controllers (IMCs) 1008. IMC 1008 can control and manage memory, such as, but not limited to, various memory types, e.g., DDR RAM, such as DDR4, or others described elsewhere herein. System Agent 1010 may include a display controller (not shown) to support display(s). System Agent 1010 may also incorporate PCIe 1004 (e.g., up to 20 PCIe lanes), which can connect, for example, to an external dedicated graphics hookup via DMI bus (e.g., Intel's DMI 3.0 bus) 1006.System Agent 1010 can include an Image Processing Unit (IPU) (not shown) that incorporates an on-die image signal processor (ISP). Fabric 702 can provide connectivity scalability.

[0091] Fig. Figure 10B illustrates components within core 1012 according to at least one embodiment. Core 1012 may include front-end 1018, back-end or execution engine 1032, and memory subsystem 1042. Front-end 1018 may provide execution engine 1032 with operations (e.g., operations described elsewhere herein) by decoding instructions stored in memory. For example, front-end 1018 may include a micro-operations (µOps) cache path and / or a legacy path, along with a branch prediction unit 1020 that can determine path instructions. A legacy path for instructions may involve retrieving variable-length instructions (e.g., x86) from the L1 instruction cache, queuing the instructions into instruction queue 1024, and decoding instructions using decoder 1026 in µOps, which can then be provided to allocation queue 1028.Alternatively, a µOPs cache path can include a cache containing pre-decoded µOps (µOps 1030) that can be sent to allocation queue 1028. Allocation queue 1028 can act as an interface between front end 1018 and execution engine 1032 and can provide instructions to execution engine 1032. For example, one or more of the API(s) described herein can be compiled into instructions that can be stored, processed, and executed by front end 1018 and execution engine 1032, and stored in storage subsystem 1042.

[0092] Execution Engine 1032 can receive micro-operations in Reorder Buffer 1034, which can register the allocation, renaming, and shutdown of micro-operations. From the Reorder Buffer, micro-operations can be sent to Scheduler 1036, which can be connected to one or more different Execution Units 1038. Execution Units 1038 can perform, for example, basic arithmetic logic unit (ALU) operations, multiplication, division, and / or more complex operations, such as, but not limited to, various vector operations. Scheduler 1036 can manage the queuing of micro-operations for one or more Execution Units 1038, for example, depending on the operations that need to be performed.

[0093] Storage subsystem 1042 can handle load and store requests as well as ordering operations. For example, micro-operations (µOPs) can relate to memory access (e.g., load and store), and these can be sent to dedicated scheduler ports that can perform these memory operations. Store and load operations can, for example, be sent to load and store buffers 1044. Storage subsystem 1042 can also include shared or separate L1 data and instruction caches 1046 and L2 caches 1048, which can be used and shared by L1 data and instruction caches 1046. As above for Fig. As described in 10A, each 1012 core can be connected to a slice of a third-level cache (e.g., LLC 1014) that can be shared by all 1012 cores.

[0094] In at least one embodiment, processor 1000 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1- Fig. 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0095] Fig. Figure 11 illustrates an AI Accelerator 1100 according to at least one embodiment. Processor 900 may include a processor with AI Accelerator architecture from Intel Corporation in Santa Clara, CA, or any other processor that shares at least some of the components described herein. AI Accelerator 1100 may use instructions which, when executed by AI Accelerator 1100, cause AI Accelerator 1100 to perform some or all of the processes and techniques described elsewhere herein. With respect to AI Accelerator 1100 and any of its components described above or elsewhere herein, one or more of the APIs described herein may, for example, be compiled into instructions that are retrieved by instruction-fetching logic or equivalents, decoded by a processor decoder or equivalents, scheduled for execution by a scheduler or equivalents (e.g.,APIs (and / or compiled instructions containing APIs) can be executed (in order or out of order) by execution logic or equivalents, reordered, and then shut down by shutdown logic or equivalents. APIs can be stored in any memory outside or inside the processor (e.g., cache and / or memory). A result of APIs can then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory. The AI ​​Accelerator 1100 can include one or more compute dies, which can contain homogeneous or heterogeneous processors. Compute dies can contain one or more central processing units (CPUs), one or more graphics processing units (GPUs), or combinations of both.

[0096] In at least one embodiment, compute dies can include compute engines for performing AI computations. In at least one embodiment, compute dies of AI accelerator 1100 can be divided into any number (e.g., four) clusters, which can be referred to as a DCORE (Deep Learning Core) 1106 and contain any number of matrix multiplication engines (MMEs) 1108, tensor processor cores (TPCs) 1110, and L2 cache 1114 in any combination. MME(s) 1108 can perform operations that use matrix multiplication, such as fully connected layers, convolutions, and batched-general matrix multiplications (GEMMs).MMEs 1108 can be equipped with multiply-accumulate units (MACs) (not shown) that can perform, for example, general matrix multiplication (GEMM) operations, such as, but not limited to, an AxB multiplication, which involves generating tensor C[NxM] from two input tensors, A[NxK] and B[KxN]. MMEs 1108 can be programmed with array dimensions, locations, data types, and various execution operands. MMEs 1108 can retrieve tensors A and B from memory and pull them into their streaming buffers for the matrix multiplication, which is to be performed in parallel by the MACs. MMEs 1108 can push tensor C back into memory upon completion. TPC(s) 1110 can have any number of scalar units for performing scalar operations, any number of vector units for performing vector operations, any number of register files or local storage units (e.g.TPCs include a local vector memory and instruction load and store components that can be coupled with memory or cache (e.g., HBM, L3 cache, and / or L2 cache) (all not shown). TPCs can support various types of parallel processing, such as Very Long Instruction Word (VLIW) and Single-Instruction-Multiple-Data (SIMD), which supports data types such as, but not limited to, FP32, BF16, FP16 & FP8 (both E4M3 and E5M2), UINT32, INT32, UINT16, INT16, UINT8, and INT8 data types. Any number of compute dies can be connected by a federation. Federation between compute dies can be achieved via an interposer bridge, which is transparent to software, for example.

[0097] Memory on the AI ​​Accelerator 1100 can include one or more levels of cache (e.g., L1, L2, L3, and / or last-level cache) and high-bandwidth memory (e.g., HBM2e or HBM3) in any combination. Memory and / or cache systems can be unified or separate. Compute dies of the AI ​​Accelerator 1100 can include on-die memory containing one or more levels (e.g., two levels) of cache. On-die SRAM or other memory described elsewhere herein can be used as a uniformly accessible last-level (L3) cache or divided into slices of L2 cache that can be accessed by groups of MMEs 1108 and TPCs 1110. The use of on-die memory as L2 or L3 cache can be fully software-configurable, allowing the software to dynamically determine its optimal cache allocation per I / O tensor.The AI ​​Accelerator 1100 can include one or more Memory Management Units (MMUs) 1122 for managing memory, such as allowing the AI ​​Accelerator 1100 memory subsystem to operate in a virtual space when accessing VRAM.

[0098] The AI ​​Accelerator 1100 can include a communication port (e.g., a PCIe Gen5 x16 port) 1102 for communicating with a host and a Scheduling and Synchronization Unit 1104. The AI ​​Accelerator 1100 can include a Media Unit 1116, which can contain any number or combination of Media Decoder Engines (DECs) 1120 and Rotator Engines (ROTs) 1118. The AI ​​Accelerator 1100 can include a Network Unit 1124, which can contain any number or combination of Network Ports 1126 and the accompanying RDMA Engine(s) 1128, L2 cache, and memory (e.g., HBM2e or HBM3) stacks. The AI ​​Accelerator 1100 can incorporate a programmable control path entity (not shown) to manage the parallel and efficient execution of different engines.Control Path can include Submission Queues (SQs) that can be issued by the runtime system, Completion Queues (CQs) that can be used for job completion notification, a Programmable Scheduling Mechanism that can be used for task scheduling, a Programmable Hardware Synchronization Mechanism or 'Sync Manager (SM)' that can be used for hardware synchronization, and a Programmable Interrupt Service Mechanism or 'Interrupt Manager (INTR)' that can enable the passing of asynchronous events to drivers.

[0099] The AI ​​Accelerator 1100 can include media decoding units that support video formats such as, but not limited to, HEVC, Progressive H.264, SVC Base Layer, MVC, VP9, ​​JPEG, and Progressive JPEG. The AI ​​Accelerator 1100 can support post-processing of decoded media streams, such as, but not limited to, downscaling (resizing the image), vertical and horizontal scaling at different scaling ratios, upscaling, cropping, bilinear scaling, and Lancos scaling. The AI ​​Accelerator 1100 can implement two post-processing channels per decoding unit: one with scalar (top and bottom) processing and one for outputting the original image only.The AI ​​Accelerator 1100 can include a hardware rotator engine that performs the following transformations of an input image: 2D rotation, 3D rotation, projection, distortion and non-distortion of images, resample of input data at user-defined coordinates, and rescaling.

[0100] RDMA 1128 over Converged Ethernet to AI Accelerator 1100 can enable scaling from a single node (i.e., a single AI Accelerator 1100) to hundreds or thousands of nodes or AI Accelerators 1100. The NW subsystem 1124 can include an Intel® Gaudi® Communication Library (IGCL), a master conductor that orchestrates data movement, and a programmable scheduling mechanism that enables smooth engine activation while maintaining task dependencies. An accelerator network subsystem can include Gigabit Ethernet NIC ports 1126, a Layer 2 MAC (not shown), and RDMA engines 1128. The AI ​​Accelerator 1100 can include aggregation engines for performing summation activities. All engines in the processor 1100 can operate in parallel, e.g., For example, MME(s) 1108, TPC(s) 1110 and NIC(s) 1126 can all work at the same time.There can be dependencies between operations running on different engines; for example, the output of one engine can be used as the input of another, and / or the MME, TPC, and NIC can be scheduled to run in parallel. Once one engine has completed its executing operation, another engine can be scheduled to begin work on the next operation (as soon as its inputs are ready).

[0101] AI Accelerator 1100 can be operated and controlled using software layer 1128, which may include low-level components such as, but not limited to, a graph compiler, an automatic kernel fuser, and a library of pre-compiled kernels, as well as integration with AI ecosystems such as, but not limited to, PyTorch, DeepSpeed, Hugging Face, vLLM, Ray, and more, or as described elsewhere herein in relation to software and programming platforms. Software layer 1128 may include implementations of algorithms such as, but not limited to, paged attention, flash attention, and more. Software layer 1128 may generate optimized binary code that implements the given model topology, such as, but not limited to, performing operator fusion, data layout management, parallelization, pipelining and memory management, and graph layer optimizations.

[0102] In at least one embodiment, AI Accelerator 1100 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0103] A neuromorphic computing system is described that adopts a multi-core architecture, with each core housing the computing elements including neurons, synapses with on-chip learning capability, and local memory for storing synaptic weights and routing tables. Fig. Figure 12 is a simplified block diagram 1200 illustrating an example of at least one part of such a neuromorphic computing device 1205, according to at least one embodiment. The neuromorphic computing device 1205 may include a neuromorphic processor from Intel Corporation in Santa Clara, CA, or any other processor that shares at least some of the components described herein. As shown in this example, a device 1205 may be provided with a network 1210 consisting of multiple neural network cores interconnected by an on-device network, such that several different connections may potentially be defined between the cores. For example, a network 1210 may be provided with spiking neural network cores in the device 1205, and each core may communicate via short packetized spike messages sent from core to core over the network channels. Each core (e.g.,1215) may possess processing and memory resources and logic to implement a certain number of primitive nonlinear temporal computational elements, such as, but not limited to, multiple (e.g., 1000+) distinct artificial neurons (referred to herein as "neurons"). For example, each core may be capable of implementing multiple neurons simultaneously, so that the collection of neuromorphic cores may implement many multiples of neurons using the device. With respect to neuromorphic computational device 1205 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions or equivalents, which are retrieved by instruction-fetching logic or equivalents, decoded by a processor decoder or equivalents, scheduled for execution by a scheduler or equivalents (e.g.,APIs (and / or compiled instructions containing APIs) can be executed (in order or out of order) by execution logic or equivalents, reordered, and then shut down by shutdown logic or equivalents. APIs can be stored in any memory outside or inside the processor (e.g., cache and / or memory). A result of APIs can then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory equivalents.

[0104] Continuing with the example of Fig. A neuromorphic computing device 1205 can additionally include a processor 1220 and system memory 1225 to implement one or more components for managing and providing the device's functionality. For example, a system manager 1230 can be provided to manage global attributes and operations of the device (e.g., attributes affecting the network of cores 1210, multiple cores in the network, alliances of the device 1205 with other devices, managing access to global system memory 1225, among other possible examples). In one example, a system manager 1230 can define and provide a specific routing table to the various routers in the network 1210, orchestrate a network definition and attributes (e.g., weights, decay rates, etc.).), which are to be applied in the network, core synchronization and time-division multiplex management, routing of inputs to the appropriate cores, among other possible functions.

[0105] As another example, a neuromorphic computing device 1205 can additionally include a programming interface 1235 through which a user or system can specify a neural network definition to be applied (e.g., by means of a routing table and individual neuron properties) and implemented by the mesh 1210 of neuromorphic cores. A software-based programming tool can be provided with or separate from the neuromorphic computing device 1205, through which a user can provide a definition for a single neural network to be implemented using the network 1210 of neuromorphic cores. The programming interface 1235 can accept input from the programmer and then generate appropriate routing tables and local memory for individual neuromorphic cores (e.g.,1215) with the specified parameters to implement a corresponding, customer-specific network of artificial neurons, which is implemented by the neuromorphic cores.

[0106] In some cases, a neuromorphic computing device 1205 can advantageously be interfaced with other devices, including general-purpose computing devices, and thus work together in a compatible manner to realize certain applications and use cases. Accordingly, in some cases, external interface logic 1240 can be provided to communicate with one or more other devices (e.g., via one or more defined communication protocols). An external interface 1240 can be used to accept input data from another device or an external storage controller that acts as the source of the input data. An external interface 1240 can additionally or alternatively be used to allow the results or output of computations of a neural network implemented using the neuromorphic computing device 1205 to be transmitted to another device (e.g., a computer, a computer, a storage controller, or a computer).(another general-purpose processor implementing a machine learning algorithm) to enable additional applications and improvements, among other examples.

[0107] As in Fig. Figure 12 shows a network 1210 consisting of multiple neural network cores interconnected by an on-device network, illustrating a portion of a network fabric that connects multiple neuromorphic cores (e.g., 1215a-d). For example, a number of neuromorphic cores (e.g., 1215a-d) can be deployed in a mesh, with each core interconnected by a network that includes a number of routers (e.g., 1250). In one implementation, each neuromorphic core (e.g., 1215a-d) can be connected to a single router (e.g., 1250), and each router can be connected to at least one other router (as in Figure 1210). Fig. (shown in Figure 12). As an example, in a single implementation, four neuromorphic cores (e.g., 1215a-d) can be connected to a single router (e.g., 1250), and each of the routers can be connected to two or more other routers to form a multi-core mesh that allows each of the neuromorphic cores to connect to every other neuromorphic core in the device. Furthermore, since each neuromorphic core can be configured to implement multiple different neurons, the device's router network can similarly allow connections, or artificial synapses (or simply "synapses"), to be defined between any two of the potentially many (e.g., 30,000+) neurons defined using the network of neuromorphic cores provided in a neuromorphic computing device.

[0108] Fig. Figure 12 shows a block diagram illustrating internal components of an example implementation of a neuromorphic kernel 1215. In an example, a single neuromorphic kernel can implement a certain number of neurons (e.g., 1024) that share the neuromorphic kernel's architectural resources in a time-multiplexed manner. In an example, each neuromorphic kernel 1215 can include a processor block 1255 capable of performing arithmetic functions and routing in connection with the realization of a digitally implemented artificial neuron, as, but not limited to, explained herein.Each neuromorphic core 1215 can additionally provide local memory in which a routing table can be stored and accessed for a neural network. The accumulated potential of each soma of each neuron implemented using the core can be tracked, and the parameters of each neuron implemented by the core can be recorded, among other data and uses. Components or architectural resources of a neuromorphic core 1215 can further include an input interface 1265 for accepting input spike messages generated by other neurons on other neuromorphic cores, and an output interface 1270 for sending spike messages to other neuromorphic cores via the mesh network. In some cases, routing logic for the neuromorphic core 1215 can be implemented, at least partially, using the output interface 1270.Furthermore, in some cases, a single core (e.g., 1215) can implement multiple neurons within an exemplary SNN, and some of these neurons can be interconnected. In such cases, spike messages sent between the neurons hosted on that particular core can bypass communication via the neuromorphic computing device's routing fabric and instead be managed locally on the individual neuromorphic core.

[0109] Each neuromorphic core can additionally include logic to implement an artificial dendrite 1280 and an artificial soma 1285 (referred to herein simply as "dendrite" and "soma," respectively) for each neuron 1275. The dendrite 1280 can be a hardware-implemented process that receives spikes from the network. The soma 1285 can be a hardware-implemented process that receives the accumulated neurotransmitter amounts of each dendrite for the current time and develops each dendrite and the potential state of the soma to generate outgoing spike messages at appropriate times. A dendrite 1280 can be defined for each connection that receives input from another source (e.g., another neuron). In one implementation, the dendrite 1280 process can receive and handle spike messages as they arrive serially from the network in a time-division multiplexed manner.When spikes are received, the activation of the neuron (tracked using soma 1285 and local memory 1260) can increase. If the neuron's activation exceeds a threshold set for neuron 1275, the neuron can generate a spike message, which is propagated via output interface 1270 to a fixed set of fanout neurons. The network distributes the spike messages to all target neurons, and in response, these neurons can update their activations in a transient, time-dependent manner, potentially causing the activation of some of these target neurons to also exceed corresponding thresholds and trigger further spike messages, as in real biological neural networks.

[0110] As noted above, a neuromorphic computing device can reliably implement a spike-based model of neural computation. Such models can also be referred to as spiking neural networks (SNNs). In addition to neuronal and synaptic state, SNNs also incorporate the concept of time. For example, in an SNN, communication occurs via event-driven action potentials, or spikes, which convey no explicit information other than the spike time, as well as an implicit source-target neuron pair corresponding to the transmission of the spike. Computation takes place in each neuron as a result of the dynamic, nonlinear integration of weighted spike input. In some implementations, recursion and dynamic feedback can be incorporated within an SNN computational model.Furthermore, a variety of network connectivity models can be used to model different real-world networks or relationships, including fully connected (all-to-all) networks, feed-forward trees, completely random projections, and "small world" networks, among other examples. A homogeneous, two-dimensional network of neuromorphic kernels, such as, but not limited to, the one in the example of [reference missing], is one such example. Fig. As shown in Figure 12, the device can advantageously support all these network models. Since all cores of the device can be connected, all neurons defined in the cores can therefore also be fully connected via a certain number of router hops. Furthermore, the device can include fully configurable routing tables to define a variety of different neural networks by allowing the neurons of each core to distribute their spikes to any number of cores in the mesh, thus realizing completely arbitrary connectivity graphs.

[0111] In an improved implementation of a system capable of supporting SNNs, such as, but not limited to, the Very Large Scale Integration (VLSI) hardware device used in the example of Fig. As illustrated in Figure 9, high-speed and reliable circuits can be provided to implement SNNs to model the information processing algorithms used by the brain, but in a more programmable way. For example, while a biological brain can only implement a specific set of defined behaviors, as determined by years of development, a neuromorphic processor device can provide the ability to rapidly reprogram all neural parameters. Accordingly, a single neuromorphic processor can be used to realize a wider range of behaviors than those provided by a single slice of biological brain tissue.This distinction can be achieved by adopting a neuromorphic processor with neuromorphic design implementations that differ significantly from those of naturally occurring neural circuits.

[0112] As an example, a neuromorphic processor can utilize time-division multiplexing (TDM) in both the spike communication network and the device's neural machinery to implement SNNs. Accordingly, the same physical switching technology of the processor device can be shared among many neurons to achieve a higher neuron density. With TDM, the network can connect N cores with O(N) total wiring length, whereas discrete point-to-point wiring would scale as O(N²), resulting in a significant reduction in wiring resources to accommodate planar and non-plastic VLSI wiring technologies, among other examples. In the neuromorphic cores, TDM can be implemented through dense memory allocation, for example, using static random access memory (SRAM), with shared buses, address decoding logic, and other multiplexed logic elements.The state of each neuron can be stored in the processor's memory, with data describing each neuron's state including the state of each neuron's collective synapses, all currents and voltages across its membrane, along with other example information (such as, but not limited to, configuration and other information).

[0113] A neuromorphic processor can adopt a "digital" implementation, which differs from other processors that employ more "analog" or "isomorphic" neuromorphic approaches. For example, a digital implementation might integrate synaptic current using digital adder and multiplier circuits, in contrast to the analog isomorphic neuromorphic approaches that accumulate charge on capacitors in an electrically analogous way to how neurons accumulate synaptic charge on their lipid membranes. The accumulated synaptic charge could, for example, be stored for each neuron in the local memory of the corresponding nucleus.Furthermore, at the architectural level of an exemplary digital neuromorphic processor, reliable and deterministic operation can be achieved by synchronizing time across the network of cores, ensuring that any two implementations of the design, under the same initial conditions and configuration, produce identical results. Asynchronicity can be preserved at the circuit level to allow individual cores to operate as quickly and freely as possible, while maintaining determinism at the system level. Accordingly, the concept of time as a temporal variable in neural computations can be abstracted, separating it from the "wall clock" time used by the hardware to perform the computation. Consequently, some implementations can provide a time synchronization mechanism that globally synchronizes the neuromorphic cores at discrete time intervals.The synchronization mechanism allows the system to complete a neural computation as quickly as the circuitry allows, with a divergence between runtime and the biological time that models the neuromorphic system.

[0114] In operation, the neuromorphic mesh device can start in an idle state, with all neuromorphic cores inactive. Since each core cycles asynchronously through its neurons, it generates spike messages that the mesh assembly forwards to the appropriate target cores containing all the target neurons. Because the implementation of multiple neurons on a single neuromorphic core can be time-division multiplexed, a time step can be defined in which all spikes involving the multiple neurons can be processed and accounted for using the shared resources of a corresponding core. Since each core stops servicing its neurons for a given time step, in some implementations, the cores can communicate with neighboring cores using synchronization messages (e.g.,(using a handshake) to flush the mesh of all spike messages in flight, allowing the cores to safely determine that all spikes have been maintained for the time step. At this point, all cores can be considered synchronized, allowing them to advance their time step, return to the initial state, and begin the next time step.

[0115] Given this context and as introduced above, a device (e.g., 1205) can be provided that implements a mesh 1210 of interconnected neuromorphic cores, with the core potentially implementing multiple artificial neurons capable of being interconnected to implement a SNN. Each neuromorphic core (e.g., 1215) can provide two loosely coupled asynchronous processes: an input dendrite process (e.g., 1280) that receives spikes from the network and applies them to the appropriate target dendrite compartments at appropriate future times, and an output soma process (e.g., 1285) that receives the accumulated neurotransmitter amounts of each dendrite compartment for the current time and develops each dendrite and the potential state of the soma membrane, generating outgoing spike messages at appropriate times (e.g., when a soma threshold potential is reached).It should be noted that from a biological perspective, the dendrite and soma names used here only approximate the role of these functions and should not be interpreted too literally.

[0116] In at least one embodiment, neuromorphic computing device 1205 may include one or more circuits for using one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0117] Fig. Figure 13 is a block diagram of an embodiment of a multi-node network in which remote memory computation can be implemented, according to any embodiment. System 1300 can represent a network of nodes described herein, which can be used, for example, to perform some or all of the operations described herein. System 1300 can represent a data center. System 1300 can represent a server farm. System 1300 can represent a data cloud or a processing cloud. System 1300 can represent a supercomputer. System 1300 can include tens, hundreds, or thousands of nodes. The nodes of System 1300 can include processors, such as, but not limited to, central processing units (CPUs), graphics processing units (GPUs), or any combination of processors described herein, such as, but not limited to, other processors in Fig. 7-19. With respect to any of the processors in System 1300 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions containing API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory).A result from API(s) can then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory equivalents. System 1300 can include over nine thousand nodes, with each node containing two Intel Xeon Max processors, six Intel Max series GPUs, and a uniform memory architecture such as, but not limited to, that used in Intel Corporation's Intel Aurora supercomputer in Santa Clara, CA, or any other supercomputer that shares at least some of the components described herein.

[0118] One or more clients 1302 send requests to system 1300 via network 1304. Network 1304 represents one or more local area networks, wide area networks, or a combination thereof. Clients 1302 can be human or machine clients that generate requests for system 1300 to perform operations. System 1300 executes applications or data processing tasks requested by clients 1302.

[0119] System 1300 can include one or more racks, which represent structural and interconnected resources to house and connect multiple compute nodes. Rack 1310 can house multiple nodes 1330. Rack 1310 can host multiple blade components 1320. Hosting can refer to providing power, structural or mechanical support, and interconnection. Blades 1320 can refer to compute resources on printed circuit boards (PCBs), with one PCB housing the hardware components for one or more nodes 1330. Blades 1320 may or may not include a chassis, enclosure, or other "box" besides that provided by Rack 1310. Blades 1320 can include enclosures with exposed connectors for connecting within Rack 1310.System 1300 may or may not include Rack 1310, and each Blade 1320 may include a chassis or enclosure that can be stacked or otherwise reside in close proximity to other Blades, allowing for the aggregation of Node 1330. System 1300 can contain 10,624 compute Blades, which include 63,744 Intel Max Series GPUs and 21,248 Intel Xeon Max CPUs across 166 Racks.

[0120] System 1300 can include Fabric 1370, which represents one or more networks for nodes 1330. Fabric 1370 can include multiple switches 1372, routers, or other hardware for routing signals between nodes 1330. Additionally, Fabric 1370 can connect System 1300 to Network 1304 for access by clients 1302. Beyond routing equipment, Fabric 1370 can also be considered to include the cables, ports, or other hardware used to connect nodes 1330. Fabric 1370 can include one or more associated protocols for managing the routing of signals through System 1300. The protocol(s) depend, at least in part, on the hardware used in System 1300.

[0121] As illustrated, Rack 1310 can contain N Blades 1320. In addition to Rack 1310, System 1300 can contain Rack 1350. As illustrated, Rack 1350 can contain M Blades 1360. M is not necessarily the same as N; thus, it is understood that various different hardware equipment components could be used and coupled together in System 1300 via Fabric 1370. Blades 1360 can be the same as or similar to Blades 1320. Nodes 1330 can be any type of node as described herein and need not all be the same type of node. System 1300 is not restricted to being homogeneous, nor is it restricted to being heterogeneous.

[0122] A node in Blade 1320(0) is illustrated in detail. However, other nodes in System 1300 may be the same or similar. At least some nodes 1330 may be compute nodes with a processor 1332 and memory 1340. A compute node refers to a node with processing resources (e.g., one or more processors) that runs an operating system and can receive and process one or more tasks. At least some nodes 1330 may include storage server nodes with a server as processing resources 1332 and memory 1340. A storage server refers to a node with more storage resources than a compute node, and instead of having processors to perform tasks, a storage server includes processing resources to manage access to the storage nodes within the storage server.

[0123] Node 1330 can include an interface controller 1334, which can represent logic for controlling access by Node 1330 to Fabric 1370. This logic can include hardware resources for connecting to the physical federation hardware. It can also include software or firmware logic for managing the federation. Interface controller 1334 can be a host fabric interface or can include a fabric interface according to any embodiment described herein.

[0124] Node 1330 can contain storage subsystem 1340. Storage 1340 can contain memory computation resources (comp) 1342, which represent one or more capabilities through storage 1340 to perform memory computations. System 1300 enables remote memory operations, such as, but not limited to, those described elsewhere herein. Thus, node 1330 can request memory computations from remote nodes, with data for the computation remaining local to the executing node instead of being sent via Fabric 1370 or from the memory to the Fabric interface. In response to the execution of the memory computation, the executing node can provide a result to the requesting node.

[0125] Processor 1332 can include one or more separate processors. Each separate processor can include a single processing unit, a multi-core processing unit, or a combination thereof. The processing unit can be a primary processor, such as, but not limited to, a CPU (central processing unit), a peripheral processor, such as, but not limited to, a GPU (graphics processing unit), or a combination thereof. Memory 1340 can be or include memory devices and a memory controller.

[0126] The term "storage device" can refer to various types of storage. Generally, "storage device" refers to volatile memory technologies. Volatile memory is memory whose state (and therefore the data stored on it) is indeterminate when power to the device is interrupted. Non-volatile memory refers to memory whose state is definite, even when power to the device is interrupted. Dynamic volatile memory requires the data stored in the device to be refreshed to maintain its state. An example of dynamic volatile memory includes DRAM (Dynamic Random Access Memory) or any variant, such as, but not limited to, synchronous DRAM (SDRAM).A memory subsystem, as described herein, may be compatible with a number of memory technologies, such as, but not limited to, DDR3 (dual data rate version 3, originally published by JEDEC (Joint Electronic Device Engineering Council) on 27.June 2007, currently on Publication 21), DDR4 (DDR version 4, original specification, published by JEDEC in September 2012), DDR4E (DDR version 4, extended, currently under discussion by JEDEC), LPDDR3 (Low-Performance DDR version 3, JESD209-3B, Aug 2013 by JEDEC), LPDDR4 (Low-Performance Dual Data Rate (LPDDR) version 4, JESD209-4, originally published by JEDEC in August 2014), WIO2 (Wide I / O 2 (WideI02), JESD229-2, originally published by JEDEC in August 2014), HBM (High-Bandwidth Memory DRAM, JESD235, originally published by JEDEC in October 2013), DDR5 (DDR version 5, currently under discussion by JEDEC), LPDDR5 (currently under discussion from JEDEC), HBM2 (HBM version 2), currently discussed by JEDEC), or other or combinations of memory technologies and technologies based on derivatives or extensions of such specifications.

[0127] In addition to or as an alternative to volatile memory, in one embodiment, reference to storage devices may refer to a non-volatile storage device whose state is determined even when power to the device is interrupted. In one embodiment, the non-volatile storage device is a block-addressable storage device, such as, but not limited to, NAND or NOR technologies. Thus, a storage device may also include future-generation non-volatile devices, such as, but not limited to, a three-dimensional intersection point (3DXP) storage device, other byte-addressable non-volatile storage devices, or storage devices that utilize chalcogenide phase-change material (e.g., chalcogenide glass).In one embodiment, the storage device may be or include multi-threshold-level NAND flash memory, NOR flash memory, single- or multi-level phase-change memory (PCM) or phase-change memory with a switch (PCMS), resistive memory, nanowire memory, ferroelectric transistor random-access memory (FeTRAM), magnetoresistive random-access memory (MRAM) incorporating memristor technology, or spin-transfer-torque (STT) MRAM, or a combination of any of the above or other memory types.

[0128] In at least one embodiment, System 1300 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0129] Fig. Figure 14 illustrates an accelerated processing unit 1400 according to at least one embodiment. The accelerated processing unit 1400 may include a processor based on AMD Corporation's CDNA architecture in Santa Clara, CA, or another processor that shares at least some of the components described herein. The accelerated processing unit 1400 may include one or more accelerator complex dies (XCDs) 1404 for performing operations described elsewhere herein, such as, but not limited to, graphics processing and / or parallel processing, as well as instruction-level parallel computations, including support for a wide range of precisions (INT8, FP8, BF16, FP16, TF32, FP32, and FP64) and sparsity matrix data. In some cases, XCDs may be referred to as graphics compute dies (GCDs).The Accelerated Processing Unit 1400 can include one or more Complex Computing Dies (CCDs) 1406 for performing operations described elsewhere herein, such as, but not limited to, those operations performed by host processors. CCDs may in some cases be referred to as core complexes or CCXs, such as, but not limited to, CCXs used in AMD Ryzen processors. XCDs and CCDs can share any type of cache or memory (e.g., one or more Memory Units 1402) or have cache or memory allocated to each XCD or CCD or group of XCDs or CCDs. For example, on-package AMD Infinity Fabric combines XCDs and CCDs in shared AMD Infinity Cache 1408 and, in some embodiments, high-bandwidth memory (e.g., HMB3).The Accelerated Processing Unit 1400 can be an AMD MI300a processor, which includes three CPU chiplets (or CCDs) and six accelerator chiplets (XCDs) across four input / output dies (IODs). These can be stacked on a single piece of silicon, interconnected (e.g., via AMD Infinity Fabric) by eight stacks of high-bandwidth DRAM surrounding the superchip. An AMD MI300x processor replaces the CCDs with two additional XCDs for an accelerator-only system.

[0130] An Accelerated Processing Unit 1400 can include one or more input / output (I / O) interfaces. For example, XCDs 1404 and CCDs 1406 can reside together on one or more Input / Output Dies (IODs) 1410, which can include one or more I / O interfaces. IODs 1410 can include any number and any type of I / O interface (e.g., PCI, PCI-Extended (“PCI-X”), PCIe, Gigabit Ethernet (“GBE”), USB, etc.). Various types of peripheral devices can be connected to I / O interfaces 770. I / O interfaces of IODs 1410 can also be used for one or more interconnected Accelerated Processing Units 1400, for example, in a server architecture.

[0131] The Accelerated Processing Unit 1400 can include one or more Memory Units 1402 for storing instructions and other information used to perform operations described elsewhere herein. Memory Units 1402 can include any volatile memory, such as, but not limited to, memory types described elsewhere herein, and can include, for example, high-bandwidth memory (e.g., HMB3) or high-bandwidth DRAM. Memory associated with the Accelerated Processing Unit 1400 (e.g., Memory Units 1402) can include system memory, which can be used, for example, for instructions, statements, constants, and inputs and outputs. Memory Units 1402 can also include device memory, which can be used as storage and, for example, for instructions, statements, constants, and inputs and outputs, as well as a return buffer.return buffer) and can be used for private data. Memory units 1402 can be associated with one or more IODs 1410. In at least one embodiment, L1 cache 1420 initiates a memory hierarchy that includes shared L2 cache 1428, for example, within the XCDs. AMD Infinity Cache™ is a last-level cache (LLC) located on an active I / O die (IOD). CCDs 1406 and XCDs 1404 can have separate or shared memory. AMD Infinity Architecture and AMD Infinity Fabric™ technology enable coherent, high-throughput unification of GPU and CPU chiplet technologies (e.g., XCDs, CCDs, and / or CCXs) with memory (e.g., stacked HBM3 memory) in single devices and across multiple device platforms.

[0132] As in Fig. As shown in Figure 14, an XCD 1404 can include a shared set of global resources 1430, which can include hardware schedulers 1412 and asynchronous compute engines (ACEs) 1424 that send tasks (e.g., calculating shader workgroups) to compute units (CUs or cores) 1430. ACEs 1424 (e.g., four) can each be associated with CUs 1430 (e.g., 40 CUs), and some of the CUs can be disabled for yield management. CUs 1430 can include dedicated cache or shared cache (e.g., L2 cache) 1428, which can be used to aggregate all memory traffic for the die. CUs 1430 can include thread and parallel processor cores, including instruction retrieval and scheduling with Scheduler (S) 1412, Matrix Core Unit (MCU) 1416 and Shader Core (SC) 1418 (e.g., execution units for scalar, vector and matrix data types), as well as load / store pipelines with an L1 cache 1420 and Local Data Share (LDS) 1414.Local data sharing can include, for example, a scratch RAM with built-in computational capabilities that allow data to be shared between threads in a workgroup. An instruction cache 1440 (for example, for storing and providing instructions for performing operations described elsewhere herein) can be associated with one or more CUs and can be shared between two CUs. Matrix cores 1416 can process a variety of data types, such as, but not limited to, INT8, FP8, FP16, BF16, and TF32 data types. An accelerated processing unit 1400 can include compute units 1430, which can be arranged in an array format, for example, as a data parallel processor (DPP) array.The Ultra-Threaded Dispatch Processor 1442 can communicate with Computing Units 1430, and the Instruction Processor 1444 can read instructions written by the host to memory-mapped registers in a system memory address space (not shown). The Instruction Processor 1444 can send hardware-generated interrupts to a host processor (such as a CCD) when the instruction is complete. The Memory Controller 1436 can also have direct access to all device memory and the host-specified areas of system memory. To fulfill read and write requests, the Memory Controller 1436 can perform functions of a Direct Memory Access (DMA) controller, including calculating memory address offsets based on the format of the requested data in memory.For example, one or more of the APIs described herein can be compiled into instructions that are stored in Instruction Cache 1440 and then retrieved by Instruction Fetch Logic in Processor 1440, decoded by a Processor Decoder or equivalent, scheduled for execution by a Scheduler or equivalent (e.g., in sequence or out of sequence), executed by Execution Logic or equivalent, reordered, and then shut down by Shutdown Logic or equivalent. API(s) (and / or compiled instructions containing API(s)) can be stored in any memory outside or inside Processor 1400 (e.g., in cache and / or memory). A result of API(s) can then be stored in memory inside or outside the Processor, including registers, DRAM, Flash, SRAM, cache, or other memory equivalents.

[0133] An application can include a program that runs on a host processor (e.g., a CCD) and programs called kernels that run on one or more XCDs. Programs can be controlled by host instructions that set internal base addresses and other configuration registers, specify a data domain on which the Accelerated Processing Unit 1400 can operate, invalidate and flush caches on the Accelerated Processing Unit 1400, and cause the Accelerated Processing Unit 1400 to begin executing a program. Kernels can be referred to as programs executed by the Accelerated Processing Unit 1400. A kernel can be executed independently on each work item or as groups of work items, which can be called a wavefront, allowing the kernel to run on all work items in the group (e.g., 64) in a single pass.Computing units 1430 can include a scalar arithmetic logic unit (ALU) that can operate on one value per wavefront (common to all worker elements), a vector ALU that can operate on unique values ​​per worker element, a local data sharing 1414 that allows worker elements within a workgroup to communicate and share data, scalar memory (not shown) that can transfer data between scalar general-purpose registers (SGPRs) and memory through a cache, and vector memory that can transfer data between vector general-purpose registers (VGPRs) and memory, including sampling texture maps. Kernel control flow can be handled using scalar ALU instructions, which can include if / else branches and loops. Scalar ALU (SALU) and memory instructions can operate on an entire wavefront and on one or more SGPRs.Vector memory and ALU instructions can operate on all work elements in the wavefront simultaneously.

[0134] In at least one embodiment, the accelerated processing unit 1400 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0135] Fig. Figure 15 illustrates a Processor 1500, such as, but not limited to, a processor based on a Zen architecture (such as, for example, Zen 1, 2, 3, 4, 5, or others) from AMD Corporation in Santa Clara, CA, or any other processor that shares at least some of the components described herein. Processor 1500 includes one or more CPU dies 1502(1)-1502(N), where N is any integer greater than 1. CPU die 1502 can include any number of processor cores 1516 (for example, to perform any of the operations described elsewhere herein) and any number of cache memories (for example, to store instructions and other information to perform any of the operations described elsewhere herein) in any combination.For example, L2 cache units 1518 can be coupled to processor core(s) 1516, which can share L2 cache units 1518 and / or be individually coupled to them. Processor cores 1516 can individually be coupled to L3 cache 1522 and / or be shared L3 cache, which can be a low-level cache (LLC) 1522 for accessing data and other information used by the processor cores 1516. One or more processor cores 1516 and one or more L2 cache units 1518 can be contained in a core complex (CCX) 1520, which can contain (e.g., a 32 MB) shared cache (e.g., L3 cache 1522). Core complex 1520 can be manufactured on a die (CCD or CPU die) 1502. For example, up to 12 core complexes 1520 together with 8 CPU dies 1502 can be configured to form a processor, providing up to 96 processor cores 1516 for the processor.A "Zen 4c" core complex 1520, for example, can contain up to eight cores 1516 and a shared 16 MB L3 cache 1522. Two of these core complexes 1520 can be combined on a single CPU die 1502 for 16 cores per die and a total of 32 MB L3 cache 1522 per die. Up to eight of the CPU dies 1502 can be combined with an I / O unit 1504 to provide CPUs with up to 128 processor cores 1516. Up to four "Zen 4c" dies as described above can be combined to provide CPUs with up to 64 processor cores 1516.

[0136] The 1500 processor can include a variety of input / output configurations, which are described in more detail herein. The 1504 I / O unit can include one or more 1506 memory controllers that can manage memory usage (e.g., DDR5 memory) for the 1500 processor. The 1504 I / O unit can include one or more 1512 SATA disk controllers for managing memory and one or more 1514 Compute Express Link (CXL™) 1.1+ memory controllers that can provide CPU-to-device and CPU-to-memory connections and can be flexibly assigned specific functions at server design time. The 1504 I / O unit can include 1508 PCIe controllers for connecting peripherals and other components connected to the 1500 processor. The I / O unit 1504 can include USB ports 1510 for connecting to other components separate from the processor 1500. CPU dies 1502 can have any number of connections, e.g.Supports one or two connections to I / O Unit 1504. As shown, I / O Unit 1504 includes the components described herein, and I / O Unit 1504 can be an I / O die that accommodates several different components. Memory Controller 1506, PCIe Controller 1508, USB Ports 1510, SATA Controller 1512, and / or CXL Controller 1514 can be integrated anywhere within Processor 1500, either separately or in any groups or combinations thereof.

[0137] The 1500 processor can include Infinity Fabric 1524 assemblies (which may be similar to or based on PCIe architectures) that can provide connections between CPUs (e.g., CPU dies 1502(1)-1502(N)), GPU(s) 1526, inference engine(s) 1532, and other components in the multi-chip architecture, such as secure processor(s) 1528 and I / O unit 1504. One or more AMD Infinity Fabric™ 1510 assemblies can connect to CPU dies 1502(1)-1502(N) and serve as the connection used between CPUs. One or more Infinity Fabric 1510 assemblies can connect each CPU die 1502 to the I / O unit 1510.

[0138] In at least one embodiment, Processor 1500 may include central processing units (CPUs) and other associated hardware and software, as described above and further herein. Processor 1500 may also include graphics processing unit(s) 1526. Graphics processing unit 1526 may be used for image generation and processing, as well as other computations and operations, as further described herein. Graphics processing unit 1526 may be based on AMD's RDNA 3 or 3.5 architecture in Santa Clara, CA. Graphics processing unit 1526 may include graphics compute dies (GCDs) and memory cache dies (MCDs). GCDs may include any number of compute units (CUs) for graphics or other processing, such as operations performed by arithmetic logic units (ALUs), as further described herein. Graphics processing unit 1526 may include L2 cache that can be used by compute units.MCDs (not shown) can contain any number of memory units and can include cache, such as L3 cache, as well as memory interfaces for coupling to memory, such as Memory 1542(1)-(N), where N is an integer. Components within Graphics Processor 1526 can be interconnected using various approaches, such as using Infinity Fabric 1524 assemblies outside or inside Graphics Processor 1526.

[0139] Inference Engine 1532 can provide neural processing capabilities for Processor 1500 for computational processes used for neural networks, deep learning, and other operations related to artificial intelligence, which are further described herein. Processor 1500 can include a secure processor 1528 for managing processor security, a display controller 1530 for controlling displays, a system management unit 1534 for managing and operating some or all of the components on Processor 1500, multimedia engines 1536 for audio and video operations, a fusion controller hub 1538 for managing USB, SATA, and PCIe connections to the processor, and a sensor fusion hub 1540 for managing sensors, such as accelerometers. Processor 1500 can also include memory 1542(1)-(N), where N is any integer.Memory can include various memory types, such as LPDDR5 and / or DDR5 or others described elsewhere herein.

[0140] To perform the operations described herein, the Processor 1500 can include an execution pipeline containing a front end, which may include a cache (e.g., an L1 cache) that stores instructions (not shown). The flow of instructions can be modified by a branch predictor. Instructions can be decoded by a decoder, sent to a back end for execution, and renamed. Instruction fetch and decode pipes can be sent, for example, to integer or floating-point execution operations, which can be scheduled by a scheduler and transferred to vector and / or general-purpose registers. Floating-point multiplier and / or addition operations can be processed, and arithmetic logic units (ALUs) can also be used to perform calculations such as arithmetic and logical operations.Outputs from the computing units can be coupled with a load / store queue, which may be associated with cache, such as L1 cache and / or L2 cache.

[0141] With respect to Processor 1500 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions (e.g., AVX-512 instructions based on a SIMD model) that can be fetched by instruction call logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions containing API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory).A result from API(s) can then be stored in memory inside or outside the processor, including registers, DRAM, Flash, SRAM, cache or other memory equivalents.

[0142] In at least one embodiment, processor 1500 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0143] Fig. Figure 16 illustrates an example of a 1600 processing core that can implement Arm architecture (e.g., v9.0-A) or another processor that shares at least some of the components described herein. The Neoverse™ V2 1600 core can be implemented within a DynamIQ Shared Unit (DSU) cluster via DSU-110 Aggregation 1654 for one or more interconnected cores, for example, for parallel processing. The Neoverse™ V2 core can be implemented as a single core in a DSU cluster configured for Direct Connect, with or without L3 cache, snoop filtering, or snoop control unit (SCU) logic (not shown). The Neoverse™ V2 core can include a CPU bridge 1652, which connects core 1600 to a DSU-110 array, which can also connect core 1600 to an external storage system and the rest of a system-on-a-chip. The L1 instruction storage system 1602 can retrieve instructions from an instruction cache 1604 and execute the instructions (e.g.,one or more APIs described herein (which may be compiled into instructions) are delivered to an instruction decoding unit 1610, for example, to perform some or all of the operations described above or elsewhere herein. The L1 instruction storage system 1602 may include an L1 instruction cache 1604, for example, with 64-byte cache lines; an L1 instruction translation lookaside buffer (TLB) 1606, for example, with native support for 4KB, 16KB, 64KB, and 2MB page sizes; and a macro operation cache (MOP) 1608 (for example, a 1536-entry, 4-way shifted associative L0 MOP cache), which may contain decoded and optimized instructions for higher performance. The 1610 instruction decoding unit can decode AArch64 instructions into an internal format. The 1612 register renaming unit can perform register renaming to facilitate out-of-order execution and sends decoded instructions to various issue queues.Instruction Issuance Unit 1614 can control when decoded instructions can be sent to the execution pipelines, and it can include issue queues for storing instructions until they are sent to the execution pipelines. Integer Execution Pipeline 1616 can be included in an execution pipeline and can include Integer Execution Unit 1618, which can perform arithmetic and logical data processing operations. Vector Execution Unit 1620 can be included in an execution pipeline and can perform Advanced SIMD and Floating Point Operations (FPU) 1622, execute Scalable Vector Extension (SVE) and Scalable Vector Extension 2 (SVE2) instructions 1624, and can optionally execute Cryptographic Instructions (Crypto) 1626. Advanced SIMD can include media and signal processing architecture, adding instructions primarily for audio, video, 3D graphics, image, and speech processing.A floating-point architecture provides support for single-precision and double-precision floating-point operations. The L1 Data Storage System 1630 can execute load and store instructions as well as service memory coherence queries. The L1 Data Storage System 1630 can include an L1 Data Cache 1632 and a fully associative L1 Data TLB 1634 with native support for 4KB, 16KB, and 64KB page sizes and 2MB and 512MB block sizes. The Memory Management Unit (MMU) 1628 can provide fine-grained memory system control through a set of virtual-to-physical address mappings and memory attributes that can be held in translation tables, which can be stored in TLB 1634 when an address is translated. L2 storage system 1636 can include L2 cache 1638 and it can be connected to DSU-110 1654 by an asynchronous CPU bridge 1652.The Neoverse™ V2 Core 1600 can support a range of debugging, testing, and tracing options, including a Trace Unit 1642, a Trace Buffer 1640, and an Embedded Logic Analyzer (ELA) 1648. The Neoverse™ V2 Core 1600 can implement the Statistical Profiling Extension (SPE) 1644 to provide a statistical view of the performance characteristics of executed instructions, which software writers can use to optimize their code for better performance. The Performance Monitoring Unit (PMU) 1646 can provide performance monitoring, which can be configured to collect statistics on the operation of each core and the memory system. This information can be used for debugging and code profiling. The Generic Interrupt Controller (GIC) CPU Interface 1650, when integrated with an external distributor component, can be a resource for supporting and managing interrupts in a cluster system.In a cluster, there can be a CPU Bridge 1652 between each Neoverse™ V2 Core 1600 and DSU-110 1654. CPU Bridge 1652 can control buffering and synchronization between Core 1600 and the DSU-110 1654. CPU Bridge 1652 can be asynchronous to allow different frequency, power, and area implementation points for each Core 1600. CPU Bridge 1652 can run synchronously without affecting other interfaces, such as, but not limited to, debug and trace, which can be asynchronous.

[0144] In at least one embodiment, Core 1600 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0145] Fig. Figure 17 illustrates one or more chips incorporating one or more tensor processing units (TPUs) 1700 according to at least one embodiment. TPUs 1700 in Fig. TPUs 1700 can include application-specific integrated circuits (ASICs), for example, to perform some or all of the operations described above or elsewhere herein, such as, but not limited to, accelerating machine learning workloads that perform matrix operations. TPUs 1700 can be ASICs from Alphabet Corporation in Mountain View, CA. Cloud TPU includes a cloud service that makes TPUs available as a scalable resource for processing tasks, such as, but not limited to, machine learning workloads that can run on frameworks, such as, but not limited to, TensorFlow, PyTorch, and JAX.

[0146] Chip 1700 can contain any number of TPUs, which can contain Tensor Cores 1706. A Tensor Core 1706 can contain one or more Core Sequencers 1708, Vector Processing Units (VPUs) 1710, Matrix Multiplication Units (MXUs) 1712(A)-1714(N), where N is any integer greater than 1, and a Transpose Permutation Unit 1716. The 1708 core sequencer can retrieve instructions (e.g., VLIW (Very Long Instruction Word)) from the 1706 core's instruction memory (Imem), perform scalar operations using a scalar data store (Smem) and scalar registers (Sregs) (not shown), and forward vector instructions to the 1710 Vector Processing Unit (VPU). The instructions can initiate, for example, eight operations: two scalar ALUs, two vector ALUs, vector load and store, and a pair of slots that queue data to and from the matrix multiplication and transposition units.The VPU 1710 can perform vector operations using a large on-chip vector memory (Vmem) and vector registers (Vregs). The VPU 1710 can stream data to and from the MXU by decoupling FIFOs. The VPU 1710 can collect and distribute data to the Vmem via data-level parallelism (2D matrix and vector functional units) and instruction-level parallelism (8 operations per instruction). For example, a large two-dimensional matrix multiplier unit (MXU) 1712(A)-1712(N) can use a systolic array to reduce area and energy consumption, plus large, software-controlled on-chip memories instead of caches. The Transpose Reduction Permute Unit 1716 can perform matrix transpositions, reductions, and permutations of the VPU 1710 lanes (e.g., 128 x 128). High Bandwidth Memory 1704 can be used for on-chip applications. One or more 1700 chips can be interconnected for computation. For example, one or more 1700 chips can be configured as a torus, e.g.a 2D torus, connected. Chip 1700 can also include any number (e.g. four) Inter-Core Interconnect (ICI) links 1718, which can allow direct connections between chips to form a supercomputer.

[0147] With respect to any of the processors in Chip 1700 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions containing API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory).A result from API(s) can then be stored in memory inside or outside the processor, including registers, DRAM, Flash, SRAM, cache or other memory equivalents.

[0148] In at least one embodiment, Chip 1700 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0149] Fig. Figure 18 illustrates a vector processor according to at least one embodiment. Vector Processor 1800 can support a RISC-V standard. Vector Processor 1800 can include one or more Cores 1810 (e.g., scalar units) with one or more Vector Processing Units (VPUs) 1842 (e.g., vector units) that can perform, for example, some or all of the operations described above or elsewhere herein. Core 1810 can include Andes Custom Extension (ACE) 1816, which can be used to communicate customer-specific instructions to Processor 1800. Core 1810 can include a 1-cycle multiplier and 1-cycle instruction / data local memory (ILM / DLM) for increased parallelism by allowing concurrent instruction fetches and data accesses.The Memory Management Unit (MMU) 1824 can manage system memory and cache and provide branch execution, instruction-pair issuance, L1 instruction / data caches, and local memory storage. The 1810 core can include Physical Memory Protection and the Programmable Physical Memory Attribute Unit (PMP / PPMA) 1822. The 1810 core can include a Digital Signal Processor (DSP) 1828 and a Floating Point Unit (FPU) 1826, as well as a Load-Storage Unit (LSU) 1832 for interface connection to the memory hierarchy (D$ 1834 and I$ 1830). The 1810 core can include a Branch Prediction Unit 1818 and a Multiplier Unit 1820.

[0150] The Vector Processing Unit (VPU) 1842 can include one or more Vector Functional Units (FUs) 1846(A)-1846(N) that can be chained together for parallel processing, independent memory paths for RISC-V vector (RVV) load / store via ACE-RVV 1848 and Andes streaming port (ASP) 1844 load / store, and a Vector Load / Store Unit (VLSU) 1850.

[0151] The Vector Processor 1800 can include bus interfaces such as, but not limited to, L2 cache memory port 1856 for cacheable access, an MMIO port 1854 for non-cacheable access, an input / output coherence port (IOCP) 1858 for a cacheless bus master, local memory access ports for ILM / DLM 1812 and access to high-bandwidth vector memory (HVM) 1836, and a shared peripheral port (SPP) 1852 for external peripherals. Other memory ports include LM slave port AXI 1802 and HVM sub-port AXI 1804.

[0152] With respect to any of the processors in Processor 1800 and any of its components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions containing API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory).A result from API(s) can then be stored in memory inside or outside the processor, including registers, DRAM, Flash, SRAM, cache or other memory equivalents.

[0153] In at least one embodiment, Vectorprocessor 1800 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0154] Fig. Figure 19A illustrates a diagram of an exemplary multi-core tiled processor microarchitecture. Multi-core tiled processor in Fig. 19 can include a speech processing processor. As in Fig. As illustrated in Figure 19A, each "tile" of the processor architecture is a processing element bound together using a network-on-chip (NoC) that can be used to perform, for example, some or all of the operations described above or elsewhere herein. For example, each tile may include an instruction dispatcher 1904 and an integer (INT) 1906 and floating-point (FP) unit 1908, as well as a load-store unit (LSU) 1912 to connect to the memory hierarchy (data cache (D$) 1910 and instruction cache (I$) 1914) and a network (NET) 1916 interface for interface communication with other tiles of the architecture. Some tiles in Processor 1900 may include memory controllers 1902 for managing and controlling memory, as further described herein. Processor 1900 may have a functional slice architecture.The 1900 processor can be located on an application-specific integrated circuit (ASIC), and . Fig. 19A can represent the ASIC layout. Processor 1900 can include a co-processor designed to execute instructions for a predictive model. The predictive model is any model configured to make a prediction from input data. The predictive model can use a classifier to make a classification prediction. The predictive model can be a machine learning model, such as, but not limited to, a tensor streaming model, and Processor 1900 is a tensor streaming processor.

[0155] The 1900 processor can use different microarchitectures, which are in Fig. 19B disaggregate the functional units shown in each tile. Instead, the functional tiles of the 1900 processor can be aggregated into a variety of functional process units (hereafter referred to as "slices") 1904, each corresponding to a specific function type (e.g., FP / INT, NET, MEM). For example, as in Fig. Figure 19B illustrates that each slice corresponds to a column of functional tiles extending in a north-south direction. Additionally, the processor includes communication lanes for carrying data between the tiles of different slices, each running horizontally in an east-west direction. Each communication lane can be connected to any of the slices of the 1900 processor.

[0156] The slices of the @ processor can each correspond to a different function and can include arithmetic logic slices (e.g., FP / INT), lane-switching slices (e.g., NET), and memory slices (e.g., MEM). The arithmetic logic units perform one or more arithmetic and / or logical operations on data received over the communication lanes to produce output data. Examples of arithmetic logic units include matrix multiplication units and vector multiplication units. The memory slices contain memory cells that store data. The memory slices can provide data to other slices over the communication lanes. The memory slices can also receive data from other slices over the communication lanes. The lane-switching slices can configurably route data from one communication lane to any other communication lane.For example, data from a first lane can be provided to a second lane via a lane-switching slice. In some embodiments, the lane-switching slice can be implemented as a crossbar switch. Each 1904 slice also includes its own instruction queue (not shown), which stores instructions, and an instruction control unit (ICU) for controlling the execution of those instructions. The instructions in a given instruction queue can only be executed by tiling within their associated functional slice and cannot be executed by the processor's other slice.

[0157] By arranging the tiles of the 1900 processor into different functional slices 1904, the on-chip instruction and control flow of the 1900 processor can be decoupled from the data flow. For example, an arrow in Fig. 19B the flow of instructions within the processor architecture according to some embodiments. Another arrow in Fig. Figure 19B illustrates a data flow within the processor architecture according to at least one embodiment. As illustrated, the instructions and control flow in a first direction across the tiles of the 1900 processor (e.g., north-south, along the length of the functional slices, as shown by the first arrow), while the data flows in a second direction across the tiles of the 1900 processor (e.g., east-west, across the functional slices, as shown by the second arrow), which is perpendicular to the first direction.

[0158] Different functional slices of the processor can correspond to MEM (memory), VXM (vector execution module), MXM (matrix execution module), NIM (numeric interpretation module), and SXM (switching and permutation module). Each slice can contain N tiles, all of which can be controlled by the same instruction control unit (ICU) (not shown). Each of the slices can operate completely independently and can only be coordinated using barrier-like synchronization primitives or by the compiler through the exploitation of steerable determinism. Each tile of the processor can correspond to an execution unit organized as an xM SIMD tile. For example, each tile of the processor's on-chip memory can be organized to atomically store an L-element vector. As such, a MEM slice can work together with N tiles to store or process a large vector (e.g., with a total of N x M elements).

[0159] Tiles within the same slice can execute instructions in a staggered manner, with instructions being issued tile by tile within the slice over a period of N cycles. Functional slices can be physically arranged on-chip to enable efficient data flow for pipelined execution over hundreds of cycles for common patterns. Data flows can perform a single 180° turn (change of direction), corresponding to a single matrix operation, before being written back to memory. In some embodiments, a single data flow can change direction multiple times (due to multiple matrix and vector operations) before the resulting data is written back to memory.

[0160] To achieve good single-threaded performance, a conventional multi-core processor design (e.g., as in Fig. Figure 19A illustrates this. Typically, a significant portion of silicon area is dedicated to exposing and exploiting instruction-level parallelism (ILP). This usually involves register renaming schemes and large instruction windows, through which the instructions have no explicit understanding of the hardware on which they are executed, while maintaining the illusion of in-order program execution. In contrast, when a processor (e.g., TSP) with a functional slice architecture is used, the TSP compiler generates an explicit plan for how the processor will execute the microprogram. The compiler specifies when each operation will be performed, which functional slices will do the work, and which STREAM registers will hold the operands. The compiler maintains a highly reliable (cycle-accurate) model of the TSP's hardware state, enabling the microprogram to orchestrate the data flow.

[0161] The Processor 1900 (e.g., TSP) can use a web-hosted compiler that takes a model as its input (e.g., a machine learning model such as, but not limited to, a TensorFlow model) and outputs a proprietary instruction stream targeting the Processor-TSP hardware. The compiler is responsible for coordinating the program's control and data flow, specifying any instruction-level parallelism by explicitly bundling instructions that can and should be executed concurrently so they can be sent together. The primary hardware structure is the architecture-visible streaming register file (STREAMs), described in more detail below, which serves as the conduit through which operands flow from memory slices (e.g., SRAM) to functional slices and vice versa.

[0162] The processor's MEM unit serves as: (1) memory for model parameters, microprograms, and the data on which they operate, and (2) a network-on-chip (NoC) for communicating data operands from MEM to the functional slices and computed results back to MEM. In some embodiments, the on-chip memory consumes approximately 75% of the processor's chip area. In some embodiments, the on-chip memory of the MEM tiles may be SRAM rather than DRAM due to the processor's bandwidth requirements. The processor's on-chip memory capacity determines (i) the number of machine learning models that can reside on the chip simultaneously, (ii) the size of any given model, and (iii) the partitioning of large models to fit into multi-chip systems.In some embodiments, the processor's MEM system provides a multitude of memory slices organized into two distinct hemispheres (referred to as "MEM WEST" and "MEM EAST").

[0163] The memory slices of each hemisphere can be mirrored so that the slices can be physically numbered as {0,...L} in the east hemisphere (410) and {L,...0} in the west hemisphere (405), such that memory slice 0 for each hemisphere corresponds to the slice closest to the VXM slices (415) between the hemispheres, with each hemisphere having L slices. The direction of data transmission toward the center of the chip can be described as inward, while data transmission toward the outer (easternmost or westernmost) edge of the chip can be described as outward. Although the processor's memory hemispheres can be referred to as east and west, it is understood that in other embodiments, other names may be used to refer to the different memory hemispheres.

[0164] In some embodiments, a streaming register file, called STREAMS, transfers operands and results between the SRAM of the MEM slices and the functional slices of the processor. In some embodiments, a large number of MEM slices (e.g., between 2 and 10 adjacent MEM slices) can be physically organized as a set. Each set of slices can be located between a pair of STREAM register files, so that each slice is able to read from or write to the STREAM registers in both directions. Placing STREAM register files between sets of MEM slices reduces the number of cycles required for data operands to be transferred across one hemisphere (e.g., by a factor equal to the number of slices per set). The number of slices per set can be configured based on a distance over which data can be transferred in a single clock cycle.

[0165] With regard to any of the processors in Fig. 19 and any components described above or elsewhere herein, one or more of the APIs or equivalents described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic or equivalent, decoded by a processor decoder or equivalent, scheduled for execution by a scheduler or equivalent (e.g., in sequence or out of sequence), executed by execution logic or equivalent, reordered, and then shut down by shutdown logic or equivalent. API(s) (and / or compiled instructions containing API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory). A result of API(s) may then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory equivalents.

[0166] In at least one embodiment, processor 1900 can include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1 - 5 discussed, or otherwise perform any of the operations described above or elsewhere herein. Software designs

[0167] The following figures present, without limitation, examples of software constructs for implementing at least one embodiment.

[0168] Fig. Figure 20 illustrates a software stack of a programming platform according to at least one embodiment. A programming platform may include a platform for utilizing hardware on a computing system to accelerate computational tasks. A programming platform may be accessible to software developers through libraries, compiler directives, and / or extensions of programming languages ​​in at least one embodiment. A programming platform may be CUDA, Radeon Open Compute Platform (“ROCm”), OpenCL (OpenCL™ is developed by the Khronos group), SYCL, or Intel OneAPI.

[0169] A Software 2000 stack of a programming platform can provide an execution environment for an Application 2001. Application 2001 can include any computer software capable of running on Software 2000. Application 2001 can include an artificial intelligence (“AI”) / machine learning (“ML”) application, a high-performance computing (“HPC”) application, a virtual desktop infrastructure (“VDI”), or a data center workload.

[0170] Application 2001 and Software Stack 2000 run on Hardware 2008. Hardware 2008 can include one or more GPUs, CPUs, FPGAs, AI engines, and / or other types of computing devices that support a programming platform. Software Stack 2000 can be vendor-specific and compatible only with devices from a single vendor or vendors, such as CUDA, ROCm, OneAPI, OpenCL, or other implementations. Hardware 2008 can include a host connected to one or more devices that can be accessed to perform computing tasks via application programming interface (API) calls. A device within Hardware 2008 can, in at least one embodiment, include a GPU, FPGA, AI engine, or other computing device (but may also include a CPU) and its memory, as opposed to a host within Hardware 2008, which can include a CPU (but may also include a computing device) and its memory.With respect to any of the Hardware 2008 described above or elsewhere herein, one or more of the APIs described herein may, for example, be compiled into instructions that may be fetched by instruction-fetch logic, decoded by a processor decoder, scheduled for execution by a scheduler (e.g., sequentially or out of sequence), executed by execution logic, reordered, and then shut down by shutdown logic. API(s) (and / or compiled instructions containing API(s)) may be stored in any memory outside or inside the processor (e.g., in cache and / or memory). A result of API(s) may then be stored in memory inside or outside the processor, including registers, DRAM, flash, SRAM, cache, or other memory. One or more of the APIs described herein may contain a call.One or more of the APIs described herein may include a library or a portion of a library to perform a function described by the call. One or more of the APIs described herein may include a call and a library or a portion of a library to perform a function described by the call.

[0171] A Software 2000 stack of a programming platform can include a number of Libraries 2003, a Runtime 2005, an optional Driver / Interface 2007, and a Device Kernel Driver 2008. Each of the Libraries 2003 can contain data and program code that can be used by computer programs and exploited during software development. Libraries 2003 can include previously written code and subroutines, classes, values, type specifications, configuration data, documentation, auxiliary data, and / or message templates. Libraries 2003 can include functions that can be optimized to run on one or more types of devices. Libraries 2003 can include functions for performing mathematical, deep learning, and / or other types of operations on devices.Library 2003 can be associated with corresponding APIs 2002, which may include one or more APIs that expose functionality implemented in Library 2003. A processor (e.g., CPU, GPU) can execute, call, or otherwise use one or more APIs to prioritize kernels. For example, a first kernel (e.g., parent) can start a second kernel (e.g., child kernel), and the second kernel can be used by a processor to start additional kernels (e.g., grandchildren kernels) independently of the first kernel. A processor can execute an API or call an API from the memory to be executed to support dynamic stream priority (e.g., updating priority while a stream is being used to perform operations). For example, when a processor executes the API, it allows a programmer to copy stream priority from one stream to one or more other streams.

[0172] Software Stack 2000 can include an API to support dynamic stream priority (e.g., updating priority while a stream is used to perform operations), allowing a programmer to set a stream's priority at any time after its creation. Software Stack 2000 can also include an API to support dynamic stream priority (e.g., updating priority while the stream is used to perform operations), allowing a programmer to retrieve a stream's current priority, where the priority is one of a variety of a stream's attributes. Finally, Software Stack 2000 can include an API to support dynamic stream priority (e.g., updating priority while the stream is used to perform operations), allowing a programmer to retrieve a stream's current priority as a single attribute.Software Stack 2000 can include an API to support dynamic stream priority (e.g., updating priority while the stream is being used to perform operations), allowing a programmer to start a kernel to perform operations on a stream with a set priority that may differ from the stream priority. Software Stack 2000 can also include an API to indicate whether an object (e.g., a thread synchronization object such as, but not limited to, a barrier) is tracking, whether all data movement operations for a set of threads operating on a GPU can be completed, encompassing a specified state after a specified period of time, where a specified state can be a state indicating that data has been moved and is ready for use, and is specified using an expected parity value as input to the API.

[0173] Software Stack 2000 can include one or more APIs for updated kernels. A processor can execute an API or call an API from the memory to be executed to update to an existing API and support context-free kernels, which allows a programmer to add a kernel node to a graph without a graphics context, so that a graphics context can be dynamically associated with a kernel at runtime. Software Stack 2000 can include one or more APIs to allow a programmer to receive a kernel identifier and a graphics context as separate parameters from a kernel node, so that parameters can be obtained from kernels and from context-free kernels. Software Stack 2000 can include one or more APIs to use parallel processor(s), such as, but not limited to, one or more graphics processing units, to process task graphs (e.g.,to start task graphs) and execute one or more task graphs (e.g., including one or more programs).

[0174] Software Stack 2000 can include one or more APIs to associate one or more instructions with one or more memory ordering operations, such as, but not limited to, a fence or membar operation. Instructions can be associated with one or more domains, so that a memory ordering operation is performed in association with one or more individual domains without interfering with instructions in other domains. An API can indicate that a thread has arrived (for example, at a thread synchronization barrier) or has completed a stage of work with respect to asynchronous data movement operations on a GPU.Software Stack 2000 can include one or more to allow programmers to manually display an expected transaction count when a thread has finished a work stage, which can be used to update an object that tracks whether all data movement operations can be completed for a set of threads.

[0175] Application 2001 can be written as source code, which is compiled into executable code, as described below in conjunction with the Fig. 21 and Fig. This is discussed in more detail in section 22. Executable code from Application 2001 can run, at least partially, on an execution environment provided by Software Stack 2000. During Application 2001 execution, code may need to run on a device, as opposed to a host. In such a case, Runtime 2005 can be invoked to load and start the necessary code on the device. Runtime 2005 can include any technically feasible runtime system capable of supporting Application 2001 execution.

[0176] Runtime 2005 can be implemented as one or more runtime libraries associated with corresponding APIs shown as API(s) 2004. One or more such runtime libraries may include, among other things, functions for memory management, execution control, device management, error handling, and / or synchronization. Memory management functions may include functions for allocating, unallocating, and copying device memory, as well as for transferring data between host memory and device memory. Execution control functions may include functions for starting a function (sometimes called a "kernel" if a function is a global function callable from a host) on a device and setting attribute values ​​in a buffer maintained by a runtime library for a given function to be executed on a device.

[0177] Runtime libraries and their corresponding APIs (as defined in 2004) can be implemented in any technically feasible way. One (or any number of) APIs can expose a low-level set of functions for fine-grained control of a device, while another (or any number of) APIs can expose a higher-level set of such functions. A high-level runtime API can be built upon a low-level API. One or more runtime APIs can be language-specific APIs that can be layered on top of a language-independent runtime API.

[0178] An optional 2007 driver or interface can be implemented, for example, for CUDA and ROCm implementations, which are described below. The optional 2007 driver / interface can be associated with optional driver or interface API(s), such as, but not limited to, CUDA and / or ROCm API(s).

[0179] One or more processors, disclosed in "Processing Systems," can execute, access, or otherwise use Software Stack 2000. For example, System-on-a-Chip 700, Parallel Processor 800, Graphics Multiprocessor 834, Processor 900, Processor 1000, Accelerator 1100, Neuromorphic Processor 1205, Supercomputer 1300, Acceleration Processing Unit 1400, Processor 1500, Processor 1600, Tensor Processing Unit 1700, Processor 1800, and Speech Processing Unit 1900 can execute, use, call, or otherwise implement (for example, by accessing memory) one or more APIs included in Software Stack 2000.

[0180] Device Kernel Driver 2008 can be configured to facilitate communication with an underlying device. Device Kernel Driver 2008 can provide low-level functionality upon which APIs, such as but not limited to API 2004 and / or other software, depend. Device Kernel Driver 2008 can be configured to compile intermediate representation ("IR") code into binary at runtime. For CUDA or other implementations, such as but not limited to ROCm, OneAPI, or OpenCL, Device Kernel Driver 2008 can compile non-hardware-specific parallel-threaded execution ("PTX") IR code into binary for a specific target device at runtime (with caching of compiled binary), sometimes referred to as "finalizing" code. This can allow finalized code to run on a target device that may not have existed when the source code was originally compiled into PTX code.Alternatively, device source code can be compiled offline into binary code, without the need for the device kernel driver to compile IR code at runtime.

[0181] Processors described elsewhere herein, such as, but not limited to, processors in Fig. 7-19, may include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software, e.g., Software Stack 2000, to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0182] According to at least one embodiment, software stack 2000 of Fig. 20 in a CUDA implementation. A CUDA software stack 2000, on which a CUDA application 2001 can be started, can include CUDA libraries 2003, a CUDA runtime 2005, a CUDA driver 2007, and a device kernel driver 2008. A CUDA software stack 2000 can run on hardware 2409, which can include a GPU that supports CUDA and is developed by NVIDIA Corporation of Santa Clara, CA.

[0183] Application 2001, CUDA Runtime 2005, and Device Kernel Driver 2008 can perform functionalities described above and elsewhere herein. CUDA Driver 2007 can include a library (libcuda.so) that can implement a CUDA Driver API 2006. Similar to a CUDA Runtime API 2004 implemented through a CUDA Runtime Library (cudart), CUDA Driver API 2006 can expose, among other things, functions for memory management, execution control, device management, error handling, synchronization, and / or graphics interoperability. CUDA Driver API 2006 may differ from CUDA Runtime API 2004 in that CUDA Runtime API 2004 simplifies device code management by providing implicit initialization, context (analogous to a process) management, and module (analogous to dynamically loaded libraries) management.Unlike the high-level CUDA Runtime API 2004, the CUDA Driver API 2006 can be a low-level API that provides more granular control of the device, particularly regarding contexts and module loading. The CUDA Driver API 2006 can expose context management features that cannot be exposed by the CUDA Runtime API 2004. The CUDA Driver API 2006 can also be language-independent and support languages ​​such as OpenCL, in addition to those supported by the CUDA Runtime API 2004. Furthermore, development libraries, including the CUDA Runtime 2005, can be considered separate from driver components, including the user-mode CUDA Driver 2007 and the kernel-mode device driver 2008 (sometimes referred to as a "display" driver).

[0184] CUDA 2003 libraries can include mathematical libraries, deep learning libraries, parallel algorithm libraries, and / or signal / image / video processing libraries that can utilize parallel computing applications, such as, but not limited to, Application 2001. CUDA 2003 libraries can include mathematical libraries such as, but not limited to, a cuBLAS library, which is an implementation of Basic Linear Algebra Subprograms ("BLAS") for performing linear algebra operations; a cuFFT library for calculating fast Fourier transforms ("FFTs"); and a cuRAND library for generating random numbers. CUDA 2003 libraries can include deep learning libraries such as, but not limited to, a cuDNN library of primitives for deep neural networks and a TensorRT platform for high-performance deep learning inference.

[0185] In at least one embodiment, processors described elsewhere herein, such as, but not limited to, processors in Fig. 7-19, include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software, e.g., Software Stack 2000, to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0186] According to at least one embodiment, software stack 2000 of Fig. 20 in an ROCm implementation. An ROCm software stack 2000, on which an application 2001 can be launched, includes a language runtime 2003, a system runtime 2005, a Thunk 2007, and an ROCm kernel driver 2008. The ROCm software stack 2000 runs on hardware 2009, which may include a GPU that supports ROCm and is developed by AMD Corporation of Santa Clara, CA.

[0187] Application 2001 can perform similar functionalities as described above in conjunction with Fig. 20 is discussed. Additionally, Language Runtime 2003 and System Runtime 2005 can perform similar functionalities to Runtime 2005, which are discussed above in conjunction with Fig. 20 is discussed. Language Runtime 2003 and System Runtime 2005 can differ in that System Runtime 2005 is a language-independent runtime that implements a ROCr System Runtime API 2004 and makes use of a Heterogeneous System Architecture (“HSA”) runtime API. The HSA runtime API can include, among other things, a thin, user-mode API that exposes interfaces for accessing and interacting with an AMD GPU, including functions for memory management, execution control via architected kernel dispatch, error handling, system and agent information, and runtime initialization and shutdown. In contrast to System Runtime 2005, Language Runtime 2003 can be an implementation of a language-specific Runtime API 2002 layered on top of an ROCr System Runtime API 2004.A language runtime API can include, among other things, a Heterogeneous Compute Interface for Portability (“HIP”) language runtime API, a Heterogeneous Compute Compiler (“HCC”) language runtime API, or an OpenCL API. HIP, in particular, is an extension of the C++ programming language with functionally similar versions of CUDA mechanisms, and a HIP language runtime API can include functions similar to those of the CUDA runtime API described above. Fig. 20 is discussed, such as, but not limited to, functions for memory management, execution control, device management, error handling and synchronization.

[0188] Thunk (ROCt) 2007 can be an interface 2006 that can be used to interact with the underlying ROCm driver 2008. The ROCm driver 2008 can be a ROCK driver that is a combination of an AMDGPU driver and an HSA kernel driver (amdkfd). The AMDGPU driver can be a device kernel driver for GPUs, developed by AMD, that performs similar functionalities to the device kernel driver 2009, which is mentioned above in conjunction with... Fig. 20 is discussed. An HSA kernel driver can be a driver that allows different types of processors to share system resources more effectively via hardware features.

[0189] Various libraries (not shown) can be included in ROCm software stack 2000 via language runtime 2003 and provide functionality similar to CUDA libraries 2003, which are mentioned above in conjunction with Fig. 20 is discussed. Various libraries may include mathematical, deep learning and / or other libraries, such as, but not limited to, a hipBLAS library that implements functions similar to those of CUDA cuBLAS, a rocFFT library for calculating FFTs, which is similar to CUDA cuFFT.

[0190] Processors described elsewhere herein, such as, but not limited to, processors in Fig. 7-19, may include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software, e.g., Software Stack 2000, to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0191] According to at least one embodiment, software stack 2000 of Fig. 20 in an OpenCL implementation. An OpenCL 2000 software stack on which a 2001 application can be launched may include an OpenCL 2003 framework, an OpenCL 2005 runtime, and a 2008 driver. An OpenCL 2000 software stack can run on 2009 hardware that is not vendor-specific. Because OpenCL is supported by devices developed by various vendors, specific OpenCL drivers may be required to work compatiblely with hardware from such vendors.

[0192] Application 2001, OpenCL Runtime 2005, Device Kernel Driver 2008, and Hardware 2009 can perform similar functionalities to other implementations of Application 2001, Runtime 2005, Device Kernel Driver 2008, and Hardware 2009, respectively, as described above in conjunction with Fig. 20 are discussed. Application 2001 can also include an OpenCL kernel (not shown) with code to be executed on a device.

[0193] OpenCL can define a "platform" that allows a host to control devices connected to it. An OpenCL framework can provide a platform-layer API and a runtime API, shown as Platform API 2002 and Runtime API 2004. Runtime API 2004 can use contexts to manage kernel execution on devices. Each identified device can be associated with a specific context, which Runtime API 2004 can use to manage command queues, program objects, and kernel objects, as well as to share memory objects for that device, among other things. Platform API 2002 can expose functions that allow the use of device contexts to select and initialize devices, submit work to devices via command queues, and enable data transfer to and from devices.Additionally, the OpenCL framework can provide various built-in functions (not shown), including mathematical functions, relational functions, and image processing functions, among others.

[0194] A compiler (not shown) may also be included in the OpenCL Framework 2003. Source code can be compiled offline before an application is run, or online during application execution. Unlike CUDA and ROCm, OpenCL applications can be compiled online by a compiler that is representative of any number of compilers that can be used to compile source code and / or IR code, such as, but not limited to, Standard Portable Intermediate Representation ("SPIR-V") code, into binary code. Alternatively, OpenCL applications can be compiled offline before such applications are executed.

[0195] In at least one embodiment, processors described elsewhere herein, such as, but not limited to, processors in Fig. 7-19, include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software, e.g., Software Stack 2000, to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0196] According to at least one embodiment, software can be supported by a programming platform configured to support various programming models, middleware, and / or libraries and frameworks upon which an application may rely. The application may be an AI / ML application implemented, for example, using a deep learning framework such as, but not limited to, MXNet, PyTorch, or TensorFlow, which may rely on libraries such as, but not limited to, cuDNN, NVIDIA Collective Communications Library (“NCCL”), and / or NVIDIA Developer Data Loading Library (“DALI”) CUDA libraries to provide accelerated computing on underlying hardware.

[0197] The programming platform can be one of a CUDA, ROCm, or OpenCL platform, as described above in conjunction with Fig. As described in section 20, a programming platform can support multiple programming models, which can be abstractions of an underlying computer system and allow expressions of algorithms and data structures. Programming models can expose features of underlying hardware to improve performance. Programming models can include CUDA, HIP, OpenCL, C++ Accelerated Massive Parallelism (“C++AMP”), Open Multi-Processing (“OpenMP”), Open Accelerators (“OpenACC”), and / or Vulcan Compute.

[0198] Libraries and / or middleware can provide implementations of programming model abstractions. Such libraries can include data and code that can be used by computer programs and exploited during software development. Middleware can also include software that provides services to applications beyond those available from the programming platform. Examples of libraries and / or middleware include cuBLAS, cuFFT, cuRAND, and other CUDA libraries, or rocBLAS, rocFFT, rocRAND, and other ROCm libraries.Additionally, libraries and / or middleware may include NCCL and ROCm Communication Collectives Library (“RCCL”) libraries, which provide communication routines for GPUs, an MIOpen library for deep learning acceleration, and / or a proprietary library for linear algebra, matrix and vector operations, geometric transformations, numerical solvers, and related algorithms.

[0199] Application frameworks can depend on libraries and / or middleware. Any application framework can be a software framework used to implement a standard application software structure. Returning to the AI / ML example discussed above, an AI / ML application can be implemented using a framework such as, but not limited to, Caffe, Caffe2, TensorFlow, Keras, PyTorch, or MxNet deep learning frameworks.

[0200] In at least one embodiment, processors described elsewhere herein, such as, but not limited to, processors in Fig. 7-19, include one or more circuits to use one or more neural networks to generate image object features, based at least partially on a variable sampling rate of one or more audio signals, as above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein. One or more circuits may be configured by software, e.g., programming platforms described herein, to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1-5 discussed, or otherwise perform any of the operations described above or elsewhere herein.

[0201] Fig. Figure 21 illustrates compiling code for execution on one of the programming platforms described above. Fig. 20 according to at least one embodiment. A compiler 2101 is configured to receive source code 2100, compile source code 2100, and output an executable file 2110. Compiler 2101 can be configured to convert source code 2100 into host executable code 2107 for execution on a host and device executable code 2108 for execution on a device. Source code 2100 can be compiled either offline before an application is executed or online during the execution of an application. Source code 2100 can contain code in any programming language supported by compiler 2101, such as, but not limited to, C++, C, Fortran, etc. Source code 2100 can be contained in a single source file, with a mixture of host code and device code, the positions of which are indicated therein. A single source file can be a .cu file containing CUDA code, or a .hip file.A .cpp file containing HIP code, or a file in another format containing both host code and device code. Alternatively, source code can contain multiple source code files instead of a single source file, in which host code and device code can be separated. Compiler 2101 includes or has access to one or more libraries to detect a sequence of API calls to execute a single fused API, where a single fused API is a combined API for two or more APIs. In at least one embodiment, Compiler 2101 can be an NVIDIA CUDA compiler (“NVCC”) for compiling CUDA code into .cu files, an HCC compiler for compiling HIP code into .hip.cpp files, or other compilers.

[0202] Compiler 2101 can be configured to compile source code 2100 into host executable code 2107 for execution on a host and device executable code 2108 for execution on a device. Compiler 2101 performs operations including parsing source code 2100 into an abstract system tree (AST), performing optimizations, and generating executable code. If source code 2100 contains a single source file, Compiler 2101 can separate device code from host code in such a single source file, compile device code and host code into device executable code 2108 and host executable code 2107 respectively, and then join the device executable code 2108 and host executable code 2107 together in a single file.

[0203] Compiler 2101 can include a compiler frontend 2102, a host compiler 2105, a device compiler 2106, and a linker 2109. Compiler frontend 2102 can be configured to separate device code 2104 from host code 2103 into source code 2100. Device code 2104 can be compiled by device compiler 2106 into executable device code 2108, which, as described, can contain binary code or IR code, in at least one embodiment. Separately, host code 2103 can be compiled by host compiler 2105 into executable host code 2107. For NVCC other compilers, such as, but not limited to, those for oneAPI, ROCm and OpenCL, Host Compiler 2105 can be a general-purpose C / C++ compiler that outputs native object code, while Device Compiler 2106 can be a Low-Level Virtual Machine (“LLVM”)-based compiler that forks an LLVM compiler infrastructure and outputs PTX code or binary code.For HCC, both host compiler 2105 and device compiler 2106 can be LLVM-based compilers that output binary target code.

[0204] After compiling source code 2100 into executable host code 2107 and executable device code 2108, linker 2109 can link executable host code 2107 and device code 2108 together in executable file 2110. Native object code for a host and PTX or binary code for a device can be linked together in an Executable and Linkable Format (“ELF”) file, which is a container format used to store object code. Executable host code 2107 and executable device code 2108 can be in any suitable format, such as, but not limited to, binary code and / or IR code. In a case of CUDA, executable host code 2107 can contain native object code and executable device code 2108 can contain code in PTX intermediate representation, in at least one embodiment.In one case of ROCm, both executable host code 2107 and executable device code 2108 can contain binary target code, in at least one embodiment. Other implementations, such as, but not limited to, oneAPI and OpenCL, are considered and can be implemented similarly to the CUDA and ROCm implementations above.

[0205] Source code 2100 can be translated before compiling source code. Source code is passed through a translation tool (not shown) that translates source code 2100 into translated source code. A compiler 2101 can be used to compile translated source code into executable host code 2107 and executable device code 2108 in a process similar to the compilation of source code 2100 by compiler 2101 into executable host code 2107 and executable device code 2108, as described above in conjunction with Fig. 21 is discussed.

[0206] A translation performed by a translation tool can be used to port Source Code 2100 for execution in a different environment than the one in which it was originally intended to run. The translation tool may include a HIP translator, which is used to "translate" (hipify) CUDA code intended for a CUDA platform into HIP code that can be compiled and executed on a ROCm platform. Source Code 2100 translation may involve parsing Source Code 2100 and converting API calls provided by one programming model (e.g., CUDA) into corresponding API calls provided by another programming model (e.g., HIP), as shown below in conjunction with... Fig. 22 is discussed in more detail. For example, by translating CUDA code into HIP code (hipifying), calls to the CUDA runtime API, CUDA driver API, and / or CUDA libraries can be converted into corresponding HIP API calls. Automated translations performed by translation tool 2101 can sometimes be incomplete, requiring additional manual effort to fully port source code 2100.

[0207] One or more of the techniques described herein may use other methods for converting one code type to another to enable interchangeability between different device architectures. In at least one embodiment, an application for one platform (e.g., a CUDA application) may be compiled into code for implementation on another platform (e.g., an AMD processor, an Intel processor, or another processor). For example, source code 2100 may contain source code for one platform (e.g., CUDA). Compiler 2101 may compile the source code 2100 into an executable file 2110 that can be used by another platform (e.g., AMD or Intel). Programming toolkits may allow applications for one platform (e.g., CUDA) to be compiled for another platform (e.g., AMD or Intel) (e.g., natively).For example, a GPGPU programming toolset can allow CUDA applications to be compiled natively for AMD GPUs. Programs (e.g., CUDA programs) or their construction system do not need to be modified or translated into another language before compilation to compile code for a different platform. A compiler can adopt the same command-line options and programming dialect (e.g., CUDA dialect) as another compiler (e.g., nvcc for CUDA), acting as a drop-in replacement to mimic an installation of a toolset (e.g., NVIDIA CUDA Toolkit), so that existing construction tools and scripts (e.g., such as cmake) function without further modification. In at least one embodiment, an nvcc-compatible compiler can be used to compile nvcc-dialect CUDA for AMD GPUs, including PTX assembly. Implementations of CUDA runtime and driver APIs for AMD GPUs can be used. Libraries (e.g.,Open-source wrapper libraries can provide APIs, such as CUDA-X APIs, by delegating to the corresponding ROCm libraries. One exemplary implementation is SCALE from Spectral Compute in London, England. Rather than providing a new way to write GPGPU software, SCALE allows programs written using the widely popular CUDA language to be compiled directly for AMD GPUs. Additional implementations may include a Clang compiler, which provides a language front-end and tooling infrastructure for languages ​​in the C language family (C, C++, Objective C / C++, OpenCL, CUDA, and RenderScript). In at least one embodiment, compilers described herein, such as, but not limited to, Compiler 2101, Compiler 2105, and / or Compiler 2106, may include one or more circuits to compile code (e.g.,to compile CUDA, HIP, OpenCL, OneAPI or others) to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as above in relation to . Fig. 1-5 discussed, and / or to perform one of the operations described above or elsewhere herein.

[0208] Fig. Figure 22 illustrates a System 2200 configured to compile and execute CUDA source code 2210 using different types of processing units according to at least one embodiment. System 2200 includes CUDA source code 2210, a CUDA compiler 2250, host executable code 2270(1), host executable code 2270(2), CUDA device executable code 2284, a CPU 2290, a CUDA-enabled GPU 2294, a GPU 2292, a CUDA-to-HIP translation tool 2220, HIP source code 2230, a HIP compiler driver 2240, an HCC 2260, and HCC device executable code 2282.

[0209] CUDA source code 2210 can be a collection of human-readable code written in a CUDA programming language. A CUDA programming language can be an extension of the C++ programming language that includes mechanisms to define device code and distinguish between device code and host code. Device code can contain source code that, after compilation, is executable in parallel on a device. A device can be a processor optimized for parallel instruction processing, such as, but not limited to, the CUDA-enabled GPU 2290, GPU 2292, or another GPGPU, etc. Host code is source code that, after compilation, is executable on a host. A host is a processor optimized for sequential instruction processing, such as, but not limited to, the CPU 2290.

[0210] CUDA source code 2210 can contain any number (including zero) of global functions 2212, any number (including zero) of device functions 2214, any number (including zero) of host functions 2216, and any number (including zero) of host / device functions 2218. Global functions 2212, device functions 2214, host functions 2216, and host / device functions 2218 can be mixed in CUDA source code 2210. Each of the global functions 2212 can be executable on a device and callable by a host. One or more of the global functions 2212 can therefore act as entry points to a device. Each of the global functions 2212 can be a kernel. In a technique known as dynamic parallelism, one or more of global functions 2212 can define a kernel that is executable on and invoked by a device.A kernel can be executed N times in parallel (where N is any positive integer) by N different threads on one device during execution.

[0211] Each of device functions 2214 can be executed on a device and can be called exclusively by such a device. Each of host functions 2216 can be executed on a host and can be called exclusively by such a host. Each of host / device functions 2216 can define both a host version of a function that can be executed on a host and can be called exclusively by such a host, and a device version of the function that can be executed on a device and can be called exclusively by such a device.

[0212] CUDA source code 2210 can also include any number of calls to any number of functions that can be defined via a CUDA runtime API 2202. A CUDA runtime API 2202 can include any number of functions that run on a host to allocate and unallocate device memory, transfer data between host memory and device memory, manage systems with multiple devices, and so on. CUDA source code 2210 can also include any number of calls to any number of functions that can be specified in any number of other CUDA APIs. A CUDA API can be any API designed for use by CUDA code. CUDA APIs can include the CUDA runtime API 2202, a CUDA driver API, APIs for any number of CUDA libraries, and so on, including any API(s) described elsewhere herein.In relation to the CUDA runtime API 2202, a CUDA driver API can be a lower-level API, but it can provide more granular control of a device. Examples of CUDA libraries include cuBLAS, cuFFT, cuRAND, cuDNN, etc.

[0213] CUDA Compiler 2250 can compile input CUDA code (e.g., CUDA source code 2210) to produce executable host code 2270(1) and executable CUDA device code 2284. CUDA Compiler 2250 can be NVCC, but is not limited to it. Executable host code 2270(1) can be a compiled version of host code contained in input source code that is executable on CPU 2290. CPU 2290 can be any processor optimized for sequential instruction processing.

[0214] CUDA Device Executable Code 2284 can be a compiled version of device code contained in input source code that is executable on CUDA-enabled GPU 2294. CUDA Device Executable Code 2284 can include binary code. CUDA Device Executable Code 2284 can include IR code, such as, but not limited to, PTX code, which is further compiled at runtime into binary code for a specific target device (e.g., CUDA-enabled GPU 2294) by a device driver. CUDA-enabled GPU 2294 can include any processor optimized for parallel instruction processing that supports CUDA. CUDA-enabled GPU 2294 can be developed by NVIDIA Corporation of Santa Clara, CA.

[0215] The CUDA-to-HIP translation tool 2220 can be configured to translate CUDA source code 2210 into functionally similar HIP source code 2230. HIP source code 2230 can include a collection of human-readable code in a HIP programming language. HIP code can include human-readable code in a HIP programming language. A HIP programming language can include an extension of the C++ programming language that includes functionally similar versions of CUDA mechanisms to define device code and distinguish between device code and host code. A HIP programming language can include a subset of the functionality of a CUDA programming language. For example, a HIP programming language includes a mechanism, respectively.Mechanisms to define global functions 2212 exist, but such a HIP programming language may lack support for dynamic concurrency, and therefore global functions 2212 defined in HIP code may be callable exclusively by one host.

[0216] HIP source code 2230 can include any number (including zero) of global functions 2212, any number (including zero) of device functions 2214, any number (including zero) of host functions 2216, and any number (including zero) of host / device functions 2218. HIP source code 2230 can also include any number of calls to any number of functions that can be specified in a HIP runtime API 2232. HIP runtime API 2232 can include functionally similar versions of a subset of functions contained in CUDA runtime API 2202. HIP source code 2230 can also include any number of calls to any number of functions that can be specified in any number of other HIP APIs. A HIP API can be any API designed for use by HIP code and / or ROCm.HIP APIs can include the HIP runtime API 2232, a HIP driver API, APIs for any number of HIP libraries, APIs for any number of ROCm libraries, etc.

[0217] The CUDA-to-HIP translation tool 2220 can convert any kernel call in CUDA code from CUDA syntax to HIP syntax and can convert any number of other CUDA calls in CUDA code into any number of other functionally similar HIP calls. A CUDA call can involve a call to a function specified in a CUDA API, and a HIP call can involve a call to a function specified in a HIP API. The CUDA-to-HIP translation tool 2220 can convert any number of calls to functions specified in CUDA Runtime API 2202 into any number of calls to functions specified in HIP Runtime API 2232.

[0218] CUDA-to-HIP translation tool 2220 may include a tool known as hipify-perl, which performs a text-based translation process. CUDA-to-HIP translation tool 2220 may also include a tool known as hipify-clang, which, relative to hipify-perl, performs a more complex and robust translation process involving parsing CUDA code using clang (a compiler front end) and then translating the resulting symbols. Converting CUDA code to HIP code may include modifications (e.g., manual edits) in addition to those performed by CUDA-to-HIP translation tool 2220.

[0219] The HIP compiler driver 2240 can include a front end that determines a target device 2246 and then configures a compiler compatible with target device 2246 to compile HIP source code 2230. Target device 2246 can include a processor optimized for parallel instruction processing. The HIP compiler driver 2240 can determine target device 2246 in any technically feasible way.

[0220] If the target device 2246 is compatible with CUDA (e.g., CUDA-enabled GPU 2294), the HIP compiler driver 2240 can generate a HIP / NVCC compilation command 2242. The HIP / NVCC compilation command 2242 can configure the CUDA compiler 2250 to compile HIP source code 2230 using a HIP-to-CUDA translation header and a CUDA runtime library. In response to the HIP / NVCC compilation command 2242, the CUDA compiler 2250 can generate host executable code 2270(1) and CUDA device executable code 2284.

[0221] If target device 2246 is not compatible with CUDA, then HIP compiler driver 2240 can generate a HIP / HCC compilation command 2244. HIP / HCC compilation command 2244 can configure HCC 2260 to compile HIP source code 2230 using an HCC header and a HIP / HCC runtime library. In response to HIP / HCC compilation command 2244, HCC 2260 can generate host executable code 2270(2) and HCC device executable code 2282. HCC device executable code 2282 can be a compiled version of device code contained in HIP source code 2230, which is executable on GPU 2292. GPU 2292 can be any processor optimized for parallel instruction processing, incompatible with CUDA, and compatible with HCC. GPU 2292 may be developed by AMD Corporation of Santa Clara, CA. GPU 2292 may include a non-CUDA-enabled GPU 2292.

[0222] The following are for illustrative purposes only: Fig. Figure 22 shows three different flows that can be implemented in at least one embodiment to compile CUDA source code 2210 for execution on CPU 2290 and different devices. A direct CUDA flow can compile CUDA source code 2210 for execution on CPU 2290 and CUDA-enabled GPU 2294 without translating CUDA source code 2210 into HIP source code 2230. An indirect CUDA flow can translate CUDA source code 2210 into HIP source code 2230 and then compile the HIP source code 2230 for execution on CPU 2290 and CUDA-enabled GPU 2294. A CUDA / HCC flow can translate CUDA source code 2210 into HIP source code 2230 and then compile the HIP source code 2230 for execution on CPU 2290 and GPU 2292.

[0223] A direct CUDA flow that can be implemented is shown by dashed lines and a series of bubbles labeled A1-A3. As shown by bubble A1, CUDA compiler 2250 can receive CUDA source code 2210 and a CUDA compilation command 2248, which can configure CUDA compiler 2250 to compile CUDA source code 2210. CUDA source code 2210, which can be used in a direct CUDA flow, can be written in a CUDA programming language based on a programming language other than C++ (e.g., C, Fortran, Python, Java, etc.). In response to CUDA compilation command 2248, CUDA compiler 2250 can generate host executable code 2270(1) and CUDA device executable code 2284 (shown with bubble, notated A2). As shown with bubble, notated A3, host executable code 2270(1) and CUDA device executable code 2284 can each be executed on CPU 2290 and CUDA-enabled GPU 2294, respectively.CUDA device executable code 2284 can contain binary code. CUDA device executable code 2284 can contain PTX code and can furthermore be compiled into binary code for a specific target device at runtime.

[0224] An indirect CUDA flow that can be implemented is shown by dotted lines and a series of bubbles labeled B1-B6. As shown by bubble B1, CUDA-to-HIP translation tool 2220 can receive CUDA source code 2210. As shown by bubble B2, CUDA-to-HIP translation tool 2220 can translate CUDA source code 2210 into HIP source code 2230. As shown by bubble B3, HIP compiler driver 2240 can receive HIP source code 2230 and determine that target device 2246 is CUDA-enabled.

[0225] As shown in bubble, B4 notes, HIP compiler driver 2240 can generate HIP / NVCC compilation command 2242 and can submit both HIP / NVCC compilation command 2242 and HIP source code 2230 to CUDA compiler 2250. HIP / NVCC compilation command 2242 can configure CUDA compiler 2250 to compile HIP source code 2230 using a HIP-to-CUDA translation header and a CUDA runtime library. The HIP-to-CUDA translation header can translate any number of mechanisms (e.g., functions), specified in any number of HIP APIs, into any number of mechanisms specified in any number of CUDA APIs. CUDA compiler 2250 can use HIP-to-CUDA translation headers in conjunction with a CUDA runtime library, according to CUDA runtime API 2202, to generate host executable code 2270(1) and CUDA device executable code 2284.In response to HIP / NVCC compilation instruction 2242, CUDA compiler 2250 can generate host executable code 2270(1) and CUDA device executable code 2284 (shown by bubble, notated B5). As shown by bubble, notated B6, host executable code 2270(1) and CUDA device executable code 2284 can each be executed on CPU 2290 and CUDA-enabled GPU 2294, respectively. CUDA device executable code 2284 can contain binary code. CUDA device executable code 2284 can contain PTX code and can furthermore be compiled at runtime into binary code for a specific target device.

[0226] A CUDA / HCC flow that can be implemented is shown by solid lines and a series of bubbles labeled C1-C6. As shown by bubble labeled C1, CUDA-to-HIP translation tool 2220 can receive CUDA source code 2210. As shown by bubble labeled C2, CUDA-to-HIP translation tool 2220 can translate CUDA source code 2210 into HIP source code 2230. As shown by bubble labeled C3, HIP compiler driver 2240 can receive HIP source code 2230 and determine that target device 2246 is not CUDA-enabled.

[0227] HIP compiler driver 2240 can generate HIP / HCC compilation command 2244 and can submit both HIP / HCC compilation command 2244 and HIP source code 2230 to HCC 2260 (shown with bubble, notated C4). HIP / HCC compilation command 2244 can configure HCC 2260 to compile HIP source code 2230 using an HCC header and a HIP / HCC runtime library. The HIP / HCC runtime library can correspond to HIP runtime API 2232. The HCC header can contain any number and type of interoperability mechanisms for HIP and HCC. In response to HIP / HCC compilation command 2244, HCC 2260 can generate host executable code 2270(2) and HCC device executable code 2282 (shown with bubble, notated C5). As shown with bubble, notated C6, host executable code 2270(2) and HCC device executable code 2282 can each be executed on CPU 2290 and GPU 2292, respectively.

[0228] After CUDA source code 2210 has been translated into HIP source code 2230, HIP compiler driver 2240 can then be used to generate executable code for either CUDA-enabled GPU 2294 or GPU 2292 without re-executing the CUDA-to-HIP translation tool 2220. CUDA-to-HIP translation tool 2220 can translate CUDA source code 2210 into HIP source code 2230, which is then stored in memory. HIP compiler driver 2240 can then configure HCC 2260 to generate host executable code 2270(2) and HCC device executable code 2282 based on HIP source code 2230. In at least one embodiment, HIP compiler driver 2240 subsequently configures CUDA compiler 2250 to generate host executable code 2270(1) and CUDA device executable code 2284 based on stored HIP source code 2230.

[0229] An example kernel can be generated by the CUDA-to-HIP translation tool 2220 from Fig. 22 according to at least one embodiment. CUDA source code 2210 partitions an overarching problem, designed to be solved by a given kernel, into relatively coarse subproblems that can be solved independently using thread blocks. Each thread block contains any number of threads. Each subproblem can be partitioned into relatively fine pieces that can be solved cooperatively in parallel by threads within a thread block. Threads within a thread block can cooperate by sharing data through shared memory and by synchronizing execution to coordinate memory accesses.

[0230] CUDA source code 2210 can organize thread blocks associated with a given kernel into a one-dimensional, two-dimensional, or three-dimensional lattice of thread blocks. Each thread block can contain any number of threads, and a lattice can contain any number of thread blocks.

[0231] A kernel can be a function in device code defined using a "__global__" declaration specifier. The dimension of a grid that executes a kernel for a given kernel call and associated streams can be specified using a CUDA kernel start syntax. CUDA kernel start syntax is specified as "KernelName<<<GridSize, BlockSize, SharedMemorySize, Stream> >>(KernelArguments);“ An execution configuration syntax can include a construct “<<<...>>” inserted between a kernel name (“KernelName”) and a list of kernel arguments (“KernelArguments”) enclosed in parentheses. CUDA kernel startup syntax can include a CUDA startup function syntax instead of an execution configuration syntax.

[0232] `GridSize` can be of type `dim3` and specify the dimension and size of a grid. Type `dim3` can be a CUDA-defined structure containing unsigned integers `x`, `y`, and `z`. If `z` is not specified, it defaults to one. If `y` is not specified, it defaults to one. The number of thread blocks in a grid can be equal to the product of `GridSize.x`, `GridSize.y`, and `GridSize.z`. `BlockSize` can be of type `dim3` and specify the dimension and size of each thread block. The number of threads per thread block can be equal to the product of `BlockSize.x`, `BlockSize.y`, and `BlockSize.z`. Each thread executing a kernel can be given a unique thread ID, accessible within the kernel through a built-in variable (e.g., `threadIdx`).

[0233] Regarding CUDA kernel startup syntax, `SharedMemorySize` can be an optional argument that specifies the number of bytes in shared memory that are dynamically allocated per thread block for a given kernel call, in addition to statically allocated memory. By default, `SharedMemorySize` can be zero. Similarly, `Stream` can be an optional argument that specifies an associated stream and is zero by default to specify a standard stream. A stream can be a sequence of instructions (possibly issued by different host threads) that execute sequentially. Different streams can execute instructions out of order or concurrently.

[0234] CUDA source code 2210 can include a kernel definition for an example kernel "MatAdd" and a main function. The main function can be host code that runs on a host and includes a kernel call that causes the kernel MatAdd to run on a device. The kernel MatAdd can add two matrices A and B of size NxN, where N is a positive integer, and store the result in a matrix C. The main function can define a threadsPerBlock variable as 16 by 16 and a numBlocks variable as N / 16 by N / 16. The main function can then make the kernel call "MatAdd<<<numBlocks, threadsPerBlock> >(A, B, C);“ specify. According to the CUDA kernel startup syntax, Kernel MatAdd can be executed using a grid of thread blocks with a dimension of N / 16 by N / 16, where each thread block has a dimension of 16 by 16.Each thread block can contain 256 threads, a grid can be created with enough blocks to have one thread per matrix element, and each thread in such a grid can execute Kernel MatAdd to perform pairwise addition.

[0235] While CUDA source code 2210 is translated into HIP source code 2230, the CUDA-to-HIP translation tool 2220 can translate any kernel call in CUDA source code 2210 from CUDA kernel launch syntax to HIP kernel launch syntax and can convert any number of other CUDA calls in source code 2210 into any number of other functionally similar HIP calls. HIP kernel launch syntax can be specified as "hipLaunchKernelGGL(KernelName, GridSize, BlockSize, SharedMemorySize, Stream, KernelArguments);". Each of the following elements—KernelName, GridSize, BlockSize, ShareMemorySize, Stream, and KernelArguments—can have the same meaning in HIP kernel startup syntax as in CUDA kernel startup syntax (described earlier herein). The SharedMemorySize and Stream arguments may be required in HIP kernel startup syntax and may be optional in CUDA kernel startup syntax.

[0236] A portion of HIP source code 2230 can be identical to a portion of CUDA source code 2210, with the exception of a kernel call that causes Kernel MatAdd to run on a device. Kernel MatAdd can be defined in HIP source code 2230 with the same "__global__" declaration specifier used to define Kernel MatAdd in CUDA source code 2210. A kernel call in HIP source code 2230 might be "hipLaunchKernelGGL(MatAdd, numBlocks, threadsPerBlock, 0, 0, A, B, C);", while a corresponding kernel call in CUDA source code 2210 might be "MatAdd<<numBlocks, threadsPerBlock> >(A, B, C);“ is.

[0237] Other implementations are being considered and can be carried out similarly to the CUDA and HIP implementations above, such as oneAPI, OpenCL, and other programming platforms. Code can be translated in either direction. For example, CUDA can be translated to HIP, and CUDA can be translated to OpenCL. SnuCL-Tr and CUCL can be used to translate OpenCL to CUDA and CUDA to OpenCL, respectively. Compiled code or intermediate representations (e.g., CUDA-PTX code) can also be translated to run on other processor platforms (e.g., AMD or Intel). For example, PTX code can be translated to run on Intel or AMD processors using a translation tool such as ZLUDA.

[0238] One or more of the techniques described herein can use a oneAPI programming model. A oneAPI programming model can refer to a programming model for interacting with different compute accelerator architectures. OneAPI can refer to an application programming interface (API) designed to interact with different compute accelerator architectures. A oneAPI programming model can use a DPC++ programming language. A DPC++ programming language can refer to a high-level language for data-parallel programming productivity. A DPC++ programming language can be based, at least in part, on C and / or C++ programming languages. A oneAPI programming model can be a programming model such as, but not limited to, those developed by Intel Corporation of Santa Clara, CA.

[0239] OneAPI and / or the oneAPI programming model can be used to interact with various accelerator, GPU, processor, and / or variations thereof architectures. OneAPI can include a set of libraries that implement different functionalities. OneAPI can include at least one oneAPI DPC++ library, oneAPI mathematical kernel library, oneAPI data analytics library, oneAPI deep neural network library, oneAPI collective communication library, oneAPI threading component library, oneAPI video processing library, and / or variations thereof.

[0240] A oneAPI DPC++ library, also known as oneDPL, can be a library that implements algorithms and functions to accelerate DPC++ kernel programming. OneDPL can implement one or more Standard Template Library (STL) functions. OneDPL can implement one or more parallel STL functions. OneDPL can provide a set of library classes and functions, such as, but not limited to, parallel algorithms, iterators, function object classes, range-based APIs, and / or variations thereof. OneDPL can implement one or more classes and / or functions from a C++ standard library. OneDPL can implement one or more random number generator functions.

[0241] A oneAPI mathematical kernel library, also known as oneMKL, can be a library that implements various optimized and parallelized routines for different mathematical functions and / or operations. OneMKL can implement one or more densely populated linear algebra routines from Basic Linear Algebra Subroutines (BLAS) and / or Linear Algebra Packages (LAPACK). OneMKL can implement one or more sparsely populated BLAS linear algebra routines. OneMKL can implement one or more random number generators (RNGs). OneMKL can implement one or more vector mathematics (VM) routines for mathematical operations on vectors. OneMKL can implement one or more Fast Fourier Transform (FFT) functions.

[0242] A oneAPI data analytics library, also known as oneDAL, can comprise a library that implements various data analysis applications and distributed computations. OneDAL can implement various algorithms for preprocessing, transformation, analysis, modeling, validation, and decision-making for data analytics, in batch, online, and distributed processing modes. OneDAL can implement various C++ and / or Java APIs and various connectors to one or more data sources. OneDAL can implement DPC++ API extensions to a traditional C++ interface and enables GPU utilization for various algorithms.

[0243] A oneAPI Deep Neural Network library, also known as oneDNN, can include a library that implements various deep learning functions. OneDNN can implement various neural network, machine learning, and deep learning functions, algorithms, and / or variations thereof.

[0244] A oneAPI collective communication library, also known as oneCCL, can include a library that implements various applications for deep learning and machine learning workloads. OneCCL can be built on top of lower-level communication middleware, such as, but not limited to, the Message Passing Interface (MPI) and libfabrics. OneCCL can enable a number of deep learning-specific optimizations, such as, but not limited to, prioritization, persistent operations, out-of-order execution, and / or variations thereof. OneCCL can implement various CPU and GPU functions.

[0245] A oneAPI threading component library, also known as oneTBB, can include a library that implements various parallelized processes for different applications. OneTBB can be used for task-based, shared parallel programming on a single host. OneTBB can implement generic parallel algorithms. OneTBB can implement concurrent containers. OneTBB can implement a scalable memory allocator. OneTBB can implement a work-stealing task scheduler. OneTBB can implement low-level synchronization primitives. OneTBB can be compiler-independent and usable on various processors, such as, but not limited to, GPUs, PPUs, CPUs, and / or variations thereof.

[0246] A oneAPI video processing library, also known as oneVPL, can include a library used to accelerate video processing in one or more applications. OneVPL can implement various video decoding, encoding, and processing functions. OneVPL can implement various functions for media pipelines on CPUs, GPUs, and other accelerators. OneVPL can implement device discovery and selection in media-centric and video analytics workloads. OneVPL can implement API primitives for zero-copy buffer sharing.

[0247] A oneAPI programming model can use a DPC++ programming language. A DPC++ programming language can include functionally similar versions of CUDA mechanisms to define device code and distinguish between device code and host code. A DPC++ programming language can include a subset of the functionality of a CUDA programming language. One or more operations of the CUDA programming model can be performed using a oneAPI programming model with a DPC++ programming language.

[0248] Each application programming interface (API) described herein can be compiled by a compiler, interpreter, or other software tool into one or more instructions, operations, or any other signal. Compilation can involve generating one or more machine-executable instructions, operations, or other signals from source code. An API compiled into one or more instructions, operations, or other signals, when executed, can cause one or more processors, such as, but not limited to, processors located, for example, in a computer, to execute a command. Fig. 7-19, or any other logic circuit further described herein, perform one or more arithmetic operations.

[0249] In at least one embodiment, translation tools described elsewhere herein, such as, but not limited to, one or more circuits to translate CUDA code, to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as described above in relation to Fig. 1-5 discussed, to HIP, oneAPI, OpenCL, or another language used to perform any of the operations described above or elsewhere herein. One or more circuits can be configured by software to translate CUDA code to use one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals, as discussed above in relation to Fig. 1-5 discussed, to HIP, oneAPI, OpenCL or any other language used to perform any of the operations described above or elsewhere herein. AUTONOMOUS VEHICLE

[0250] Fig. Figure 23 illustrates an example of an autonomous vehicle 2300 according to at least one embodiment. Autonomous vehicle 2300 (hereinafter referred to alternatively as "vehicle 2300") can be a passenger vehicle, such as, but not limited to, a car, a truck, a bus, and / or another type of vehicle that can accommodate one or more passengers. In at least one embodiment, vehicle 2300 can be a semi-trailer truck used for carrying cargo. Vehicle 2300 can be an aircraft, a robotic vehicle, or another type of vehicle.

[0251] Autonomous vehicles can be described in terms of automation levels defined by the National Highway Traffic Safety Administration (“NHTSA”), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (“SAE”) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (e.g., Standard No. J3016-201806, published on June 15, 2018, Standard No. J3016-201609, published on September 30, 2016, and previous and future versions of this standard). In at least one embodiment, Vehicle 2300 can be capable of functionality corresponding to one or more of the Level 1 through Level 5 autonomous driving levels. For example, in at least one embodiment, Vehicle 2300 can be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment.

[0252] Vehicle 2300 may include components such as, but not limited to, a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 2300 may include a propulsion system 2350, such as, but not limited to, an internal combustion engine, a hybrid electric power plant, a purely electric motor, and / or another type of propulsion system. Propulsion system 2350 may be connected to a drivetrain of vehicle 2300, which may include a transmission to enable propulsion of vehicle 2300. Propulsion system 2350 may be controlled in response to receiving signals from a throttle / accelerator(s) 2352.

[0253] A steering system 2354, which may include a steering wheel, is used to steer vehicle 2300 (e.g., along a desired path or route) when propulsion system 2350 is in operation (e.g., when vehicle 2300 is in motion). Steering system 2354 can receive signals from steering actuator(s) 2356. A steering wheel may be optional for full automation (Level 5) functionality. A brake sensor system 2346 can be used to operate vehicle brakes in response to receiving signals from brake actuator(s) 2348 and / or brake sensors.

[0254] Controllers 2336, which may include one or more system-on-chips (“SoCs”) and / or graphics processing units (“GPUs”), can provide signals (e.g., representing commands) to one or more components and / or systems of vehicle 2300. For example, controllers 2336 may send signals to operate vehicle brakes via brake actuator(s) 2348, to operate steering system 2354 via steering actuator(s) 2356, and to operate propulsion system 2350 via throttle / accelerator(s) 2352. Controllers 2336 may include one or more onboard (e.g., integrated) computing devices that process sensor signals and issue operating commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving vehicle 2300.Controller 2336 can include a first controller for autonomous driving functions, a second controller for functional safety functions, a third controller for artificial intelligence functionality (e.g., computer vision), a fourth controller for infotainment functionality, a fifth controller for emergency redundancy, and / or other controllers. A single controller can handle two or more of the above functionalities; two or more controllers can handle a single functionality and / or any combination thereof.

[0255] Controller 2336 can provide signals to control one or more components and / or systems of the vehicle 2300 in response to sensor data received from one or more sensors (e.g. sensor inputs). Sensor data can be received, for example from global navigation satellite system (“GNSS”) sensor(s) 2358 (e.g., Global Positioning System sensor(s)), radar sensor(s) 2360, ultrasonic sensor(s) 2362, lidar sensor(s) 2364, inertial measurement unit (“IMU”) sensor(s) 2366 (e.g., accelerometer(s), gyroscope(s), a magnetic compass or compasses, magnetometer(s), etc.), microphone(s) 2396, stereo camera(s) 2368, wide-angle camera(s) 2370 (e.g., fisheye cameras), infrared camera(s) 2372, surround camera(s) 2374 (e.g., 360-degree cameras), long-range cameras 2398, medium-range camera(s) 2376, speed sensor(s) 2344 (e.g.for measuring vehicle speed 2300), vibration sensor(s) 2342, steering sensor(s) 2340, brake sensor(s) (e.g. as part of brake sensor system 2346) and / or other sensor types.

[0256] One or more of the controllers 2336 can receive inputs (e.g., represented by input data) from an instrument cluster 2332 of vehicle 2300 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (“HMI”) display 2334, an audible signaling device (annunciator), a loudspeaker, and / or other components of vehicle 2300. Outputs can include information such as, but not limited to, vehicle speed, velocity, time, map data (e.g., a high-definition map (not shown)), location data (e.g., the location of vehicle 2300, such as, but not limited to, on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by controller 2336, etc. For example, the HMI display 2334 can provide information about the presence of one or more objects (e.g.,(a road sign, warning sign, traffic light change, etc.) and / or display information about driving maneuvers that the vehicle has made, is making, or will make (e.g., change lanes now, take exit 34B in two miles, etc.).

[0257] Each of the components, features and systems of vehicle 2300 in Fig. Vehicle 2300 can be connected via a bus 2302. Bus 2302 can include a CAN data interface (referred to herein alternatively as a "CAN bus"). A CAN bus can be a network within Vehicle 2300 that is used to assist in controlling various features and functionality of Vehicle 2300, such as, but not limited to, actuation of brakes, acceleration, braking, steering, windshield wipers, etc. Bus 2302 can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). Bus 2302 can be read to obtain steering wheel angle, ground speed, engine revolutions per minute ("RPMs"), button positions, and / or other vehicle status indicators. Bus 2302 can be a CAN bus that is ASIL B compliant.

[0258] In addition to or as an alternative to CAN, FlexRay and / or Ethernet protocols can be used. Any number of buses can constitute Bus 2302, which may include zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using different protocols. Two or more buses can be used to perform different functions and / or for redundancy. For example, a first bus can be used for collision avoidance functionality, and a second bus can be used for actuation control. Each bus of Bus 2302 can communicate with any component of Vehicle 2300, and two or more buses of Bus 2302 can communicate with corresponding components.Each of any number of system(s) on chip(s) (“SoC(s)”) 2304 (such as, but not limited to, SoC 2304(A) and SoC 2304(B)), each of controller(s) 2336 and / or each computer within vehicle can have access to the same input data (e.g. inputs from sensors of vehicle 2300) and can be connected to a common bus, such as CAN bus.

[0259] Any number of cameras can be used with any choice of camera locations and fields of view for autonomous vehicle 2100. Fig. 21A are positioned according to at least one embodiment. Cameras and their respective fields of view are exemplary embodiments and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or cameras may be located at different locations on vehicle 2100.

[0260] Camera types may include digital cameras designed for use with components and / or systems of Vehicle 2100. Camera(s) may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. Camera types may be capable of any frame rate, such as, but not limited to, 60 frames per second (fps), 1220 fps, 240 fps, etc., depending on the embodiment. Cameras may be capable of using rolling shutters, global shutters, another type of shutter, or a combination thereof. In at least one embodiment, the color filter array may include a red clear clear clear (RCCC) color filter array, a red clear clear blue (RCCB) color filter array, a red blue green clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array.Clear pixel cameras, such as, but not limited to, cameras with an RCCC, an RCCB and / or an RBGC color filter array, can be used in an effort to increase light sensitivity.

[0261] One or more cameras can be used to perform advanced driver assistance systems (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono camera can be installed to provide functions including lane departure warning, traffic sign recognition, and intelligent headlight control. One or more cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0262] One or more cameras can be mounted in an assembly, such as, but not limited to, a custom-designed (three-dimensional ("3D") printed) assembly to filter out stray light and reflections from within the vehicle (e.g., reflections from the dashboard reflected in the side mirrors) that can interfere with camera image data acquisition capabilities. With regard to side mirror mounting assemblies, side mirror assemblies can be custom 3D printed so that a camera mounting plate conforms to the shape of a side mirror. Camera(s) can be integrated into the side mirror(s). For side-view cameras, camera(s) can also be integrated into four pillars at each corner of a cabin.

[0263] Cameras with a field of view that includes portions of the environment in front of the vehicle (e.g., forward-facing cameras) can be used for surround view to help identify forward paths and obstacles, and to help provide information, with the aid of one or more of the controller(s) and / or control SoCs, that is critical for generating an occupancy grid and / or determining preferred vehicle paths. Forward-facing cameras can be used to perform many similar ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS functions and systems, including Lane Departure Warnings (“LDW”), Autonomous Cruise Control (“ACC”), and / or other functions, such as, but not limited to, traffic sign recognition.

[0264] A variety of cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform incorporating a CMOS (Complementary Metal Oxide Semiconductor) color imager. A long-range camera 2370 can be used to detect objects coming into view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Any number (including zero) of long-range cameras 2370 can be mounted on the vehicle 2300. Any number of long-range camera(s) 2398 (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. Long-range camera(s) 2398 can also be used for object detection and classification, as well as basic object tracking.

[0265] Any number of stereo camera(s) 2368 can also be included in a forward-facing configuration. One or more of the stereo camera(s) 2368 can include an integrated control unit that has a scalable processing unit, which can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the environment of vehicle 2300, including a distance estimate for all points in an image. One or more of the stereo camera(s) 2368 can include compact stereo vision sensor(s), which can include two camera lenses (one each on the left and right) and an image processing chip that measures the distance from vehicle 2300 to the target object and generates information (e.g.,Metadata) can be used to activate autonomous emergency braking and lane departure warning functions. Other types of stereo camera(s) 2368 can be used in addition to or as an alternative to those described here.

[0266] Cameras with a field of view that includes portions of the surroundings and sides of vehicle 2300 (e.g., side-view cameras) can be used for surround view, providing information used to create and update an occupancy grid and to generate side-impact collision warnings. For example, surround camera(s) 2374 (e.g., four surround cameras) could be positioned on vehicle 2300. Surround camera(s) 2374 can include any number and combination of wide-view cameras, fisheye camera(s), 360-degree camera(s), and / or similar cameras. For example, four fisheye cameras could be positioned on the front, rear, and sides of vehicle 2300. Vehicle 2300 can use three surround-view camera(s) 2374 (e.g. left, right and rear) and can utilize one or more other camera(s) (e.g. a forward-facing camera) as a fourth surround-view camera.

[0267] Cameras with a field of view that includes portions of the area behind vehicle 2300 (e.g., rear-view cameras) can be used for parking assistance, surround view, rear collision warnings, and creating and updating an occupancy grid. A wide variety of cameras can be used, including, but not limited to, cameras that can also be used as forward-facing cameras (e.g., long-range cameras 2398 and / or medium-range cameras 2376, stereo cameras 2368, infrared cameras 2372, etc.), as described herein.

[0268] Vehicle 2300 may include any number of SoCs 2304 or other processors described elsewhere herein, such as, but not limited to, processors and / or components designed for Fig. Figures 7-19 illustrate and describe the SoCs 2304. Each SoC 2304 can include central processing units (“CPU(s)”) 2306, graphics processing units (“GPU(s)”) 2308, processor(s) 2310, cache(s) 2312, accelerators 2314, data storage 2316, and / or other components and features not illustrated. SoC(s) 2304 can be used to control vehicle 2300 in a variety of platforms and systems. For example, SoC(s) 2304 can be combined in a system (e.g., vehicle 2300 system) with a high-definition (“HD”) map 2322, which can receive map refreshes and / or updates via network interface 2324 from one or more servers (not shown).

[0269] CPU(s) 2306 may contain a CPU cluster or CPU complex (hereinafter referred to as "CCPLEX"). CPU(s) 2306 may contain multiple cores and / or Level 2 ("L2") caches. For example, CPU(s) 2306 may contain eight cores in a coherent multiprocessor configuration. CPU(s) 2306 may contain four dual-core clusters, each cluster including a dedicated L2 cache (e.g., a 2-megabyte (MB) L2 cache). CPU(s) 2306 (e.g., CCPLEX) may be configured to support concurrent cluster operations, allowing any combination of CPU(s) 2306 clusters to be active at any given time.

[0270] One or more CPU(s) 2306 can implement power management capabilities that include one or more of the following features: individual hardware blocks can be automatically clocked while idle to save dynamic power; each core can be clocked when that core is not actively executing due to the execution of Wait for Interrupt (“WFI”) / Wait for Event (“WFE”) instructions; each core can be clocked independently; each core cluster can be clocked independently if all cores can be clocked; and / or each core cluster can be clocked independently if all cores can be clocked. CPU(s) 2306 can further implement an extended power state management algorithm, where allowed power states and expected wake-up times can be specified, and hardware / microcode determines the optimal power state for the core, cluster, and CCPLEX.Processing kernels can support simplified performance state entry sequences in software, offloading work to microcode.

[0271] GPU(s) 2308 may include an integrated GPU (referred to herein alternatively as an "iGPU"). GPU(s) 2308 may be programmable and efficient for parallel workloads. GPU(s) 2308 may use an extended tensor instruction set. GPU(s) 2308 may include one or more streaming microprocessors, each of which may include a level one ("L1") cache (e.g., an L1 cache of at least 96 KB), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache of 512 KB). GPU(s) 2308 may include at least eight streaming microprocessors. GPU(s) 2308 may use compute application programming interface(s) (API(s)). GPU(s) 2308 can use one or more parallel computing platforms and / or programming models (e.g. NVIDIA's CUDA model).Streaming microprocessors can be referred to as streaming multiprocessors (“SMs”), stream processors (“SPs”), stream processing units (“SPUs”), compute units (“CUs”), execution units (“EUs”) and / or slices, where a slice in this context can refer to a portion of processing resources in a processing unit (e.g., 16 cores, a ray tracing unit, a thread judge or scheduler).

[0272] One or more GPU(s) 2308 can be performance-optimized for best performance in automotive and embedded applications. For example, GPU(s) 2308 could be manufactured using FinFET (Fin-Fin) circuitry. Each streaming microprocessor can incorporate a number of mixed-precision processing cores partitioned into multiple blocks. For example, 64 PF32 cores and 32 FP64 cores could be partitioned into four processing blocks. Each processing block could be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, a zero-level ("L0") instruction cache, a scheduler (e.g., warp scheduler) or sequencer, a dispatch unit, and / or a 64 KB register file.Streaming microprocessors can include independent parallel integer and floating-point data paths to provide efficient execution of workloads involving a mix of computation and addressing calculations. Streaming microprocessors can include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combined L1 data cache and shared memory to improve performance while simplifying programming.

[0273] One or more GPU(s) 2308 can incorporate high-bandwidth memory (“HBM”) and / or a 16 GB HBM2 memory subsystem to provide a peak memory bandwidth of approximately 900 GB / second in some examples. In addition to or as an alternative to HBM memory, synchronous graphics random-access memory (“SGRAM”) can be used, such as, but not limited to, synchronous graphics double-rate type five random-access memory (“GDDR5”).

[0274] GPU(s) 2308 can incorporate unified memory technology. Support for Address Translation Services (“ATS”) can be used to allow GPU(s) 2308 to directly access CPU(s) 2306 page tables. If a GPU(s) 2308 memory management unit (MMU) encounters a mismatch, an address translation request can be sent to CPU(s) 2306. In response, two CPU(s) 2306 can search their page tables for a virtual-to-physical mapping for an address and send the translation back to GPU(s) 2308. Unified memory technology can allow a single unified virtual address space for memory from both CPU(s) 2306 and GPU(s) 2308, simplifying GPU(s) 2308 programming and porting applications to GPU(s) 2308.

[0275] GPU(s) 2308 can contain any number of access counters that track the frequency of GPU(s) 2308 accesses to the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses pages most frequently, thereby improving efficiency for memory areas shared between processors.

[0276] One or more SoC(s) 2304 can include any number of Cache(s) 2312, including those described herein. For example, Cache(s) 2312 could include a Level 3 (“L3”) cache available to both CPU(s) 2306 and GPU(s) 2308 (connected, for example, to CPU(s) 2306 and GPU(s) 2308). Cache(s) 2312 can include a write-back cache capable of tracking line states, such as, but not limited to, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). An L3 cache can include 4 MB of memory or more, depending on the implementation, although smaller cache sizes are possible.

[0277] One or more SoC(s) 2304 can include one or more Accelerators 2314 (e.g., hardware accelerators, software accelerators, or a combination thereof). SoC(s) 2304 can include a hardware acceleration cluster, which may contain optimized hardware accelerators and / or large on-chip memory. Large on-chip memory (e.g., 4 MB SRAM) can enable a hardware acceleration cluster to accelerate neural networks and other computations. A hardware acceleration cluster can be used to complement GPU(s) 2308 and offload some tasks from GPU(s) 2308 (e.g., to free up more GPU cycles for other tasks). Accelerator 2314 could be used for targeted workloads (e.g., perception, convolutional neural networks (“CNNs”), recurrent neural networks (“RNNs”), etc.) that may be stable enough to be accessible for acceleration.A CNN can include region-based or regional Convolutional Neural Networks (“RCNNs”) and fast RCNNs (e.g., as used for object detection) or another type of CNN.

[0278] Accelerator 2314 (e.g., hardware acceleration cluster) can include one or more deep learning accelerators (“DLAs”). DLAs can include one or more tensor processing units (“TPUs”), which can be configured to provide an additional ten trillion operations per second for deep learning applications and inference, such as TPUs in Fig. 17. TPUs can be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). DLA(s) can further be optimized for a specific set of neural network types and floating-point operations, as well as inference. Designed DLA(s) can deliver more performance per millimeter than a typical general-purpose GPU and typically far exceeds the performance of a CPU. TPU(s) can perform multiple functions, including a single-instance convolution function that supports, for example, INT8, INT16, and FP16 data types for both features and weights, as well as post-processor functions.DLA(s) can quickly and efficiently execute neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, including, for example: a CNN for object identification and detection using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety and / or security-related events.

[0279] DLA(s) can perform any function of GPU(s) 2308, and by using an inference accelerator, for example, a designer can target either DLA(s) or GPU(s) 2308 for any given function. For example, a designer can focus CNN processing and floating-point operations on DLA(s) and leave other functions to GPU(s) 2308 and / or accelerator(s) 2314.

[0280] Accelerator 2314 may include a programmable vision accelerator (“PVA”), which may alternatively be referred to herein as a computer vision accelerator. PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (“ADAS”) 2338, autonomous driving, augmented reality (“AR”) applications, and / or virtual reality (“VR”) applications. PVA can provide a balance between performance and flexibility. For example, each PVA may include any number of reduced instruction set computer (“RISC”) cores, direct memory access (“DMA”), and / or any number of vector processors.

[0281] RISC cores can interact with image sensors (e.g., image sensors from any camera described herein), image signal processor(s), and so on. Each RISC core can include any amount of memory. Depending on the implementation, RISC cores can use any of a number of protocols. RISC cores can run a real-time operating system (RTOS). RISC cores can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, RISC cores could include an instruction cache and / or tightly coupled RAM.

[0282] DMA can enable PVA components to access system memory independently of the CPU(s) 2306. DMA can support any number of features used to provide optimization to a PVA, including support for multidimensional addressing and / or circular addressing. DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0283] Vector processors can be programmable processors designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. A PVA can include a PVA core and two vector processing subsystem partitions. A PVA core can include a processor subsystem, DMA engine(s) (e.g., two DMA engines), and / or other peripherals. A vector processing subsystem can act as the primary processing engine of a PVA and can include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core can include a digital signal processor, such as, but not limited to, a single-instruction, multiple-data digital signal processor (SIMD) or a very long instruction word (VLIW). A combination of SIMD and VLIW can improve throughput and speed.

[0284] Each vector processor can include an instruction cache and can be coupled with dedicated memory. As a result, each vector processor can be configured to execute independently of other vector processors. Vector processors contained within a single PVA can be configured to utilize data parallelism. For example, multiple vector processors contained within a single PVA can execute a common computer vision algorithm, but on different regions of an image. Vector processors contained within a single PVA can simultaneously execute different computer vision algorithms on an image, or even execute different algorithms on sequential images or portions of an image.Among other things, any number of PVAs can be included in hardware acceleration clusters, and any number of vector processors can be included in each PVA. PVAs can also include additional error correction code (ECC) memory to improve overall system security.

[0285] Accelerator 2314 can include an on-chip computer vision network and static random-access memory (“SRAM”) to provide high-bandwidth, low-latency SRAM for Accelerator 2314. On-chip memory can include a minimum of 4 MB of SRAM, including, for example, eight field-configurable memory blocks accessible by both a PVA and a DLA. Each pair of memory blocks can include an extended peripheral bus (“APB”) interface, configuration switching technology, a controller, and a multiplexer. Any type of memory can be used. A PVA and a DLA can access memory via a backbone that provides high-speed memory access to both the PVA and a DLA. A backbone can include an on-chip computer vision network that connects a PVA and a DLA to memory (e.g., using APB).

[0286] An on-chip computer vision network can include an interface that, prior to the transmission of any control signal / address / data, determines that both a PVA and a DLA provide complete and valid signals. An interface can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transmission. An interface can conform to the standards of the International Organization for Standardization (ISO) 26262 or the International Electrotechnical Commission (IEC) 61508, although other standards and protocols may be used.

[0287] One or more SoC(s) 2304 can include a real-time ray-tracing hardware accelerator. A real-time ray-tracing hardware accelerator can be used to quickly and efficiently determine the positions and dimensions of objects (e.g., within a world model), to generate real-time visualization simulations, for radar signal interpretation, for sound propagation synthesis and / or analysis, for simulating SONAR systems, for general wave propagation simulation, for comparison with lidar data for localization and / or other functions, and / or for other uses.

[0288] Accelerator 2314 can encompass a wide range of uses for autonomous driving. A PVA can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of a PVA can align well with algorithmic domains that require predictable processing with low power consumption and low latency. In other words, a PVA can perform well on semi-densely populated or densely populated regular computation, even on small datasets, which might require predictable runtimes with low latency and low power consumption. In Vehicle 2300, PVAs could be designed to execute classical computer vision algorithms, as they can be efficient in object detection and operation on integer mathematics. For example, a PVA is used to perform computer stereo vision. A semi-global adaptation-based (SGB 2314) PVA can be used to perform computer stereo vision.A matching-based algorithm can be used in some examples, although this is not intended to be restrictive. Level 3-5 autonomous driving applications use on-the-fly motion estimation / stereo matching (e.g., structure from motion, pedestrian detection, lane detection, etc.). A PVA can perform computer stereo vision functions on inputs from two monocular cameras. A PVA can be used to perform dense optical flow processing. For example, a PVA could process raw radar data (e.g., using a 4D fast Fourier transform) to provide processed radar data. A PVA is used for runtime depth processing, for example, by processing raw runtime data to provide processed runtime data.

[0289] A DLA (Dynamic Load Alert) can be used to power any type of network to improve control and driving safety, including, for example, a neural network that outputs a confidence measure for each object detection. Confidence can be represented or interpreted as a probability or as providing a relative "weight" for each detection compared to other detections. A confidence measure allows a system to make further decisions about which detections should be considered true positives rather than false positives. A system can set a confidence threshold and consider only detections that exceed this threshold as true positives. If an automatic emergency braking (AEB) system is used, false positives can cause the vehicle to automatically perform emergency braking, which is obviously undesirable.High-confidence detections can be considered triggers for AEB. A DLA can run a neural network for a regressing confidence value. A neural network can take as its input at least a subset of parameters, such as, but not limited to, bounding box dimensions, ground plane estimation (obtained, for example, from another subsystem), output from IMU sensor(s) 2366 correlated with orientation, distance, 3D position estimates of the object obtained by the neural network, and / or other sensors (e.g., LIDAR sensor(s) 2364 or RADAR sensor(s) 2360), among others.

[0290] One or more SoC(s) 2304 can include data storage 2316 (e.g., memory). Data storage 2316 can be on-chip memory of SoC(s) 2304 capable of storing neural networks to be executed on GPU(s) 2308 and / or a DLA. Data storage 2316 can be large enough to store multiple instances of neural networks for redundancy and security. Data storage 2316 can include L2 or L3 cache(s).

[0291] One or more SoC(s) 2304 can include any number of Processor(s) 2310 (e.g., embedded processors). Processor(s) 2310 can include a boot and power management processor, which can be a dedicated processor and subsystem for handling boot power and management functions and associated security enforcement. A boot and power management processor can be part of a boot sequence of SoC(s) 2304 and can provide runtime power management services. A boot power and management processor can provide clock and voltage programming, support for system low-power state transitions, management of thermal and temperature sensors of SoC(s) 2304, and / or management of SoC(s) 2304 power states.Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC(s) 2304 can use ring oscillators to detect temperatures of CPU(s) 2306, GPU(s) 2308, and / or accelerator(s) 2314. If it is determined that temperatures exceed a threshold, then a boot and power management processor can enter a temperature fault routine and put SoC(s) 2304 into a lower power state and / or put vehicle 2300 into a chauffeur-in-safe-stop mode (e.g., safely stop vehicle 2300).

[0292] The 2310 processor(s) can also include a number of embedded processors that can serve as an audio processing engine, which can be an audio subsystem providing full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. An audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0293] The 2310 processor(s) can also include an always-on processor engine, which can provide the necessary hardware features to support low-power sensor management and wake-up use cases. An always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0294] The 2310 processor(s) can also include a security cluster engine, which may contain a dedicated processor subsystem for handling security management for automotive applications. A security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (such as timers, an interrupt controller, etc.), and / or routing logic. In a security mode, two or more cores can operate in lockstep mode and function as a single core with comparison logic to detect differences between their operations. The 2310 processor(s) can also include a real-time camera engine, which may contain a dedicated processor subsystem for handling real-time camera management.Processor(s) 2310 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of a camera processing pipeline.

[0295] The 2310 processor(s) can include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce a final image for a player window. A video image compositor can perform lens distortion correction on the 2370 wide-view camera(s), 2374 omnidirectional camera(s), and / or cabin monitoring camera sensor(s). The cabin monitoring camera sensor(s) can preferably be monitored by a neural network running on a separate instance of the 2304 SoC, configured to identify cabin events and respond accordingly.A cabin system can perform lip reading to activate cellular service and make a phone call, dictate emails, change a vehicle's destination, activate or change a vehicle's infotainment system and settings, or provide voice-activated web browsing. Certain functions may be available to a driver when a vehicle is operating in autonomous mode and may be disabled otherwise.

[0296] A video image compositor can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if there is motion in a video, noise reduction appropriately weights spatial information, thereby reducing the weight of information provided by neighboring frames. Where an image or portion of an image does not contain motion, temporal noise reduction performed by the video image compositor can use information from a previous image to reduce noise in the current image.

[0297] A video image compositor can also be configured to perform stereo rectification on input stereolens frames. Furthermore, a video image compositor can be used for user interface composition when an operating system desktop is in use, and GPU(s) 2308 may not be required to continuously render new surfaces. If GPU(s) 2308 are allocated power and actively performing 3D rendering, a video image compositor can be used to offload GPU(s) 2308 to improve performance and responsiveness.

[0298] One or more SoCs of SoC(s) 2304 may further include a Mobile Industry Processor Interface (“MIPI”) camera serial interface for receiving video and receiving input from cameras, a high-speed interface, and / or a video input block that can be used for a camera and related pixel input functions. One or more SoC(s) 2304 may further include one or more input / output controllers that can be controlled by software and used to receive I / O signals that may not be assigned to a specific role.

[0299] One or more SoCs of the SoC(s) 2304 series can further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio encoders / decoders ("codecs"), power management, and / or other devices. SoC(s) 2304 can be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet channels), sensors (e.g., LiDAR sensor(s) 2364, RADAR sensor(s) 2360, etc., which may be connected via Ethernet channels), data from bus 2302 (e.g., vehicle speed 2300, steering wheel position, etc.), data from GNSS sensor(s) 2358 (e.g., connected via an Ethernet bus or a CAN bus), etc. One or more SoCs of SoC(s) 2304 can further include dedicated high-performance mass storage controllers, which may include their own DMA engines and which can be used to offload CPU(s) 2306 from routine data management tasks.

[0300] The 2304 SoC(s) can be an end-to-end platform with a flexible architecture spanning automation levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The 2304 SoC(s) can be faster, more reliable, and even more energy-efficient and compact than conventional systems. For example, when combined with the 2306 CPU(s), 2308 GPU(s), and 2316 data storage, the 2304 accelerator(s) can provide a fast, efficient platform for autonomous vehicles at levels 3-5.

[0301] Computer vision algorithms can be run on CPUs, which can be configured using a high-level programming language, such as, but not limited to, C, to execute a wide variety of processing algorithms over a wide variety of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Many CPUs are unable to execute complex object detection algorithms in real time, which are used in in-vehicle ADAS applications and in practical Level 3-5 autonomous vehicles.

[0302] The embodiments described herein allow multiple neural networks to run simultaneously and / or sequentially and to combine their results to enable autonomous driving functionality at levels 3-5. For example, a CNN running on a DLA or a discrete GPU (e.g., GPU(s) 2320) can include text and word recognition, enabling the reading and understanding of traffic signs, including signs for which a neural network has not been specifically trained. Furthermore, a DLA can include a neural network capable of identifying, interpreting, and providing semantic understanding of a sign and passing this semantic understanding to path planning modules running on a CPU complex.

[0303] Multiple neural networks can operate simultaneously, such as for Level 3, 4, or 5 driving. For example, a warning sign indicating "Caution: Flashing lights indicate icy conditions" can be interpreted independently or collectively by multiple neural networks, along with an electrical warning light. Such a warning sign itself can be identified as a traffic sign by a first neural network (e.g., a trained neural network), while the text "Flashing lights indicate icy conditions" can be interpreted by a second neural network. This second network informs the vehicle's path planning software (preferably running on a CPU complex) that the detection of flashing lights indicates the presence of icy conditions.A flashing light can be identified by operating a third, deployed neural network across multiple frames, informing a vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can run simultaneously, for example, but not limited to, within a DLA and / or on GPU(s) 2308.

[0304] A CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 2300. An always-on sensor processing engine can be used to unlock a vehicle when an owner approaches a driver's door and turns on lights, and to disable such a vehicle in a security mode when an owner leaves it. In this way, SoC(s) 2304 can provide security against theft and / or carjacking.

[0305] A CNN for emergency vehicle detection and identification can use data from microphones 2396 to detect and identify emergency vehicle sirens. SoC(s) 2304 use a CNN to classify environmental and urban sounds, as well as visual data. A CNN running on a DLA is trained to identify the relative approach speed of an emergency vehicle (e.g., using a Doppler effect). A CNN can also be trained to identify emergency vehicles specific to a local area where a vehicle operates, as identified by GNSS sensor(s) 2358. When operating in Europe, a CNN can attempt to detect European sirens, and when operating in North America, a CNN can attempt to identify only North American sirens.Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, slow down a vehicle, pull over to one side of a road, park a vehicle and / or keep a vehicle idling using ultrasonic sensor(s) 2362 until emergency vehicles pass.

[0306] Vehicle 2300 can include CPU(s) 2318 (e.g., discrete CPU(s) or dCPU(s)) that can be coupled to SoC(s) 2304 via a high-speed interconnect (e.g., PCIe). CPU(s) 2318 can, for example, include an x86 processor. CPU(s) 2318 can be used to perform any of a variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and SoC(s) 2304 and / or monitoring the status and health of Controller(s) 2336 and / or an infotainment system on a single chip (“infotainment SoC”) 2330. SoC(s) 2304 can include one or more interconnects, and an interconnect can include a Peripheral Component Interconnect Express (PCIe).

[0307] Vehicle 2300 can include GPU(s) 2320 (e.g., discrete GPU(s) or dGPU(s)) that can be coupled to SoC(s) 2304 via a high-speed interconnect (e.g., NVIDIA's NVLINK channel). GPU(s) 2320 can provide additional artificial intelligence functionality, such as, but not limited to, running redundant and / or different neural networks, and can be used to train and / or update neural networks, at least partially, based on inputs (e.g., sensor data) from sensors of a Vehicle 2300.

[0308] Vehicle 2300 may also include network interface 2324, which may contain wireless antenna(s) (e.g., one or more wireless antennas 2326 for various communication protocols, such as, but not limited to, a cellular antenna, a Bluetooth antenna, etc.). Network interface 2324 can be used to enable wireless connectivity to internet cloud services (e.g., with server(s) and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct link can be established between vehicle 2300 and another vehicle, and / or an indirect link can be established (e.g., via networks and the internet). Direct links can be provided using a vehicle-to-vehicle communication link.A vehicle-to-vehicle communication link can provide Vehicle 2300 with information about vehicles in its vicinity (e.g., vehicles in front of, to one side of, and / or behind Vehicle 2300). Such functionality can be part of a cooperative adaptive cruise control functionality of Vehicle 2300.

[0309] Network interface 2324 can include a SoC that provides modulation and demodulation functionality, enabling controller 2336 to communicate over wireless networks. Network interface 2324 can include a radio frequency front-end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. Frequency conversions can be performed in any technically feasible way. For example, frequency conversions could be performed using well-known processes and / or superheterodyne receiver processes. Radio frequency front-end functionality can be provided by a separate chip. Network interfaces can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0310] Vehicle 2300 may further include data storage 2328, which may include off-chip (e.g., off-SoC(s) 2304) memory. Data storage 2328 may include one or more memory elements, including RAM, SRAM, dynamic random-access memory (“DRAM”), video random-access memory (“VRAM”), flash memory, hard disks, and / or other components and / or devices capable of storing at least one data bit.

[0311] Vehicle 2300 can further include GNSS sensor(s) 2358 (e.g., GPS and / or supported GPS sensors) to assist with mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensor(s) 2358 can be used, including, for example, a GPS that uses a USB connector with an Ethernet-to-serial bridge (e.g., RS-232).

[0312] Vehicle 2300 may also include RADAR sensor(s) 2360. RADAR sensor(s) 2360 can be used by Vehicle 2300 for long-range vehicle detection, even in darkness and / or severe weather conditions. RADAR functional safety levels can be ASIL B. RADAR sensor(s) 2360 can use a CAN bus and / or bus 2302 (e.g., to transmit data generated by RADAR sensor(s) 2360) for control and to access object tracking data, with access to Ethernet channels to access raw data in some examples. A wide variety of RADAR sensor types can be used. For example, RADAR sensor(s) 2360 can be suitable for front, rear, and side RADAR use. One or more of the RADAR sensor(s) 2360 is a pulse Doppler RADAR sensor.

[0313] RADAR sensor(s) 2360 can include various configurations, such as, but not limited to, long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. Long-range RADAR can be used for adaptive cruise control functionality. Long-range RADAR systems can provide a wide field of view, achieved through two or more independent scans, such as, but not limited to, within a 250 m (meter) range. RADAR sensor(s) 2360 can help distinguish between stationary and moving objects and can be used by the ADAS system 2338 for emergency braking assistance and forward collision warning. Sensor(s) 2360 included in a long-range RADAR system can include monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface.With six antennas, a central four-antenna system can generate a focused beam pattern designed to map the area around vehicle 2300 at higher speeds with minimal interference from traffic in adjacent lanes. Two additional antennas can extend the field of view, enabling the rapid detection of vehicles entering or exiting vehicle 2300's lane.

[0314] Medium-range radar systems, for example, can have a range of up to 160 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 150 degrees (rear). Short-range radar systems can include any number of radar sensor 2360 designed to be installed at both ends of a rear bumper. When installed at both ends of a rear bumper, a radar sensor system can create two beams that continuously monitor blind spots in a reversing direction and alongside a vehicle. Short-range radar systems can be used in ADAS system 2338 for blind spot detection and / or lane change assistance.

[0315] Vehicle 2300 may also include ultrasonic sensor(s) 2362. Ultrasonic sensor(s) 2362, which may be positioned at the front, rear, and / or side of vehicle 2300, may be used for parking assistance and / or for creating and updating an occupancy grid. A wide variety of ultrasonic sensor(s) 2362 may be used, and different ultrasonic sensor(s) 2362 may be used for different detection ranges (e.g., 2.5 m, 4 m). Ultrasonic sensor(s) 2362 may operate at functional safety levels of ASIL B.

[0316] Vehicle 2300 can include LiDAR sensor(s) 2364. LiDAR sensor(s) 2364 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. LiDAR sensor(s) 2364 can operate at functional safety level ASIL B. Vehicle 2300 can include multiple LiDAR sensors 2364 (e.g., two, four, six, etc.) that can use one Ethernet channel (e.g., to provide data to a Gigabit Ethernet switch).

[0317] LiDAR sensor(s) 2364 may be capable of providing a list of objects and their distances for a 360-degree field of view. Commercially available LiDAR sensor(s) 2364 may have an advertised range of approximately 100 m, with an accuracy of 2 cm to 3 cm, and with support for a 100 Mbps Ethernet connection, for example. One or more non-protruding LiDAR sensors may be used. LiDAR sensor(s) 2364 may include a small device that can be embedded in a front, rear, side, and / or corner position of vehicle 2300. In such an embodiment, the LIDAR sensor(s) 2364 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m even for objects with low reflectivity. Front-mounted LIDAR sensor(s) 2364 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0318] LiDAR technologies, such as, but not limited to, 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source to illuminate the area around vehicle 2300 up to approximately 200 m. A flash LiDAR unit can include a receiver that records the laser pulse transit time and reflected light at each pixel, corresponding to an area from vehicle 2300 to objects. Flash LiDAR can allow highly accurate and distortion-free images of the environment to be generated with each laser flash. Four flash LiDAR sensors can be used, one on each side of vehicle 2300. 3D flash LiDAR systems include a solid-state 3D staring array LiDAR camera with no moving parts except for a fan (e.g., a non-scanning LiDAR device).Flash-LIDAR device can use a 5 nanosecond Class I (eye-safe) laser pulse per frame and can capture reflected laser light as a 3D area point cloud and co-registered intensity data.

[0319] Vehicle 2300 may further include IMU sensor(s) 2366. IMU sensor(s) 2366 may be located at the center of a rear axle of Vehicle 2300. IMU sensor(s) 2366 may include, for example, accelerometers, magnetometers, gyroscope(s), a magnetic compass, magnetic compasses, and / or other sensor types. In six-axis applications, but not limited thereto, IMU sensor(s) 2366 may include accelerometers and gyroscopes. In nine-axis applications, but not limited thereto, IMU sensor(s) 2366 may include accelerometers, gyroscopes, and magnetometers.

[0320] The IMU sensor(s) 2366 can be implemented as a miniature, GPS-Aided Inertial Navigation System (“GPS / INS”) that combines microelectromechanical system (“MEMS”) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and orientation. The IMU sensor(s) 2366 can enable the vehicle 2300 to estimate its course without requiring input from a magnetic sensor by directly observing and correlating velocity changes from a GPS to the IMU sensor(s) 2366. The IMU sensor(s) 2366 and GNSS sensor(s) 2358 can be combined in a single integrated unit.

[0321] Vehicle 2300 may contain microphone(s) 2396, which are located in and / or around vehicle 2300. Microphone(s) 2396 may be used, among other things, for emergency vehicle detection and identification.

[0322] Vehicle 2300 may further include any number of camera types, including stereo camera(s) 2368, wide-angle camera(s) 2370, infrared camera(s) 2372, omnidirectional camera(s) 2374, long-range camera(s) 2398, medium-range camera(s) 2376, and / or other camera types. Cameras may be used to capture image data around the entire periphery of Vehicle 2300. The types of cameras used may depend on the specific Vehicle 2300. Any combination of camera types may be used to provide the necessary coverage around Vehicle 2300. The number of cameras used may vary depending on the embodiment. For example, Vehicle 2300 could include six cameras, seven cameras, ten cameras, twelve cameras, or any other number of cameras. Cameras may, as an example, support Gigabit Multimedia Serial Link (“GMSL”) and / or Gigabit Ethernet communications. Each camera could have been described in more detail beforehand.

[0323] Vehicle 2300 may also include vibration sensor(s) 2342. Vibration sensor(s) 2342 may measure vibrations of components of vehicle 2300, such as, but not limited to, axle(s). For example, changes in vibration may indicate a change in road surfaces. If two or more vibration sensors 2342 are used, differences in vibrations may be used to determine friction or slippage of road surfaces (e.g., if there is a difference in vibration between a driven axle and a freely rotating axle).

[0324] Vehicle 2300 may include ADAS system 2338. ADAS system 2338 may include a SoC in some examples. ADAS system 2338 may include any number and combination of an autonomous / adaptive / automatic cruise control ("ACC") system, a cooperative adaptive cruise control ("CACC") system, a forward collision warning ("FCW") system, an automatic emergency braking ("AEB") system, a lane departure warning ("LDW") system, a lane keeping assist ("LKA") system, a blind spot warning ("BSW") system, a rear cross traffic alert ("RCTW") system, a collision warning ("CW") system, a lane centering ("LC") system, and / or other systems, features, and / or functionality.

[0325] The ACC system can use radar sensor(s) 2360, lidar sensor(s) 2364, and / or any number of camera(s). The ACC system can include a longitudinal ACC system and / or a lateral ACC system. A longitudinal ACC system monitors and controls the distance to another vehicle immediately in front of vehicle 2300 and automatically adjusts vehicle 2300's speed to maintain a safe distance from vehicles ahead. A lateral ACC system performs distance control and advises vehicle 2300 to change lanes when necessary. Lateral ACC refers to other ADAS applications, such as, but not limited to, lane keeping assist (LC) and cruise control (CW).

[0326] A CACC system uses information from other vehicles, which can be received via network interface 2324 and / or wireless antenna(s) 2326 from other vehicles via a wireless link or indirectly via a network connection (e.g., via the Internet). Direct links can be provided by a vehicle-to-vehicle (“V2V”) communication link, while indirect links can be provided by an infrastructure-to-vehicle (“I2V”) communication link. In general, V2V communication provides information about vehicles immediately ahead (e.g., vehicles immediately in front of and in the same lane as vehicle 2300), while I2V communication provides information about traffic further ahead. A CACC system can incorporate one or both I2V and V2V information sources.Given information from vehicles ahead of vehicle 2300, a CACC system can be more reliable and has the potential to improve traffic flow smoothness and reduce congestion on the road.

[0327] An FCW system is designed to warn a driver of a hazard so that the driver can take corrective action. An FCW system uses a forward-facing camera and / or radar sensor(s) 2360, coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to provide driver feedback, such as, but not limited to, a display, a speaker, and / or a vibrating component. An FCW system can provide a warning, such as, but not limited to, an audible signal, a visual warning, vibration, and / or a rapid braking pulse.

[0328] An AEB system detects an impending forward collision with another vehicle or object and can automatically apply the brakes if a driver does not take corrective action within a specified time or distance parameter. The AEB system may use forward-facing camera(s) and / or radar sensor(s), coupled with a dedicated processor, DSP, FPGA, and / or ASIC. When an AEB system detects a hazard, it will typically first warn a driver to take corrective action to avoid a collision. If that driver fails to take corrective action, the AEB system can automatically apply the brakes to prevent or at least mitigate the impact of a predicted collision. An AEB system may include techniques such as, but not limited to, dynamic brake assist and / or anticipatory braking.

[0329] An LDW system provides visual, audible, and / or tactile warnings, such as, but not limited to, steering wheel or seat vibrations, to warn the driver if the vehicle crosses lane markings. An LDW system is not activated if a driver indicates an intentional lane departure, such as, but not limited to, activating a turn signal. An LDW system may use forward-facing cameras coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to provide driver feedback, such as, but not limited to, a display, speaker, and / or vibrating component. A LKA system is a variation of an LDW system. An LKA system provides steering input or braking to correct the vehicle if the vehicle begins to leave its lane.

[0330] A Blind Spot Warning (BSW) system detects and warns a driver of vehicles in a car's blind spot. A BSW system can provide a visual, audible, and / or tactile warning to indicate that merging or changing lanes is unsafe. A BSW system can provide an additional warning if a driver uses a turn signal. A BSW system can use rear-facing camera(s) and / or radar sensor(s), coupled with a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback, such as, but not limited to, a display, speaker, and / or vibrating component.

[0331] An RCTW system can provide visual, audible, and / or tactile alerts when an object is detected outside the rear camera's field of view while the vehicle is reversing. An RCTW system includes an AEB system to ensure that vehicle braking can be applied to avoid a collision. An RCTW system can use one or more rear-facing radar sensors, coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to provide driver feedback, such as, but not limited to, a display, speaker, and / or vibrating component.

[0332] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for a driver, but are typically not catastrophic because conventional ADAS systems warn the driver and allow them to decide whether a safety condition truly exists and act accordingly. In the case of conflicting results, the vehicle itself decides whether to heed the result from a primary computer or a secondary computer (e.g., a first controller or a second controller of controllers 2336). For example, ADAS system 2338 can be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. A backup computer rationality monitor can run redundant diverse software on hardware components to detect errors in perception and dynamic driving tasks.Output from ADAS System 2338 can be provided to a supervisory MCU. If output from a primary computer and output from a secondary computer conflict, a supervisory MCU can determine how to reconcile the conflict to ensure safe operation.

[0333] A primary computer can be configured to provide a supervisory MCU with a confidence score indicating that primary computer's confidence in a selected outcome. If this confidence score exceeds a threshold, the supervisory MCU can follow the direction of that primary computer, regardless of whether the secondary computer provides a conflicting or inconsistent result. If a confidence score does not meet a threshold, and if the primary and secondary computers display different results (e.g., a conflict), a supervisory MCU can mediate between the computers to determine an appropriate outcome.

[0334] A supervising MCU can be configured to run a neural network or networks that are trained and configured to determine, at least in part based on outputs from a primary computer and outputs from a secondary computer, conditions under which the latter will provide false alarms. The neural network or networks in a supervising MCU can learn when the output of a secondary computer can be trusted and when it cannot. For example, if this secondary computer is a radar-based FCW system, the neural network or networks in this supervising MCU can learn when the FCW system identifies metallic objects that cannot actually pose a hazard, such as, but not limited to, a drainage grate or manhole cover, triggering an alarm.If a secondary computer is a camera-based lane departure warning (LDW) system, a neural network in a supervising MCU can learn to override the LDW when cyclists or pedestrians might be present and leaving the lane is indeed the safest maneuver. A supervising MCU can include at least one DLA or GPU suitable for running neural networks or neural networks with associated memory. A supervising MCU can feature a SoC 2304 and / or include it as a component.

[0335] The ADAS system 2338 can include a secondary computer that performs ADAS functionality using traditional computer vision rules, and this secondary computer can use classic computer vision rules (if-then). The presence of a neural network or neural networks in a supervisory MCU can improve reliability, security, and performance. For example, diverse implementation and intentional non-identity make an overall system more fault-tolerant, especially to errors caused by software (or software-hardware interface) functionality.For example, if a software bug or error exists in software running on a primary computer, and non-identical software code running on a secondary computer provides a consistent overall result, then a supervising MCU can have greater confidence that an overall result is correct, and a bug in software or hardware on that primary computer will not cause a material defect.

[0336] An output from ADAS System 2338 can be fed into a perception block and / or a dynamic driving task block of a primary computer. For example, if ADAS System 2338 displays a forward collision warning due to an object immediately ahead, a perception block can use this information when identifying objects. A secondary computer can include its own trained neural network, thus reducing the risk of false positives, as described herein.

[0337] V...

Claims

A processor comprising: one or more circuits for using one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals. The processor according to claim 1, wherein the variable sampling rate of one or more audio signals is used to apply a matching bias, learned from encoded audio data of the one or more audio signals as part of cross-modal multi-head self-attention in a decoder of the one or more neural networks. The processor according to claim 1, wherein generating image object features is further based on the application of a temporal bias learned from encoded audio data of one or more audio signals as part of causal multi-head self-attention in a decoder of one or more neural networks. The processor according to claim 1, wherein the one or more circuits further generate a video that is at least partially based on the image object features. The processor according to claim 1, wherein the one or more audio signals are encoded, at least partially based on the application of a sparse multi-head self-awareness in an audio encoder in the one or more neural networks. The processor according to claim 1, wherein the one or more neural networks include an autoregressive motion decoder to generate the image object features. The processor according to claim 1, wherein the object is a face and wherein the one or more audio signals are speech. A method comprising: using one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals. The method according to claim 8, wherein the variable sampling rate of one or more audio signals is used to apply a matching bias learned from encoded audio data of the one or more audio signals as part of cross-modal multi-head self-attention in a decoder of the one or more neural networks. The method according to claim 8, wherein generating image object features is further based on the application of a temporal bias learned from encoded audio data of one or more audio signals as part of causal multi-head self-attention in a decoder of one or more neural networks. The method according to claim 8, further comprising generating a video based at least partially on the image object features. The method according to claim 8, wherein the one or more audio signals are encoded, at least partially based on the application of a sparse multi-head self-awareness in an audio encoder in the one or more neural networks. The method according to claim 8, wherein the one or more neural networks include an autoregressive motion decoder to generate the image object features. The method according to claim 8, wherein the object is a face and wherein the one or more audio signals are speech. A system comprising: one or more processors for using one or more neural networks to generate image object features based at least partially on a variable sampling rate of one or more audio signals; and one or more memories for storing weights of the one or more neural networks. The system according to claim 15, wherein the variable sampling rate of one or more audio signals is used to apply a matching bias, learned from encoded audio data of the one or more audio signals as part of cross-modal multi-head self-attention in a decoder of the one or more neural networks. The system according to claim 15, wherein generating image object features is further based on the application of a temporal bias learned from coded audio data of one or more audio signals as part of causal multi-head self-attention in a decoder of one or more neural networks. System according to claim 15, wherein the one or more processors further generate a video based at least partially on the image object features. The system according to claim 15, wherein the one or more audio signals are encoded, at least partially based on the application of sparse multi-head self-attention in an audio encoder in the one or more neural networks. System according to claim 15, wherein the object is a face and wherein one or more audio signals are speech.

Images