Neural networks for generating objects in different images

Abstract

Devices, processors, computer systems, appliances, non-transitory computing media, and / or methods for using neural networks to generate multiple contiguous images. In at least one embodiment, a processor includes circuitry for using one or more neural networks to generate multiple images, each image containing a similar object (e.g., the same item) and different backgrounds.For example, a processor includes one or more circuits for using one or more neural networks to identify one or more objects (e.g., an animal, a vehicle, a person) within two or more different images (e.g., different backgrounds such as weather, season, environment) based at least in part on one or more statements (e.g., text prompts) from one or more users providing statements about at least one of the two or more different images (e.g., objects and / or backgrounds for each image in text such as adjectives and nouns) that are not related to the one or more objects.

Description

TECHNICAL FIELD

[0001] At least one embodiment relates to processors, computer systems, devices, non-transitory computing media, and / or methods for using neural networks to generate multiple contiguous images (e.g., images including a same object in different poses). In at least one embodiment, a processor includes circuitry for using one or more neural networks to generate multiple images, each image including a same object (e.g., the same object) in different backgrounds. BACKGROUND

[0002] Training and using neural networks to generate images based on text input can be challenging. For example, it can be difficult to generate training data (e.g., labeled images) that can be used to train a neural network to identify objects (e.g., animals) in images. Training can also be difficult because it requires computational resources such as processing power and memory space. If neural networks are not adequately trained and / or there is not enough processing power available to train them, neural networks may not produce accurate results. Using neural networks to generate images is also challenging because images can be complex (e.g., thousands of pixels, where objects within an image have a specific orientation and location relative to other objects).Accordingly, there is a need to improve neural networks that generate images and to find ways to improve the training of these neural networks. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows a computing environment including a neural network in accordance with at least one embodiment; Fig. 2 shows a block diagram for training a neural network to generate different images comprising a same object, according to at least one embodiment; Fig. 3 shows another block diagram for training a neural network to generate different images comprising a same object, according to at least one embodiment; Fig. 4 shows another block diagram for training a neural network to generate images with a same object according to at least one embodiment; Fig. 5 shows another block diagram for training a neural network to generate images with a same object according to at least one embodiment; Fig. 6 shows an example of a neural network generating images comprising a same object against different backgrounds, according to at least one embodiment; Fig. 7 is a process flow diagram for training a neural network to generate an object within different images according to at least one embodiment; Fig. 8 is another process flow diagram for training a neural network to generate an object within different images according to at least one embodiment; Fig. 9 is a flowchart showing an example of a process of inferencing using one or more neural networks to generate one or more objects within two or more different images; Fig. 10 shows an example including a processor and modules according to at least one embodiment; Fig. 11 is a block diagram illustrating a driver and / or runtime including one or more libraries to provide one or more application programming interfaces (APIs) according to at least one embodiment; Fig. 12A shows logic according to at least one embodiment; Fig. 12B shows the logic according to at least one embodiment; Fig. 13 shows the training and deployment of a neural network according to at least one embodiment; Fig. 14 shows an example of a data center system according to at least one embodiment; Fig. 15A shows an example of an autonomous vehicle according to at least one embodiment; Fig. Figure 15B shows an example of camera positions and fields of view for the autonomous vehicle of Fig. 15A, according to at least one embodiment; Fig. 15C is a block diagram illustrating an example system architecture for the autonomous vehicle of Fig. 15A according to at least one embodiment; Fig. 15D is a diagram illustrating a system for communication between one or more cloud-based servers and the autonomous vehicle of Fig. 15A according to at least one embodiment; Fig. 16 is a block diagram showing a computer system according to at least one embodiment; Fig. 17 is a block diagram illustrating a computer system according to at least one embodiment; Fig. 18 shows a computer system according to at least one embodiment; Fig. 19 shows a computer system according to at least one embodiment; Fig. 20A shows a computer system according to at least one embodiment; Fig. 20B shows a computer system according to at least one embodiment; Fig. 20C shows a computer system according to at least one embodiment; Fig. 20D shows a computer system according to at least one embodiment; Fig. 20E and Fig. 20F show a common programming model according to at least one embodiment; Fig. 21 shows exemplary integrated circuits and associated graphics processors according to at least one embodiment; Fig. 22A and Fig. 22B illustrate exemplary integrated circuits and associated graphics processors according to at least one embodiment; Fig. 23A and Fig. 23B illustrate additional exemplary graphics processor logic according to at least one embodiment; Fig. 24 shows a computer system according to at least one embodiment; Fig. 25A shows a parallel processor according to at least one embodiment; Fig. 25B shows a partition unit according to at least one embodiment; Fig. 25C shows a processing cluster according to at least one embodiment; Fig. 25D shows a graphics multiprocessor according to at least one embodiment; Fig. 26 shows a multi-graphics processing system (GPU) according to at least one embodiment; Fig. 27 shows a graphics processor according to at least one embodiment; Fig. 28 is a block diagram illustrating a processor microarchitecture for a processor according to at least one embodiment; Fig. 29 shows a deep learning application processor according to at least one embodiment; Fig. 30 is a block diagram illustrating an exemplary neuromorphic processor according to at least one embodiment; Fig. 31 shows at least portions of a graphics processor according to one or more embodiments; Fig. 32 shows at least portions of a graphics processor according to one or more embodiments; Fig. 33 shows at least portions of a graphics processor according to one or more embodiments; Fig. 34 is a block diagram of a graphics processing engine of a graphics processor in accordance with at least one embodiment; Fig. 35 is a block diagram of at least portions of a graphics processor core according to at least one embodiment; Fig. 36A and Fig. 36B illustrates thread execution logic including an array of processing elements of a graphics processor core, according to at least one embodiment; Fig. 37 shows a parallel processing unit (“PPU”) according to at least one embodiment; Fig. 38 shows a general processing cluster (“GPC”) according to at least one embodiment; Fig. 39 shows a memory partition unit of a parallel processing unit ("PPU") according to at least one embodiment; Fig. 40 shows a streaming multiprocessor according to at least one embodiment; Fig. 41 is an example data flow diagram for an advanced computing pipeline, according to at least one embodiment; Fig. 42 is a system diagram for an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment; Fig. 43 includes an example illustration of an advanced computer pipeline 4210A for processing image data in accordance with at least one embodiment; Fig. 44A includes an example data flow diagram of a virtual instrument supporting an ultrasound device, in accordance with at least one embodiment; Fig. 44B includes an example data flow diagram of a virtual instrument supporting a CT scanner, in accordance with at least one embodiment; Fig. 45A shows a data flow diagram for a process for training a machine learning model in accordance with at least one embodiment; Fig. 45B is an example illustration of a client-server architecture for enhancing annotation tools with pre-trained annotation models, according to at least one embodiment; and Fig. 46 shows components of a system for accessing a large language model according to at least one embodiment. DETAILED DESCRIPTION

[0003] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of at least one embodiment. However, it will be apparent to one skilled in the art that the inventive concepts may be practiced without one or more of these specific details.

[0004] In at least one embodiment, processors use a neural network to receive different text prompts and generate images of the same object (e.g., the same dog) against different backgrounds corresponding to different text prompts. For example, a neural network may generate a first image of a fox in the snow and a second image of the same fox in the woods in response to receiving text prompts indicating that images of a "fox in the snow" and a "fox in the woods" are being generated. In at least one embodiment, these images are generated simultaneously, for example, in response to a user inputting different text prompts into an interface. In at least one embodiment, a neural network includes a diffusion neural network, a diffusion model, and / or other software that uses a diffusion neural network to generate images.

[0005] In at least one embodiment, software executed by one or more processors trains a neural network using pairs of images comprising the same object in different settings with corresponding text prompts (e.g., an image of a dog in the woods and an image of the same dog in the snow with corresponding text prompts for the different settings and / or backgrounds). In at least one embodiment, an encoding portion of a neural network identifies features corresponding to an object (e.g., the same item) from both images and is trained to identify the same features corresponding to said object from both images.In at least one embodiment, software executed by one or more processors causes a decoding portion trained to associate features with corresponding keywords from text prompts (e.g., by associating features of the dog with the word "dog" and associating features of snow with "snow") to decode the features identified by the decoding portion. In at least one embodiment, the training causes a neural network to associate specific features with a word or words indicating an object, such that when the word or words are used in different text prompts, the neural network generates different images depicting the same object (e.g., the same object).

[0006] In at least one embodiment, a processor comprises circuitry for using one or more neural networks to generate two or more images depicting a same object (e.g., a same dog, a same cat) in different settings based at least in part on two or more different input text prompts. In at least one embodiment, a processor comprises circuitry for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating content other than the one or more objects in at least one of the two or more different images.In at least one embodiment, a neural network generates images of the same object (e.g., a cat, a dog, a bird, a helicopter) against different backgrounds, with the same object in different poses, states, or positions in each of the images. For example, in response to receiving text prompts to generate images of a dog running in the snow, running in the rain, and swimming underwater, a neural network generates images of the same dog running in the snow, running in the rain, and swimming underwater.

[0007] In at least one embodiment, specific layers of a neural network are trained to recognize which features of an object are common across different images and how different text prompts correspond to the image features. In at least one embodiment, a neural network comprises layers that receive training images of the same object (e.g., images of the same dog against different backgrounds) and learn to recognize which features of the object should be given greater weight because they are present in all images of a (same) object.In at least one embodiment, a layer that receives a feature map, for example, a data structure with image features identified during a convolution of an input image, and generates a new feature map that includes assigned weights for features common to a (same) object in the images (for example, common features are assigned more weight and other features are assigned no or less weight). For example, given an image of a cat, an introspection layer generates a feature map with identified features that includes a higher weight for features that identify the cat as such, for example, the eyes, the ears, or the fur texture.

[0008] In at least one embodiment, a generated feature map comprising identified features and weight values ​​is then input to another layer that identifies image features that correspond to text prompts. In at least one embodiment, a layer receives a feature map (for each input image) and text prompts corresponding to each feature map (e.g., the feature map corresponding to a fox in the snow and the corresponding text prompt "fox in the snow," the feature map corresponding to the fox in the forest and the corresponding text prompt "fox in the forest"), and then outputs a data structure comprising a representation of features of the feature map that correspond to the features of the text prompts.In at least one embodiment, a layer includes a cross-attention layer that receives feature maps from a self-attention layer and text prompts as input and then outputs a data structure comprising a representation of the image features and corresponding features representing text (e.g., image features for green, leaves, and trees are associated with the representation of forest in the data structure). In at least one embodiment, after a data structure is generated, a decoder can be used to decode the features in the data structure to generate images (in response to receiving a text prompt).

[0009] Fig. 1 shows a computing environment 100 including a neural network in accordance with at least one embodiment. In at least one embodiment, the computing environment 100 includes one or more processors, one or more circuits, one or more data centers, or other computer hardware (e.g., graphics processing units) for executing software, neural networks, and other modules as described in Fig. 1. In at least one embodiment, the computing environment 100 includes the neural network 102 executed by one or more processors (e.g., graphics processing units (GPUs), data processing units (DPUs), central processing units (CPUs) as shown in Fig. 20A-40) or otherwise operates. In at least one embodiment, the neural network 102 receives one or more inputs 101. In at least one embodiment, the inputs 101 may be text, images, videos, or a combination thereof. In at least one embodiment, the inputs 101 may be provided by a user (e.g., via an interface), a server, a different neural network, or another source of textual input (e.g., from a large language model, a user interface). In at least one embodiment, the inputs 101 indicate the content of at least two or more different images (e.g., backgrounds, settings).In at least one embodiment, inputs 101 include at least one or more statements from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. For example, inputs 101 may include words other than an object, such as forest, background, snow, weather, and season. In at least one embodiment, neural network 102 uses words in a sentence as content to generate features of a background.

[0010] In at least one embodiment, neural network 102 includes a convolutional layer 103, a self-attention layer 104, and a cross-attention layer 105. In at least one embodiment, neural network 102 includes a diffusion model that is pre-trained and learned to reverse a diffusion process. In at least one embodiment, a diffusion model receives a random noise sample (e.g., from a Gaussian distribution) and applies steps of reverse diffusion to denoise it incrementally (e.g., in each layer of a diffusion neural network). In at least one embodiment, a diffusion model is conditional and uses additional conditional information, such as text descriptions or class labels, to guide generation toward a desired outcome. In at least one embodiment, a diffusion model includes learned parameters for prediction and subtraction of noise at each step.In at least one embodiment, a diffusion model includes a sampling algorithm (e.g., ancestral sampling) that controls, influences, or otherwise modifies a diffusion process. In at least one embodiment, neural network 102 may provide outputs for post-processing steps that refine, modify, or otherwise adapt the output. In at least one embodiment, a processor uses neural network 102 to determine, identify, or otherwise infer a relationship described in input 101, which is transferred to one or more outputs 106a-106n.For example, the inputs may include 101 sentences describing an object (e.g., fox in snow, fox in rain, fox walking on grass), where these sentences have words that represent relationships, and the output images may depict these relationships (e.g., the same fox in these different backgrounds and the same fox performing related actions according to the text inputs).

[0011] In at least one embodiment, inputs 101 may include text prompts, such as "Generate an image of a fox in the snow" and "Generate an image of a fox in a forest." Neural network 102 receives these inputs 101 and generates outputs 106-106n, which include two or more images, each of which includes an image of the fox, but each of the two or more output images includes the fox in a snowy background and the same fox in a forest background. In at least one embodiment, the number of outputs 106a-106n is not limited to four and includes a closed set of outputs corresponding to a number of inputs (e.g., 4, 8, 10, 1000).

[0012] In at least one embodiment, the self-attention layer 104 is, includes, or corresponds to a self-attention layer. In at least one embodiment, a neural network layer may be referred to as a part of a neural network. In at least one embodiment, a neural network layer includes individual units called neurons or weights. In at least one embodiment, a processor executes or trains a self-attention layer 104 that compares all elements of an input sequence and modifies the corresponding output sequences.For example, a self-attention layer discriminates an input sequence for key values ​​for each input and adds the results to the output sequence so that a neural network focuses on significant features of a data modality (e.g., important relevant parts of a text, important relevant parts of an image, such as a common object). In at least one embodiment, the self-attention layer 104 receives features or feature maps from residual blocks.

[0013] In at least one embodiment, the cross-attention layer 105 includes identifying how those features in one data type or modality correlate with features in another data type or modality. In at least one embodiment, a cross-attention layer enables a model to weight different portions of input data relative to a different sequence. In at least one embodiment, the cross-attention layer 105 includes a set of query vectors from one sequence and a set of key and value vectors from another sequence that a processor uses to calculate attention scores using a compatibility function between each query and all keys. In at least one embodiment, the scores determine a weighting of the values, which are then aggregated to form an output of a cross-attention layer.

[0014] Fig. 2 shows a block diagram for training a neural network to generate different images with a same object according to at least one embodiment. Fig. 2 includes the input 201, the neural network 202, text descriptions 204, the neural network 205, the image set 206, and the post-processing 208. In at least one embodiment, the training process 200 performed by one or more processors trains the neural network 102 from Fig. 1 partially or completely. In at least one embodiment, Fig. 2 illustrates a training process 200, which may be stored in software and performed by one or more processors, to generate training data (e.g., pairs of text and images with or without labels). For example, the training process performed by one or more processors may be used to generate training images of a blue helicopter, each image paired with a different text prompt (e.g., one image paired with "blue helicopter flying near grass at sunset," another image paired with "blue helicopter landing on a grassy area," another image paired with "a blue helicopter that landed on the beach," and another image paired with "blue helicopter flying over an ocean," each image including a blue helicopter and a background corresponding to the actions and adjectives in each text prompt).In at least one embodiment, input 201 may include text, images, or a combination of text and images (for example, inputs 101 of FIG. Fig. 1).

[0015] In at least one embodiment, an item of inputs 201 is an object of two further outputs 209 (e.g., a text includes a fox, and the images each include that fox). In at least one embodiment, the neural network 202 may include a diffusion neural network, a transformation neural network, a large language model, or another language processing model. In at least one embodiment, a neural network is provided with a list of items to generate text prompts with an item of inputs 201. In at least one embodiment, a list of items forms the item set {x n , x n+1 , x n+2 , .... N}, where x n , x n+1 , x n+2, N are each different items from the list of items. In at least one embodiment, the input text 201 comprises "dog, cat, fox, mouse," and the neural network generates sentences for each item {dog runs in the rain, cat sleeps on a carriage, fox jumps in a forest, mouse hides under rocks} in response to receiving these inputs. In at least one embodiment, the neural network 202 generates many (e.g., 10, 100) sentences for each item. In at least one embodiment, the neural network 202 generates one or more text descriptions 204 of a scene (e.g., sentences). In at least one embodiment, the neural network 202 generates one or more text descriptions 204 of one or more scenes that include the description sentence {y m , y m+1 , y m+2... M}. Scenes may include, for example, one or more backgrounds, backdrops, settings, environments, surroundings, one or more contexts, or other descriptions of the portion of an image that is not in the foreground. In at least one embodiment, input 201 includes one or more indications from one or more users indicating the content of at least one of the two or more different images other than the one or more objects. For example, input 201 may include words other than an object (e.g., a noun such as a fox or a helicopter), such as forest, background, snow, weather, and season.

[0016] In at least one embodiment, neural network 205 generates images for an image set 206 and associates an object from each text description data set with the images forming text-image pairs (e.g., to generate training data). In at least one embodiment, neural network 205 receives outputs such as text descriptions 204 and generates an image set 205. In at least one embodiment, neural network 205 comprises a robust diffusion model (e.g., SDXL), a generative artificial intelligence model, a diffusion neural network, or a text-image generation neural network.In at least one embodiment, the neural network 205 receives as input a set of item text description data and generates a collage or set of images 206 corresponding to one or more items of the item description data set (e.g., based on an input prompt to generate four images corresponding to the text inputs). In at least one embodiment, for example, the second neural network 206 analyzes each item of an item set {x. n , x n+1 , x n+2 , .... N} from the object text description data set and creates a collage or set of images 206 for the image set {z l , e.g. l+1 , e.g. l+2... L}. In at least one embodiment, each of the images comprising the image set 206 are images of the input object 201 in different poses. In at least one embodiment, the image set 206 includes one or more images of each object in different poses and may also include the differently posed objects against one or more different backgrounds. For example, for the object "fox" in the object text description data set, the neural network 205 generates a fox having different poses and different backgrounds that comprise the image set 203 {z l , e.g. l+1 , e.g. l+2 ... L} for the object "fox". In at least one embodiment, the neural network 205 generates for the object "dog" in the object text description data set a dog with different poses and different backgrounds that form the image set 206 {z l , e.g.l+1 , e.g. l+2 ... L} for the item "dog." In at least one embodiment, this is repeated for each item in the item set and for each text description of the description set.

[0017] In at least one embodiment, the training process 200 includes one or more post-processings 208 of the outputs of the neural network 205 or the image-object pairs. The post-processing 208 is performed by a neural network model, machine learning, artificial intelligence, software, or hardware. The post-processing 208 includes, for each image of the image-object pairs, processing the image through object detection and segmentation to identify the object (e.g., the fox) of each image in the image set (e.g., the foxes in the image set {e.g., l , e.g. l+1 , e.g. l+2... L}) and extract foreground masks. In at least one embodiment, the extracted foreground masks are representations of the pose of each subject, separate from any background image that may be present in the image set 206. In at least one embodiment, post-processing 208 of the collage or set of extracted foreground masks of the image-subject pairs comprises splitting the collage of images 206 into individual outputs 209.

[0018] In at least one embodiment, post-processing 208 of the image-subject pairs includes filtering using a neural network trained to identify text-image pairs. Filtering the image-subject pairs determines whether the image-subject pairs are correct. For example, in at least one embodiment, if the object is a fox and the description is a forest, but the image-subject pair is a fox on the moon, filtering the image-subject pair rejects the fox-moon pair. In at least one embodiment, the filtering is performed, for example, by a Contrastive Language-Image Pretraining (CLIP) model (e.g., a neural network) that scores the image-subject pairs. Any image-subject pairs that score below a threshold (e.g., below 0.95) are filtered out.

[0019] In at least one embodiment, post-processing 208 includes generating outputs 209 that are intermediate images. In at least one embodiment, outputs 209 are images of the one or more objects of input 201 without backgrounds. Outputs 209 are generated by one or more neural networks of post-processing 208 that combine the foreground mask representations of each object's pose with an image of white space to generate outputs 209 of the one or more objects, each with a different pose and a white background. Generating outputs 209 may include using one or more neural networks, machine learning models, artificial intelligence models, software, hardware, or other post-processing processes.

[0020] Fig. 3 shows a training process 300, which may include software executed by one or more processors such as those in Fig. 20A-40. In at least one embodiment, the training process 300 includes one or more components of Fig. 1-2. In at least one embodiment, training process 200 and training process 300 may be combined and performed by one or more processors executing training software to train a neural network or generate training data (e.g., a pair of images and text). In at least one embodiment, training process 300 includes neural network 302 receiving one or more inputs 301 including one or more objects, generating one or more background prompts 303a-303n, coupling one or more intermediate images corresponding to outputs 209 to the one or more background prompts 303a-303n, and generating one or more output images 305 such that text and image pairs are generated and can be used for training.In at least one embodiment, the one or more output images 305 include the one or more items of the subject on one or more different backgrounds described by one or more background prompts 303a-303n.

[0021] In at least one embodiment, the training process 300 includes using a neural network 302 that has been trained to generate one or more background prompts 303a-303n from one or more input texts 301. The neural network 302 may be one or more transformer neural networks, large language models, or other language processing models. In at least one embodiment, the neural network 302 receives as inputs 301 the list of items that comprise the itemset {x n , x n+1 , x n+2 , .... N}, where x n , x n+1 , x n+2, N are each different items from the list of items from input 201. For each item, the neural network 302 generates one or more background prompts 303a-303n. The neural network 302 provides one or more background prompts 303a-303n that comprise a set of background prompts {b n , b n+1 , b n+2, ...B}. For example, given input text that includes the subject matter "fox," neural network 302 generates one or more background prompts 303a-303n. In at least one embodiment, generated background prompts 303a-303n are text that includes a word, phrase, or sentence that describes a background. For example, in at least one embodiment, neural network 301 generates background prompts 303a-303n "in the snow, in the forest, .... in autumn." Neural network 301 combines the subjects of input 301 and the one or more background prompts 303a-303n into an output vector space (e.g., a subject-background prompt vector space). When combining the items with the background prompts, the neural network 302 learns the connection between the text describing the item of the item set 203 and the one or more text descriptions of the one or more background prompts 303a-303n.

[0022] In at least one embodiment, training process 300 includes a neural network 304 that receives as input the item-background prompt vector space of neural network 302 and the outputs 209 of neural network 205, and outputs one or more images 305a-305n of the items of item set 203 against one or more backgrounds described by one or more background prompts 303a-303n. In at least one embodiment, neural network 304 is one or more of a text-to-image diffusion model, a latent text-to-image diffusion model, a stable diffusion inpainting model, or another neural network model trained to associate text with images described by the text.

[0023] In at least one embodiment, the training data from the training processes 200 and 300 ( Fig. 2 or 3) are used by one or more processors to train the neural network 102. For example, images paired with sentences that include the same item in each sentence and each image are used to train the neural network 102. In at least one embodiment, the neural network 102 learns to denoise a noisy image during training. In at least one embodiment, the neural network 102 may be referred to as a joint diffusion model or a multi-diffusion model because it is a diffusion model that has two distributions, for example, one distribution for text and another for images.In at least one embodiment, a joint distribution may comprise a combined distribution of text features and image features, such that this distribution may be sampled, and these samples may be used to denoise and generate the same objects (e.g., the same dog) in different backgrounds according to the text prompt inputs. In at least one embodiment, the neural network 102 learns a joint distribution during training, and after training, the neural network 102 may be used to sample the joint distribution to generate images of the same object against different backgrounds based on different prompts.

[0024] The training is done by ε-prediction and the simplified training objectives introduced by: L=Eϵ~N(0,1),t~[1,T][‖ε−εθ(xt)‖22] where ε θrepresents the network parameterized by θ, T is the number of diffusion steps, x t the noisy t-step version of the ground truth image collage from N images x = [x 1 , x 2 , ..., x N ] is.

[0025] Fig. Figure 4 shows another block diagram for training a neural network to generate images with a similar object according to at least one embodiment. In at least one embodiment, Fig. 4 Use training data from any component or process in Fig. 1-3. In at least one embodiment, a neural network 401 (for example, the neural network 102 of Fig. 1) one or more input images 402 and generates one or more output images 408, wherein each image of the output images 407 comprises a common image object (for example, one of the objects from the object set 203 {x n , xn+1 , x n+2 , .... N}, where x n , x n+1 , x n+2 , N are each different items from the list of items), displayed over one or more different backgrounds. In at least one embodiment, inputs 402 are text, an image, or a combination of text and images (e.g., inputs 101, 201). The inputs may include, for example, prompts to create a fox in the snow, the same fox in a forest, and the same fox jumping over a river.

[0026] In at least one embodiment, the inputs 402 are received by one or more convolutional layers 403 of the neural network 401. The one or more convolutional layers 403 generate a feature map from each of the inputs 402. The feature maps generated by the convolutional layer 403 are received by one or more concatenation layers 404, which concatenate each of the feature maps into a single feature map, with the single feature map being provided to one or more self-attention layers 405. In at least one embodiment, the self-attention layers 405, executed by one or more processors, compare each element of the single feature map to determine which elements are more similar to each other (e.g., that a fox's fur is the same or very similar, that a fox's eyes are the same or very similar).In at least one embodiment, one or more self-attention layers 405 generate another feature map that includes weights that indicate how significant a feature (e.g., how frequent it is) or a representation of a feature (e.g., matrix, feature map) is, so that it can be translated from one domain to another. In at least one embodiment, software executed by one or more processors receives a self-generated feature map and divides it into a corresponding number of individual self-generated feature maps, e.g., each of the self-generated feature maps. In at least one embodiment, each self-attention feature map is received by cross-attention layers 406, which can correlate the self-attention feature maps with text 407 (e.g., text prompts 303a-303n) to generate one or more output images 408.Output images 408 are images that comprise a common image object (for example, one of the objects from the object set 203 {x. n , x n+1 , x n+2 , .... N}, where each of x n , x n+1 , x n+2 , .... N different items from the list of items) displayed against one or more different backgrounds.

[0027] For example, in at least one embodiment, input 402 is the text "fox in the snow." Neural network 401, executed by one or more processors, receives input 402 and generates one or more images 408 of a fox in the snow. In at least one embodiment, inputs 402 are the text "fox in the snow" and "fox in the forest." Neural network 401 receives inputs 402 and generates one or more images 408 of a fox in the snow and a fox in a forest.

[0028] In at least one embodiment, inputs 402 are text and an image. For example, inputs 402 are an image of a fox and the text "fox in the snow." Neural network 401 can capture the image of the fox from input 402 and generate one or more images 408 of the fox against a snowy background.

[0029] Fig. 5 shows another block diagram for training a neural network to generate images with a similar object according to at least one embodiment. In at least one embodiment, neural network 501 includes one or more neural networks or one or more neural network sections, such as neural network section 504. In at least one embodiment, neural network 501 receives one or more inputs (e.g., input 502, input 503). Although in Fig. 5 only two inputs are depicted, the inputs to the neural network 501 may be more than two (e.g., 20 images). In at least one embodiment, the inputs 502, 503 may be text, images, or a combination of text and images. For each input 502, 503, the neural network section 504 (e.g., the convolutional layers 403) generates one or more image maps 505, 506. The one or more feature maps 505, 506 are concatenated by one or more layers 404.

[0030] In at least one embodiment, the concatenation of the one or more feature maps generates a single image map 507. In at least one embodiment, the feature map 507 is processed by one or more self-attention layers (e.g., scaled dot production attention). Self-attention layers 405 weigh the importance of different elements of the concatenated image map 507 and enable the self-attention layers 405 to capture relationships and dependencies between the elements of the concatenated image map 507. Each element of the concatenated image map 507 is associated with a feature vector. For each element in the concatenated image map 507, the self-attention layer 405 generates query vectors 508, key vectors 509, and value vectors 510. The query vector 508 captures the relationship between the elements of the concatenated image map 507.Key vectors 509 compare the current element with all other elements of the sequence. Value vectors 510 contain information about the current element.

[0031] In at least one embodiment, the self-attention layer 405 generates attention scores for each element of the concatenated image map 507 by point-producing the query vectors 508 with the key vectors 509 of all other elements. The attention score indicates how much attention the current element should pay to each of the other elements in the sequence.

[0032] In at least one embodiment, each attention score is passed through a softmax function to generate attention weights. Attention weights reflect the importance of each item relative to the current item. Items with higher weights are considered more relevant.

[0033] In at least one embodiment, the concatenated image map 507 is input to one or more self-attention layers 405 of the neural network 501. The self-attention layers 405 extend the information content of the concatenated image map 507 by including information about the context of the inputs (e.g., 502, 503). The self-attention layers 405 establish dependencies between the foreground images of the one or more input images (e.g., 502, 503) through self-attention of the concatenated image map 507. Additionally, the self-attention layer draws dependencies between the backgrounds of the one or more inputs (e.g., 502, 502). In effect, the self-attention layers 405 separate the foreground image from the background image through self-attention.

[0034] In at least one embodiment, the output of the self-attention layer 511 is processed to generate one or more processed feature maps (e.g., 512, 513). In at least one embodiment, the processing includes dividing the output of the self-attention layers 508 into a number of processed feature maps (512, 513) equal to the number of input images (502, 503). In at least one embodiment, for example, if two input images (e.g., 502, 503) are input to the self-attention layer 405, the output of the self-attention layer 405 is divided into two processed feature maps (e.g., 512, 513).

[0035] In at least one embodiment, the processed feature maps 512, 513 are input to the cross-attention layer 406. Additional inputs to the cross-attention layer 406 include one or more text prompts 514, 515 (e.g., text prompts 407). Although only two text prompts 514, 515 are depicted, any finite number of text prompts can be input to the cross-attention layer 406.

[0036] In at least one embodiment, layer 406 generates query vectors 516, key vectors 517, and value vectors 518 for each element of the processed feature maps 512, 513 input to cross-attention layer 406. Query vectors 516 capture the relationship between the elements. Key vectors 517 compare the current element to all other elements in the sequence. Value vectors 518 contain information about the current element.

[0037] In at least one embodiment, the cross-attention layer 406 references one or more of the query vectors 516, key vectors 517, and value vectors 518 for each element of the processed feature maps 512, 514 and connects one or more of the vectors 516-518 to input text prompts 514, 515.

[0038] In at least one embodiment, text prompts 514, 515 (e.g., prompts 303a-303n) include a word, phrase, or sentence describing a background. For example, in at least one embodiment, text prompts 514, 515 include "in the snow, in the forest," describing a snowy background and a forested background.

[0039] In at least one embodiment, cross-attention layer 406 outputs images 519, 520. Images 519, 520 are images that include inputs 502, 503 as a foreground image and background images described by prompts 514, 515. Indeed, neural network 501 combines inputs 502, 503 with the backgrounds described in prompts 514, 515, thus generating personalized images 519, 520.

[0040] In at least one embodiment, the neural network 501 generates personalized images 519, 520. For n input images x̂ 1 , x 2 , ..., x n the n input images are part of the generated multi-image group x, such that x = [x̂ 1 , x n , x n+1 , ..., x N ]. When sampling a group of images, for each diffusion step, the backward diffusion output for unknown images x̂ n+1 ,..., x Nretained, while known images are replaced by the forward diffusion output of the corresponding step (for example, the noisy real images x^t1,…,x^tn

[0041] In at least one embodiment, the input images 502, 503 may be better adapted to personalized image generation by adding an additional mask and an additional channel for the input image for diffusion model training.

[0042] In at least one embodiment, a known probability value may be randomly assigned to each image during training. In at least one embodiment, the randomly assigned known probability value may be, for example, 0.5. The mask for known images is set to 1 for all pixels and to 0 for unknown images. In at least one embodiment, all additional input channels are copied from the original images for the known images and set to maps with zeros for the unknown images.

[0043] In at least one embodiment, the training loss is equal to: L=Eϵ~N(0,1),t~[1,T],m~(0.5N)[‖ε−εθ(xt,x^,M)‖22]

[0044] Here, M is the spatial tilling of a binomial vector m; x̂ = x ⊙ M denotes the partially known image group in which the unknown elements are set to zero.

[0045] In at least one embodiment, personalized image generation adapts image generation to user-defined concepts contained in user-provided sample images, which may be diverse and potentially significantly different from the training data. This problem can be addressed by using the sample images as a guide at the time of inference. During image guidance, sampling is performed using the modified score estimates as follows: ε˜(xt,x^,M)=ε^0+λ∑i=0n[εθ(xt,x^,M)−ε^i] where λ represents the leadership strength; ε̂ 0 = ε θ (x t , 0, M) is the unconditional score when all exemplary images are set to zero; ε̂ i is the score for the i-th example image, which is set to zero.

[0046] Image guidance improves the match to the user-defined concept specified in the input images. With the image guidance from the above equation, the sampling distribution is modified to match the sample images with high probability and adapt to the sample images. The accumulation term in the above equation encourages the model to fully utilize the information from each image when there are multiple sample images, allowing it to benefit from a larger number of sample images.

[0047] Fig. 6 shows an example of a neural network that generates images comprising the same object against different backgrounds, according to at least one embodiment. In at least one embodiment, the neural network 600 comprises components and / or processes as described in Fig. 1-5, for example, a processor executing software may train the neural network 600 based on the training process in Fig. 2-4. In at least one embodiment, the neural network 600 receives one or more inputs 601 (e.g., inputs 402) and one or more prompts 602. In at least one embodiment, the inputs 601 may be text, images, or a combination of text and images. In at least one embodiment, the input 601 includes an object 610 (e.g., a fox), as indicated by a bold arrow in Fig. 6. In at least one embodiment, neural network 600 receives only text as input. In at least one embodiment, input 601 includes an image of a fox.

[0048] In at least one embodiment, prompts 602 may be one or more words, phrases, or sentences that describe a background scene. In at least one embodiment, prompts 602 include at least one or more statements from one or more users that indicate the content of at least one of the two or more different images other than the one or more objects (e.g., words that describe backgrounds or features of an image other than an object in an image). For example, inputs 101 may include words that do not describe an object, such as forest, background, snow, weather, and season. In at least one embodiment, neural network 102 uses words in a sentence as content to generate features of a background.In at least one embodiment, prompt 602 includes a phrase such as "Generate images of the fox in a forest with different seasons," where the fox refers to input 601. In at least one embodiment, neural network 600 uses a first portion of a neural network (e.g., neural network 504, convolutional layers 403) to generate feature maps (e.g., 505, 506) from input 601. In at least one embodiment, feature maps (e.g., 505, 506) of input 601 are input to concatenation layers 404, which generate a concatenated image map 507. In at least one embodiment, the neural network 600 includes a second neural network portion (e.g., self-attention layers 405) that generates a concatenated image map 507, query vectors 508, key vectors 509, and value vectors 510 for each element.In at least one embodiment, query vector 508 represents a relationship between the elements of concatenated image map 507. In at least one embodiment, key vectors 509 compare a current element with all other elements in the sequence, and value vectors 510 contain information about the current element.

[0049] In at least one embodiment, the self-attention layer 405 generates an attention score for each element of the concatenated image map 507 by point-generating the query vectors 508 with the key vectors 509 of all other elements. The attention score indicates how much attention the current element should pay to each of the other elements in the sequence.

[0050] In at least one embodiment, each attention score is passed through a softmax function to generate attention weights. Attention weights reflect the importance of each item relative to the current item. Items with higher weights are considered more relevant.

[0051] In at least one embodiment, the concatenated image map 507 is input to one or more self-attention layers 405 of the neural network 501. The self-attention layers 405 extend the information content of the concatenated image map 507 by including information about the context of the inputs (e.g., 502, 503). The self-attention layers 405 establish dependencies between the foreground images of the one or more input images (e.g., 502, 503) through self-attention of the concatenated image map 507. Additionally, the self-attention layer draws dependencies between the backgrounds of the one or more inputs (e.g., 502, 502). In effect, the self-attention layers 405 separate the foreground image from the background image through self-attention.

[0052] In at least one embodiment, the self-attention layer 511 generates one or more feature maps (e.g., 512, 513), where the features are self-attention, e.g., focused on a common feature of the object. In at least one embodiment, the processing comprises dividing the output of the self-attention layers 508 into a number of processed feature maps (512, 513) equal to the number of input images (502, 503). In at least one embodiment, for example, if two input images (e.g., 502, 503) are input to the self-attention layer 405, the output of the self-attention layer 405 is divided into two processed feature maps (e.g., 512, 513).

[0053] In at least one embodiment, the processed feature maps 512, 513 are input to the cross-attention layer 406. Additional inputs to the cross-attention layer 406 include text prompts 603 (e.g., prompts 514, 515, and 407).

[0054] In at least one embodiment, layer 406 generates query vectors 516, key vectors 517, and value vectors 518 for each element of the processed feature maps 512, 513 input to cross-attention layer 406. Query vectors 516 capture the relationship between the elements. Key vectors 517 compare the current element to all other elements in the sequence. Value vectors 518 contain information about the current element.

[0055] In at least one embodiment, the cross-attention layer 406 references one or more of the query vectors 516, key vectors 517, and value vectors 518 for each element of the processed feature maps 512, 514 and connects one or more of the vectors 516-518 to the input text prompt 603.

[0056] In at least one embodiment, text prompt 602 is "Generate images of the fox (e.g., input 601) in a forest with different seasons." In at least one embodiment, cross-attention layer 406 outputs images 603, 604, 605, and 606. Images 603, 604, 605, and 606 are generated by neural network 600. The generated images have as a foreground image input 601 with the object "fox" 611 in different poses. The background of images 603, 604, 605, and 606 are scenes described by prompt 602. In at least one embodiment, image 603 is input 601 with a background scene of a forest in spring. In at least one embodiment, image 604 is input 601 with a background scene of a forest in winter. In at least one embodiment, image 605 having a background scene of a forest in summer is input to 601.In at least one embodiment, image 606 is input with a background scene of a forest in autumn. In this way, neural network 600 generates personalized images of a fox having different poses and background scenes, as requested by text prompt 602. In at least one embodiment, fox 611 appears in different poses, states, or other postures, as shown in images 603, 604, 605, and 606. As shown in images 603, 604, 605, and 606, backgrounds 607, 608, 609, and 610 each have different backgrounds (e.g., 607 is a daytime forest background, 608 is a daytime snow and forest background, 609 is a forest background, but a different forest than in 607 and 608, and 610 is a different forest background with leaves on the ground).In at least one embodiment, backgrounds 607, 608, 609, and 610 are derived from prompts 602 (e.g., forest, daytime, snow, green forest, forest with leaves on the ground in autumn). In at least one embodiment, neural network 600 generates foxes 611 in different poses learned from its training data and / or from denoising images of foxes in different poses using diffusion layers.

[0057] In at least one embodiment, based at least in part on one or more indications, comprises using one or more neural networks (e.g., neural network 102) and / or multiple layers of a neural network to perform weighted operations, softmax operations, or other pooling operations to generate values ​​of image features.

[0058] Fig. 7 is a flowchart illustrating an example of a process 700 for training one or more neural networks to generate one or more objects within two or more different images, according to at least one embodiment. In at least one embodiment, part or all of the process 700 (or other described processes or variations and / or combinations of these processes) may be implemented in Fig. 7 using one or more systems, processors, or communication devices to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating content other than the one or more objects in at least one of the two or more different images. In at least one embodiment, one or more operations performed as part of process 700 may be performed in various orders and combinations depending on the operations described in Fig. 7, including parallel. In at least one embodiment, parts, methods, and / or a system associated with Fig. 7, as described in one of the Fig. 1-6 not exclusively shown, further illustrated.

[0059] In at least one embodiment, a neural network learns to identify features corresponding to an object from one or more inputs 702.

[0060] Inputs may be text, images, or a combination of text and images (e.g., inputs 101). In at least one embodiment, a neural network learns to identify features corresponding to the subject of inputs 201. The list of subjects forms subject set 203 {x n , x n+1 , x n+2 , .... N}, where x n , x n+1 , x n+2 , N are each different items from the list of items.

[0061] In at least one embodiment, the neural network model 704 generates the item set {x n , x n+1 , x n+2, .... N}. In at least one embodiment, a non-limiting example is that given the input 201 "dog, cat, fox, mouse", the first neural network model generates the item set 203 {dog, cat, fox, mouse}.

[0062] In at least one embodiment, the first model of the neural network 706 generates one or more text descriptions 204 of a scene. The neural network 202 generates one or more text descriptions 204 of one or more scenes that include the description set {y m , y m+1 , y m+2 ... M}. Scenes may, for example, include one or more backgrounds, backdrops, settings, environments, surroundings, or other descriptions of the part of an image that is not in the foreground. The neural network 202 combines each item from the item set {x n , x n+1 , x n+2, .... N} with each text description of the one or more scenes of the description set {y m , y m+1 , y m+2 ... M} to generate 706 an object text description dataset. Combining the objects with the text descriptions, the first neural network model learns the connection between each object of the object set 203 and the one or more text descriptions of the scenes of the description set 204.

[0063] In at least one embodiment, the neural network 205 generates 708 one or more images of an image set 206 and connects the subject of the subject text description data set with the one or more images, thus forming subject image pairs. In at least one embodiment, the neural network 205 is a robust diffusion model (e.g., SDXL), a generative artificial intelligence model, a diffusion neural network, or a text and image generating neural network model. In at least one embodiment, the neural network 205 receives as input the subject text description data set and generates 708 a collage or set of images 206 corresponding to the one or more subjects of the subject description data set. In at least one embodiment, the neural network 205 analyzes each subject from the subject set 203 {x n , x n+1 , x n+2, .... N} from the object text description data set and creates the collage or set of images 206 for the image set {z l , e.g. l+1 , e.g. l+2 ... L}. Each of the images comprising the image set 206 are images of the input object 201 in different poses. In at least one embodiment, the image set 206 includes one or more images of each object in different poses and may also include the differently posed objects against one or more different backgrounds. For example, for the object "fox" in the object text description data set, the neural network 205 generates a fox having different poses and different backgrounds that comprise the image set 203 {e.g. l , e.g. l+1 , e.g. l+2... L} for the object "fox". In at least one embodiment, the second neural network 206 generates for the object "dog" in the object text description data set a dog with different poses and different backgrounds that represent the image set 206 {z l , e.g. l+1 , e.g. l+2 ... L} for the object "dog." This is repeated for each object in the object sentence and for each text description in the description sentence.

[0064] In at least one embodiment, after generating the image set 203 {z l , e.g. l+1 , e.g. l+2... L} for each subject 710, the subject with each of the images in image set 206. For example, in at least one embodiment, neural network 205 links the subject "fox" with each of the different images in image set 206, where each of the images in image set 206 includes an image of a fox in a different pose or with a different pose or aspect. By generating these image-subject pairs, the stable diffusion model establishes a link between the subject and the pose or aspect of each image in image set 206.

[0065] In at least one embodiment, the training process 200 includes one or more post-processings 207 of the outputs of the second neural network model or the image-subject pairs. The post-processing 208 is performed by a neural network model, machine learning, artificial intelligence, software, or hardware. The post-processing 208 includes, for each image of the image-subject pairs, processing the image through object detection and segmentation to identify the subject (e.g., the fox) of each image in the image set (e.g., the foxes in the image set {e.g., l , e.g. l+1 , e.g. l+2 ... L}) and extract foreground masks. The extracted foreground masks are representations of the pose of each object, separated from any background image that may be present in the image set 206.

[0066] In at least one embodiment, post-processing 712 (e.g., post-processing 208) of the collage or set of extracted foreground masks of the image-object pairs includes separating the collage of images 206 into individual output images 209.

[0067] In at least one embodiment, post-processing 712 of the image-subject pairs includes filtering using a neural network trained to identify text-image pairs. Filtering the image-subject pairs identifies whether the image-subject pairs are correct. For example, in at least one embodiment, if the object is a fox and the description is a forest, but the image-subject pair is a fox on the moon, filtering the image-subject pair rejects the fox-moon pair. In at least one embodiment, the filtering is performed, for example, by a Contrastive Language-Image Pretraining (CLIP) model (e.g., a neural network) that determines a score for the image-subject pairs. All image-subject pairs whose score is below a threshold (e.g., below 0.95) are filtered out.

[0068] In at least one embodiment, post-processing includes generating outputs 713 (e.g., generating outputs 209) that are intermediate images. The intermediate images 209 are images of the one or more input 201 objects without backgrounds. Intermediate images 209 are generated by one or more neural networks of post-processing 208 that combine the foreground mask representations of each object's pose with a white space image to generate one or more images 209 of the one or more objects, each with a different pose and a white background. Generating intermediate images 209 may include using one or more neural networks, machine learning models, artificial intelligence models, software, hardware, or other post-processing processes.

[0069] Fig. 8 is a flowchart illustrating an example of a process 800 for training one or more neural networks to generate one or more objects in two or more different images, according to at least one embodiment. In at least one embodiment, part or all of the process 800 (or other described processes or variations and / or combinations of these processes) may be implemented in Fig. 8 using one or more systems, processors, or communication devices to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating content other than the one or more objects in at least one of the two or more different images. In at least one embodiment, one or more operations performed as part of process 800 may be performed in various orders and combinations that differ from those described in Fig. 8, including parallel. In at least one embodiment, parts, methods, and / or a system associated with Fig. 8, as described in one of the Fig. 1-7 shown non-exclusively.

[0070] In at least one embodiment, Fig. 8 illustrates the training of a neural network (e.g., a neural network 302), which includes generating 802 one or more background prompts 303a-303n from one or more input texts 301. The neural network may be one or more transformation neural networks, large language models, or other language processing models. In at least one embodiment, the neural network receives as inputs 301 the list of items comprising the item set 203 {x n , x n+1 , x n+2 , .... N}, where each of x n , x n+1 , x n+2 , .... N different items from the list of items from input 201. For each item, the neural network 302 generates one or more background prompts 303a-303n. The neural network 302 provides one or more background prompts 303a-303n that comprise a background prompt set {b n, b n+1 , b n+2, ...B}. For example, given an input comprising the subject matter "fox," the neural network 302 generates one or more background prompts 303a-303n. In at least one embodiment, the generated background prompts 303a-303n are text comprising a word, phrase, or sentence describing a background. For example, in at least one embodiment, the neural network 301 generates the background prompts 303a-303n "in the snow, in the forest, .... in autumn." The neural network 301 combines the subjects of the input 301 and the one or more background prompts 303a-303n into an output vector space (e.g., a subject-background prompt vector space). When combining the items with the background prompts, the neural network 302 learns the connection between the text describing the item of the item set 203 and the one or more text descriptions of the one or more background prompts 303a-303n.

[0071] In at least one embodiment, training the neural network includes receiving 804 object-background prompt vector space of the neural network 302 and receiving 806 intermediate images 209 of the neural network 205.

[0072] In at least one embodiment, training the neural network includes generating 808 one or more images (e.g., images 305a-305n) of the items of item set 203 against one or more backgrounds described by one or more background prompts 303a-303n. In at least one embodiment, neural network 304 is one or more of a text-to-image diffusion model, a latent text-to-image diffusion model, a stable diffusion inpainting model, or another neural network model trained to associate text with images described by the text.

[0073] Fig. Figure 9 is a flowchart illustrating an example of an inferencing process 900 using one or more neural networks to generate one or more objects within two or more different images. In at least one embodiment, the diffusion of images (e.g., diffusion neural network) includes denoising input images.

[0074] In at least one embodiment, inferencing using one or more neural networks includes generating feature maps 902. The neural network (e.g., the diffusion neural network) receives one or more inputs (e.g., input 502, input 503). Although in Fig. 5, the inputs to the neural network can be any finite number of inputs. The inputs 502, 503 can be text, images, or a combination of text and images. For each input 502, 503, the neural network section 504 (e.g., the convolutional layers 403) generates one or more feature maps 505, 506.

[0075] In at least one embodiment, inferencing using one or more neural networks includes concatenating one or more feature maps 904. The one or more feature maps 505, 506 are concatenated by one or more layers 404. In at least one embodiment, concatenating the one or more feature maps produces a single feature map 507.

[0076] In at least one embodiment, inferencing using one or more neural networks includes processing the concatenated feature maps using one or more self-attention neural networks or layers 906. In at least one embodiment, the feature map 507 is processed by one or more self-attention layers 405. Self-attention layers 405 (e.g., scaled dot production attention) weigh the importance of different elements of the concatenated image map 507, allowing the self-attention layers 405 to capture relationships and dependencies between the elements of the concatenated image map 507. Each element of the concatenated image map 507 is associated with a feature vector. For each element in the concatenated image map 507, the self-attention layer 405 generates query vectors 508, key vectors 509, and value vectors 510.The query vector 508 captures the relationship between the elements of the concatenated image map 507. Key vectors 509 compare the current element with all other elements of the sequence. Value vectors 510 contain information about the current element.

[0077] In at least one embodiment, the self-attention layer 405 generates attention scores for each element of the concatenated image map 507 by point-producing the query vectors 508 with the key vectors 509 of all other elements. The attention score indicates how much attention the current element should pay to each of the other elements in the sequence.

[0078] In at least one embodiment, each attention score is passed through a softmax function to generate attention weights. Attention weights reflect the importance of each item relative to the current item. Items with higher weights are considered more relevant.

[0079] In at least one embodiment, inferencing using one or more neural networks includes splitting self-attention feature maps 908. In at least one embodiment, the output of the self-attention layer 511 is processed to generate one or more processed feature maps (e.g., 512, 513). In at least one embodiment, the processing includes splitting the output of the self-attention layers 508 into a number of processed feature maps (512, 513) equal to the number of input images (502, 503). In at least one embodiment, for example, if two input images (e.g., 502, 503) are input to the self-attention layer 405, the output of the self-attention layer 405 is split into two processed feature maps (e.g., 512, 513).

[0080] In at least one embodiment, inferencing using one or more neural networks includes processing the split feature maps by one or more cross-attention layers 910. In at least one embodiment, the processed feature maps 512, 513 are input to the cross-attention layer 406. Additional inputs to the cross-attention layer 406 include one or more text prompts 514, 515 (e.g., text prompts 407). Although only two text prompts 514, 515 are depicted, any finite number of text prompts may be input to the cross-attention layer 406.

[0081] In at least one embodiment, layer 406 generates query vectors 516, key vectors 517, and value vectors 518 for each element of the processed feature maps 512, 513 input to cross-attention layer 406. Query vectors 516 capture the relationship between the elements. Key vectors 517 compare the current element to all other elements in the sequence. Value vectors 518 contain information about the current element.

[0082] In at least one embodiment, the cross-attention layer 406 references one or more of the query vectors 516, key vectors 517, and value vectors 518 for each element of the processed feature maps 512, 514 and connects one or more of the vectors 516-518 to input text prompts 514, 515.

[0083] In at least one embodiment, text prompts 514, 515 (e.g., prompts 303a-303n) include a word, phrase, or sentence describing a background. For example, in at least one embodiment, text prompts 514, 515 include "in the snow, in the forest," describing a snowy background and a forested background.

[0084] In at least one embodiment, inferencing using one or more neural networks includes generating one or more output images 912. In at least one embodiment, cross-attention layer 406 outputs images 519, 520. Images 519, 520 are images that include inputs 502, 503 as a foreground image and background images described by prompts 514, 515. In effect, neural network 501 combines inputs 502, 503 with the backgrounds described in prompts 514, 515, thus generating personalized images 519, 520.

[0085] Fig. 10 shows an example including a processor and modules, according to at least one embodiment. Fig. 10 shows an example 1000 including a processor 1002 and modules, according to at least one embodiment. In at least one embodiment, a processor 1002 includes one or more circuits to use one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating content other than the one or more objects in at least one of the two or more different images. In at least one embodiment.

[0086] In at least one embodiment, the processor 1002 comprises one or more processors as described in connection with Fig. 1-9. In at least one embodiment, processor 1002 is any suitable processing unit and / or combination of processing units, such as one or more CPUs, GPUs, GPGPUs, PPUs, and / or variants thereof. In at least one embodiment, processor 1002 includes or has access to convolution model 1003, concatenation module 1004, self-attention module 1005, and cross-attention module 1006 distributed among multiple processors communicating via a bus, a network, by writing to shared memory, and / or any suitable communication method, such as those described herein.

[0087] In at least one embodiment, a module, as used in an implementation described herein, unless the context otherwise indicates or expressly states otherwise, refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. In at least one embodiment, software may be embodied as a software package, code, and / or instruction set or instructions, and "hardware," such as that used by a processor in an implementation described herein, may include, for example, individually or in any combination, hardwired circuitry, programmable circuitry, state machine circuitry, fixed function circuitry, execution units, and / or firmware storing instructions executed by programmable circuitry.In at least one embodiment, modules may be implemented collectively or individually as circuits that are part of a larger system, such as an integrated circuit (IC), a system-on-chip (SoC), and so forth. In at least one embodiment, a module performs one or more processes in conjunction with any suitable processing unit and / or combination of processing units, such as one or more CPUs, GPUs, GPGPUs, PPUs, and / or variations thereof.

[0088] Fig. 10 shows an example comprising a processor and modules, according to at least one embodiment. In at least one embodiment, the processor 1002 uses the convolutional model 1003, the concatenation module 1004, the self-attention module 1005, and the cross-attention module 1006 to generate one or more output images. In at least one embodiment, a processor uses one or all of the convolutional model 1003, the concatenation module 1004, the self-attention module 1005, and the cross-attention module 1006 to generate one or more output images, as used in conjunction with Fig. 1-9. In at least one embodiment, the processor 1002 uses the convolution model 1003, the concatenation module 1004, the self-attention module 1005, and the cross-attention module 1006 to cause software programs executing using software program-based computing resources to generate images based on the receipt of text prompts in conjunction with Fig. to generate 1-9.

[0089] In at least one embodiment, processor 1002 uses convolutional model 1003 to generate one or more feature maps from one or more inputs (e.g., inputs 402). Convolutional module 1003 includes one or more layers, neural networks, or other machine learning elements. Convolutional module 1003 receives one or more inputs (e.g., input 502, input 503). The inputs to convolutional module 1003 can be any finite number of inputs. The inputs to convolutional module 1003 can be text, images, or a combination of text and images. For each input, the one or more neural networks (e.g., neural network 504, convolutional layers 403) of convolutional module 1003 generate one or more feature maps (e.g., feature maps 505, 506).

[0090] In at least one embodiment, processor 1002 uses concatenation module 1004 to concatenate one or more feature maps. Concatenation module 1004 includes one or more processing layers, neural networks, convolutional layers, or other machine learning elements. Concatenation module 1004 concatenates one or more feature maps and image maps together through one or more layers (e.g., concatenation layers 404). In at least one embodiment, concatenating the one or more feature maps creates a single feature map (e.g., feature map 507).

[0091] In at least one embodiment, processor 1002 uses self-attention module 1005 to generate one or more feature vectors. Self-attention module 1005 includes one or more self-attention layers or self-attention neural networks. In at least one embodiment, feature map 507 is processed by one or more self-attention layers (e.g., self-attention layers 405). Self-attention layers 405 (e.g., scaled dot production attention) weigh the importance of different elements of concatenated image map 507 and enable self-attention layers 405 to capture relationships and dependencies between the elements of concatenated image map 507. Each element of concatenated image map 507 is associated with a feature vector.For each element in the concatenated image map 507, the self-attention layer 405 generates query vectors 508, key vectors 509, and value vectors 510. The query vector 508 captures the relationship between the elements of the concatenated image map 507. Key vectors 509 compare the current element with all other elements in the sequence. Value vectors 510 contain information about the current element.

[0092] In at least one embodiment, the self-attention module 1005 generates attention scores for each element of the concatenated image map 507 by point-producing the query vectors 508 with the key vectors 509 of all other elements. The attention score indicates how much attention the current element should pay to each of the other elements in the sequence. In at least one embodiment, each attention score is passed through a softmax function to generate an attention weight. The attention weight reflects the importance of each element relative to the current element. Elements with higher weights are considered more relevant. In at least one embodiment, the self-attention module outputs one or more feature maps.

[0093] In at least one embodiment, processor 1002 uses cross-attention module 1006 to generate one or more images containing an image object (e.g., one or more items from item set 203) with one or more different backgrounds. Cross-attention module 1006 includes one or more cross-attention layers or cross-attention neural networks.

[0094] In at least one embodiment, an output of the self-attention module 1005 is processed by the cross-attention module 1006 to generate one or more processed feature maps (e.g., 512, 513). In at least one embodiment, the processing includes dividing the output of the self-attention layers 508 into a number of processed feature maps (512, 513) equal to the number of input images (502, 503). In at least one embodiment, for example, if two input images (e.g., 502, 503) are input to the self-attention layer 405, the output of the self-attention layer 405 is divided into two processed feature maps (e.g., 512, 513).

[0095] In at least one embodiment, the processed feature maps (e.g., processed feature maps 512, 513) are input to the cross-attention module 1006. Additionally, inputs to the cross-attention module 1006 include one or more text prompts (e.g., text prompts 407, 514, 515).

[0096] In at least one embodiment, for each element of the processed feature maps 512, 513 input to the cross-attention module 1006, query vectors 516, key vectors 517, and value vectors 518 are generated by one or more cross-attention layers (e.g., cross-attention layers 406). Query vectors 516 capture the relationship between the elements. Key vectors 517 compare the current element to all other elements in the sequence. Value vectors 518 contain information about the current element.

[0097] In at least one embodiment, the cross-attention layer 406 references one or more of the query vectors 516, key vectors 517, and value vectors 518 for each element of the processed feature maps 512, 514 and connects one or more of the vectors 516-518 to input text prompts 514, 515.

[0098] In at least one embodiment, text prompts 514, 515 (e.g., prompts 303a-303n) include a word, phrase, or sentence describing a background. For example, in at least one embodiment, text prompts 514, 515 include "in the snow, in the forest," describing a snowy background and a forested background.

[0099] In at least one embodiment, the cross-attention layer (e.g., cross-attention layer 406) of the cross-attention module 1006 outputs one or more images (e.g., images 408, 519, 520, 603-606). The output images include the subject of the inputs (e.g., inputs 502, 503) as the foreground image and background images described by prompts (e.g., prompts 514, 515). Ultimately, the processor 1002 combines the subject of the inputs with the backgrounds described in the prompts (e.g., prompts 514, 515) to generate personalized images (e.g., images 408, 519, 520, 603-606).

[0100] In at least one embodiment, parts, methods and / or a system associated with Fig. 10 as described in the Fig. 1-9 not shown exclusively.

[0101] Fig. 11 is a block diagram illustrating a driver and / or runtime including one or more libraries to provide one or more application programming interfaces (APIs) according to at least one embodiment. In at least one embodiment, a software program 1102 is a software module. In at least one embodiment, the software program 1101 includes neural networks of Fig. 1-10. In at least one embodiment, a software program 1102 comprises one or more software modules. In at least one embodiment, one or more software modules are as in Fig. 10 is not exhaustively described. In at least one embodiment, one or more APIs 1110 are sets of software instructions that, when executed, cause one or more processors to perform one or more computational operations. In at least one embodiment, one or more APIs 1110 are distributed or otherwise provided as part of one or more libraries 1106, runtimes 1104, drivers 1104, and / or other grouping of software and / or executable code, further described herein. In at least one embodiment, one or more APIs 1110 perform one or more computational operations in response to being invoked by software programs 1102.In at least one embodiment, a software program 1102 is a collection of software code, commands, instructions, or other text sequences to instruct a computing device to perform one or more computational operations and / or to invoke one or more other instruction sets, such as APIs 1110 or API functions 1112, for execution. In at least one embodiment, the functionality provided by one or more APIs 1110 includes software functions 1112, for example, those that can be used to accelerate one or more portions of software programs 1102 using one or more parallel processing units (PPUs), such as graphics processing units (GPUs). In at least one embodiment, a software program is a compiler, further not exclusively in . Fig. 1-10 shown.

[0102] In at least one embodiment, APIs 1110 are hardware interfaces to one or more circuits to perform one or more computational operations. In at least one embodiment, one or more software APIs 1110 described herein are implemented as one or more circuits to perform one or more of the following in connection with Fig. 1-9. In at least one embodiment, one or more software programs 1102 include instructions that, when executed, cause one or more hardware devices and / or circuits to perform one or more techniques described further below in connection with Fig. 1-10 are described.

[0103] In at least one embodiment, software programs 1102, such as user-implemented software programs, use one or more application programming interfaces (APIs) 1110 to perform various computational operations, such as memory allocation, matrix multiplication, arithmetic operations, or any computational operation performed by parallel processing units (PPUs), such as graphics processing units (GPUs), as further described herein. In at least one embodiment, one or more APIs 1110 provide a set of callable functions 1112, referred to herein as APIs, API functions, and / or functions, that individually perform one or more computational operations, such as computational operations related to parallel computing.For example, in one embodiment, one or more APIs 1110 provide functions 1112 to cause an image generator 1116 to generate images that include multiple images of the same object against different backgrounds, where the functions described in . Fig. 1-10 disclosed systems, methods and other components.

[0104] In at least one embodiment, one or more software programs 1102 interact or communicate with one or more APIs 1110 to perform one or more computational operations using one or more PPUs, such as GPUs. In at least one embodiment, one or more computational operations using one or more PPUs include at least one or more groups of computational operations that are accelerated by being executed at least in part by the one or more PPUs. In at least one embodiment, one or more software programs 1102 interact with one or more APIs 1110 to enable parallel computing using a remote or local interface.

[0105] In at least one embodiment, an interface consists of software instructions that, when executed, provide access to one or more functions 1112 provided by one or more APIs 1110. In at least one embodiment, a software program 1102 uses a local interface when a software developer compiles one or more software programs 1102 in conjunction with one or more libraries 1106 that include or otherwise provide access to one or more APIs 1110. In at least one embodiment, one or more software programs 1102 are statically compiled in conjunction with precompiled libraries 1106 or uncompiled source code that includes instructions for executing one or more APIs 1110.In at least one embodiment, one or more software programs 1102 are dynamically compiled, and the one or more software programs use a linker to link to one or more precompiled libraries 1106 that include one or more APIs 1110.

[0106] In at least one embodiment, a software program 1102 uses a remote interface when a software developer executes a software program that uses or otherwise communicates with a library 1106 comprising one or more APIs 1110 over a network or other remote communication medium. In at least one embodiment, one or more libraries 1106 comprising one or more APIs 1110 are executed by a remote computing service, such as a computing resource service provider. In another embodiment, one or more libraries 1106 comprising one or more APIs 1110 are executed by any other computer host that provides the one or more APIs 1110 to one or more software programs 1102.

[0107] In at least one embodiment, a processor executing or using one or more software programs 1102 invokes, uses, executes, or otherwise implements one or more APIs 1110 to allocate and otherwise manage memory used by the software programs 1102. In at least one embodiment, one or more software programs 1102 use one or more APIs 1110 to allocate and otherwise manage memory used by one or more portions of the software programs 1102 to be accelerated using one or more PPUs, such as GPUs or another accelerator or processor further described herein.These software programs 1102 may be executed by one or more processors based, at least in part, on the latency of interconnects coupled to the one or more processors, using functions 1112 provided, in one embodiment, by one or more APIs 1110.

[0108] In at least one embodiment, an API 1110 is an API for facilitating parallel computing. In at least one embodiment, an API 1110 is any other API further described herein. In at least one embodiment, an API 1110 is provided by a driver and / or runtime 1104. In at least one embodiment, an API 1110 is provided by a CUDA user-mode driver. In at least one embodiment, an API 1110 is provided by a CUDA runtime. In at least one embodiment, a driver 1104 consists of data values ​​and software instructions that, when executed, perform or otherwise facilitate the operation of one or more functions 1112 of an API 1110 during the loading and execution of one or more portions of a software program 1102.In at least one embodiment, a runtime 1104 consists of data values ​​and software instructions that, when executed, perform or otherwise facilitate the operation of one or more functions 1112 of an API 1110 during execution of a software program 1102. In at least one embodiment, one or more software programs 1102 utilize one or more APIs 1110 implemented or otherwise provided by a driver and / or runtime 1104 to perform combined arithmetic operations by the one or more software programs 1102 during execution by one or more PPUs, such as GPUs.

[0109] In at least one embodiment, one or more software programs 1102 use one or more APIs 1110 provided by a driver and / or runtime 1104 to perform combined arithmetic operations of one or more PPUs, such as GPUs. In at least one embodiment, one or more APIs 1110 provide combined arithmetic operations via a driver and / or runtime 1104, as described above. In at least one embodiment, one or more software programs 1102 use one or more APIs 1110 provided by a driver and / or runtime 1104 to allocate or otherwise reserve one or more memory blocks 1114 to one or more PPUs, such as GPUs.In at least one embodiment, one or more software programs 1102 use one or more APIs 1110 provided by a driver and / or runtime 1104 to allocate or otherwise reserve memory blocks. In at least one embodiment, one or more APIs 1110 perform combined arithmetic operations, as described below in connection with FIGS. Fig. 1-10 described.

[0110] To improve the usability of software programs 1102 and / or the optimization of one or more portions of the software programs 1102 to be accelerated by one or more PPUs, such as GPUs, in one embodiment, one or more APIs 1110 provide one or more API functions 1112 to implement a scheduling system usable or utilized by one or more computing devices, as described above and further below in connection with Fig. 1-10. In at least one embodiment, an example block diagram 1100 depicts a processor including one or more circuits executing one or more software programs to combine two or more application programming interfaces (APIs) into a single API. In at least one embodiment, an example block diagram 1100 depicts a system including one or more processors executing one or more software programs to combine two or more application programming interfaces (APIs) into a single API. In at least one embodiment, a processor uses an API to cause a scheduler to select a thread selection mechanism and / or perform other operations described herein.In at least one embodiment, an example block diagram 1100 shows an API to invoke one or more modules (e.g., modules 1003-1006) to create one or more objects within two or more different images based at least in part on one or more indications from one or more users specifying different content of at least one of the two or more different images as the one or more objects.

[0111] In at least one embodiment, a processor uses an example API to schedule one or more instructions to be executed by one or more processors based at least in part on the latency of one or more interconnects coupled to those one or more processors. In at least one embodiment, portions, methods, and / or a system associated with Fig. 5 are described in one of the Fig. 1-10 non-exclusively shown further.

[0112] As one of ordinary skill in the art will appreciate in light of this disclosure, certain embodiments may be capable of achieving certain advantages, including some or all of the following: improving the field of computing and job scheduling systems for assigning jobs to clusters of nodes. Therefore, according to the embodiments described above, one or more processors use one or more neural networks to identify the subject of one or more inputs, where the inputs may be text, images, or a combination of text and images, and generate one or more output images including the subject of the input as a foreground image and one or more background elements described by one or more prompts. LOGIC

[0113] Fig. 12A shows logic 1215, which, as described elsewhere herein, may be used in one or more devices to perform operations such as those discussed herein in accordance with at least one embodiment. In at least one embodiment, logic 1215 is used to perform inferencing and / or training operations associated with one or more embodiments. In at least one embodiment, logic 1215 is inference and / or training logic. Details of logic 1215 are described below in connection with Fig. 12A and / or 12B. In at least one embodiment, logic refers to any combination of software logic, hardware logic, and / or firmware logic to provide functionality or operations described herein, where logic may be embodied collectively or individually as circuitry that forms part of a larger system, such as an integrated circuit (IC), system-on-chip (SoC), or one or more processors (e.g., CPU, GPU).

[0114] In at least one embodiment, logic 1215 may include, without limitation, a code and / or data memory 1201 to store feedforward and / or output weights and / or input / output data and / or other parameters to configure neurons or layers of a neural network that is trained and / or used for inferencing in aspects of one or more embodiments. In at least one embodiment, logic 1215 may include or be coupled to a code and / or data memory 1201 to store graph code or other software that controls the timing and / or order in which information about weights and / or other parameters is loaded to configure logic, including integer and / or floating-point units (collectively, arithmetic logic units (ALUs)).In at least one embodiment, code, such as graph code, loads weights or other parameter information into processor ALUs based on a neural network architecture to which such code corresponds. In at least one embodiment, code and / or data storage 1201 stores weight parameters and / or input / output data of each layer of a neural network trained or used in connection with one or more embodiments during forward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of code and / or data storage 1201 may include other on-chip or off-chip data stores, including a processor's L1, L2, or L3 cache or system memory.

[0115] In at least one embodiment, any portion of the code and / or data storage 1201 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or code and / or data storage 1201 may be cache memory, dynamic random addressable memory ("DRAM"), static random addressable memory ("SRAM"), non-volatile memory (e.g., flash memory), or other memory.In at least one embodiment, the choice of whether the code and / or code and / or data memory 1201 is, for example, internal or external to a processor or comprises DRAM, SRAM, flash, or another type of memory may depend on the available on-chip memory versus off-chip memory, the latency requirements of the training and / or inference functions performed, the batch size of the data used in the inference and / or training of a neural network, or a combination of these factors.

[0116] In at least one embodiment, logic 1215 may include, without limitation, a code and / or data storage 1205 to store backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inferencing in aspects of one or more embodiments. In at least one embodiment, code and / or data storage 1205 stores weight parameters and / or input / output data of each layer of a neural network trained or used in connection with one or more embodiments during backpropagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, logic 1215 may include or be coupled to code and / or data memory 1205 for storing graph code or other software that controls the timing and / or order in which information about weights and / or other parameters is loaded to configure logic, including integer and / or floating point units (collectively referred to as arithmetic logic units (ALUs)).

[0117] In at least one embodiment, code, such as graph code, causes information about weights or other parameters to be loaded into processor ALUs based on a neural network architecture to which that code corresponds. In at least one embodiment, any portion of the code and / or data storage 1205 may include other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of the code and / or data storage 1205 may be internal or external to one or more processors or other hardware logic devices or circuitry. In at least one embodiment, the code and / or data storage 1205 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory.In at least one embodiment, the choice of whether the code and / or data memory 1205 is, for example, internal or external to a processor, or comprises DRAM, SRAM, flash memory, or another type of memory, may depend on the available on-chip memory versus off-chip memory, the latency requirements of the training and / or inference functions performed, the batch size of the data used in inference and / or training of a neural network, or a combination of these factors.

[0118] In at least one embodiment, code and / or data memory 1201 and code and / or data memory 1205 may be separate memory structures. In at least one embodiment, code and / or data memory 1201 and code and / or data memory 1205 may be a combined memory structure. In at least one embodiment, code and / or data memory 1201 and code and / or data memory 1205 may be partially combined and partially separate. In at least one embodiment, each portion of code and / or data memory 1201 and code and / or data memory 1205 may include other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.

[0119] In at least one embodiment, logic 1215 may include, without limitation, one or more arithmetic logic unit(s) ("ALU(s)") 1210, including integer and / or floating point units, to perform logical and / or mathematical operations based at least in part on or specified by training and / or inference code (e.g., graph code), the result of which may produce activations stored in activation memory 1220 (e.g., output values ​​of layers or neurons within a neural network) that are functions of input / output and / or weighting parameter data stored in code and / or data memory 1201 and / or code and / or data memory 1205.In at least one embodiment, activations stored in an activation memory 1220 are generated according to linear algebraic and / or matrix-based mathematics performed by ALU(s) 1210 in response to execution instructions or other code, using weight values ​​stored in code and / or data memory 1205 and / or data memory 1201 as operands along with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, some or all of which may be stored in code and / or data memory 1205 or code and / or data memory 1201 or other on-chip or off-chip memory.

[0120] In at least one embodiment, ALU(s) 1210 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment, ALU(s) 1210 may be external to a processor or other hardware logic device or circuit that uses them (e.g., a co-processor). In at least one embodiment, ALU(s) 1210 may be included in the execution units of a processor or otherwise in a bank of ALUs accessible by the execution units of a processor, either within the same processor or distributed among different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.).In at least one embodiment, code and / or data memory 1201, code and / or data memory 1205, and enable memory 1220 may share a processor or other hardware logic device or circuitry, while in another embodiment, they may be located in different processors or other hardware logic devices or circuitry, or in a combination of the same and different processors or other hardware logic devices or circuitry. In at least one embodiment, each portion of enable memory 1220 may include other on-chip or off-chip data stores, including a processor's L1, L2, or L3 cache or system memory.In addition, the inference and / or training code may be stored along with other code accessible by a processor or other hardware logic or circuitry and retrieved and / or processed using a processor's fetch, decode, scheduling, execution, elimination, and / or other logic circuitry.

[0121] In at least one embodiment, activation memory 1220 may be a cache memory, a DRAM, an SRAM, a non-volatile memory (e.g., flash memory), or other memory. In at least one embodiment, activation memory 1220 may be located entirely or partially within or external to one or more processors or other logic circuitry. In at least one embodiment, the choice of whether activation memory 1220 is located, for example, within or external to a processor, or comprises DRAM, SRAM, flash memory, or another type of memory may depend on the available on-chip versus off-chip memory, the latency requirements of the training and / or inference functions performed, the batch size of the data used in inference and / or training of a neural network, or a combination of these factors.

[0122] In at least one embodiment, the Fig. 12A may be used in conjunction with an application-specific integrated circuit (“ASIC”), such as a TensorFlow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® processor (e.g., “Lake Crest”) from Intel Corp. In at least one embodiment, the logic 1215 shown in Fig. 12A may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware such as field programmable gate arrays (“FPGAs”).

[0123] Fig. 12B shows logic 1215 according to at least one embodiment. In at least one embodiment, logic 1215 is inference and / or training logic. In at least one embodiment, logic 1215 may include, without limitation, hardware logic in which computational resources associated with weight values ​​or other information corresponding to one or more layers of neurons within a neural network are dedicated or otherwise exclusively used. In at least one embodiment, the logic shown in Fig. 12B may be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's TensorFlow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or a Nervana® processor (e.g., "Lake Crest") from Intel Corp. In at least one embodiment, the logic 1215 shown in Fig. 12B may be used in conjunction with CPU (Central Processing Unit) hardware, GPU (Graphics Processing Unit) hardware, or other hardware, such as FPGAs (Field Programmable Gate Arrays). In at least one embodiment, logic 1215 includes, without limitation, code and / or data memory 1201 and code and / or data memory 1205, which may be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, pulse values, and / or other parameter or hyperparameter information. In at least one embodiment shown in Fig. 12B, each of the code and / or data memories 1201 and the code and / or data memories 1205 is coupled to a dedicated computing resource, such as computer hardware 1202 and computer hardware 1206, respectively. In at least one embodiment, each of the computing hardware 1202 and the computer hardware 1206 includes one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in the code and / or data memory 1201 and the code and / or data memory 1205, respectively, the result of which is stored in the activation memory 1220.

[0124] In at least one embodiment, code and / or data storage 1201 and 1205 and the corresponding computer hardware 1202 and 1206 each correspond to different layers of a neural network, such that the activation resulting from one memory / compute pair 1201 / 1202 of code and / or data storage 1201 and computer hardware 1202 is provided as input to a next memory / compute pair 1205 / 1206 of code and / or data storage 1205 and computer hardware 1206 to reflect a conceptual organization of a neural network. In at least one embodiment, each of the memory / compute pairs 1201 / 1202 and 1205 / 1206 may correspond to more than one layer of the neural network. In at least one embodiment, additional memory / compute pairs (not shown) may be included in logic 1215 subsequent to or in parallel with memory / compute pairs 1201 / 1202 and 1205 / 1206.

[0125] In at least one embodiment, components of the Fig. 12A-12B or with components, processes and / or a combination thereof from the Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig. 12A-12B one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig. 12A-12B one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts. TRAINING AND DEPLOYING A NEURAL NETWORK

[0126] Fig. 13 illustrates the training and deployment of a deep neural network according to at least one embodiment. In at least one embodiment, an untrained neural network 1306 is trained using a training dataset 1302. In at least one embodiment, the training framework 1304 is a PyTorch framework, while in other embodiments, the training framework 1304 is a TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training framework. In at least one embodiment, the training framework 1304 trains an untrained neural network 1306 and facilitates its training with the processing resources described herein to produce a trained neural network 1308. In at least one embodiment, the weights may be selected randomly or by pre-training using a deep belief network.In at least one embodiment, the training may be performed in either a supervised, semi-supervised, or unsupervised manner.

[0127] In at least one embodiment, the untrained neural network 1306 is trained using supervised learning, where the training data set 1302 includes an input paired with a desired output for an input, or where the training data set 1302 has an input with a known output and an output of the neural network 1306 is manually evaluated. In at least one embodiment, the untrained neural network 1306 is trained in a supervised manner and processes inputs from the training data set 1302 and compares the resulting outputs to a set of expected or desired outputs. In at least one embodiment, the errors are then backtracked through the untrained neural network 1306. In at least one embodiment, the training framework 1304 adjusts the weights that control the untrained neural network 1306.In at least one embodiment, the training framework 1304 includes tools for monitoring how well the untrained neural network 1306 converges to a model, such as the trained neural network 1308, that is capable of generating correct answers, such as in the output 1314, based on input data, such as a new data set 1312. In at least one embodiment, the training framework 1304 repeatedly trains the untrained neural network 1306 while adjusting the weights to refine an output of the untrained neural network 1306 using a loss function and an adaptation algorithm, such as stochastic gradient descent. In at least one embodiment, the training framework 1304 trains the untrained neural network 1306 until the untrained neural network 1306 achieves a desired accuracy.In at least one embodiment, the trained neural network 1308 may then be used to implement any number of machine learning operations.

[0128] In at least one embodiment, the untrained neural network 1306 is trained using unsupervised learning, where the untrained neural network 1306 attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised learning training dataset 1302 comprises input data without associated output data, or "ground truth data." In at least one embodiment, the untrained neural network 1306 can learn groupings within the training dataset 1302 and determine how individual inputs relate to the untrained dataset 1302. In at least one embodiment, unsupervised training can be used to generate a self-organizing map in a trained neural network 1308 capable of performing operations useful for reducing the dimensionality of the new dataset 1312.In at least one embodiment, unsupervised training may also be used for anomaly detection, enabling the identification of data points in the new data set 1312 that deviate from normal patterns of the new data set 1312.

[0129] In at least one embodiment, semi-supervised learning may be used, i.e., a technique in which the training data set 1302 includes a mixture of labeled and unlabeled data. In at least one embodiment, the training framework 1304 may be used to perform incremental learning, for example, through transfer learning techniques. In at least one embodiment, incremental learning allows the trained neural network 1308 to adapt to a new data set 1312 without forgetting the knowledge instilled in the trained neural network 1308 during initial training.

[0130] In at least one embodiment, the training framework 1304 is a framework processed in conjunction with a software development toolkit, such as an OpenVINO (Open Visual Inference and Neural Network Optimization) toolkit. In at least one embodiment, an OpenVINO toolkit is a toolkit such as that developed by Intel Corporation of Santa Clara, CA. In at least one embodiment, OpenVINO includes logic 1215 or uses logic 1215 to perform the operations described herein. In at least one embodiment, an SoC, integrated circuit, or processor uses OpenVINO to perform the operations described herein.

[0131] In at least one embodiment, OpenVINO is a toolkit for facilitating the development of applications, in particular neural network applications, for various tasks and operations, such as human vision emulation, speech recognition, natural language processing, recommender systems, and / or variations thereof. In at least one embodiment, OpenVINO supports neural networks such as convolutional neural networks (CNNs), recurrent and / or attention-based neural networks, and / or various other neural network models. In at least one embodiment, OpenVINO supports various software libraries such as OpenCV, OpenCL, and / or variants thereof.

[0132] In at least one embodiment, OpenVINO supports neural network models for various tasks and operations, such as classification, segmentation, object detection, face recognition, speech recognition, pose estimation (e.g., humans and / or objects), monocular depth estimation, image painting, style transfer, action recognition, colorization, and / or variations thereof.

[0133] In at least one embodiment, OpenVINO includes one or more software tools and / or modules for model optimization, also referred to as model optimizers. In at least one embodiment, a model optimizer is a command-line tool that facilitates the transitions between training and deploying neural network models. In at least one embodiment, a model optimizer optimizes neural network models for execution on different devices and / or processing units, such as a GPU, CPU, PPU, GPGPU, and / or variations thereof. In at least one embodiment, a model optimizer generates an internal representation of a model and optimizes the model to generate an intermediate representation. In at least one embodiment, a model optimizer reduces a number of layers of a model.In at least one embodiment, a model optimizer removes the layers of a model used for training. In at least one embodiment, a model optimizer performs various operations of a neural network, such as changing the inputs to a model (e.g., changing the size of the inputs to a model), changing the size of the inputs of a model (e.g., changing the batch size of a model), changing a model structure (e.g., changing layers of a model), normalization, standardization, quantization (e.g., converting weights of a model from a first representation, such as floating point, to a second representation, such as integer), and / or variations thereof.

[0134] In at least one embodiment, OpenVINO includes one or more software libraries for inference, also referred to as an inference engine. In at least one embodiment, the inference engine is a C++ library or other suitable library in a programming language. In at least one embodiment, an inference engine is used to derive input data. In at least one embodiment, an inference engine implements various classes for deriving input data and producing one or more results. In at least one embodiment, an inference engine implements one or more API functions to process an intermediate representation, specify input and / or output formats, and / or execute a model on one or more devices.

[0135] In at least one embodiment, OpenVINO provides various capabilities for heterogeneously executing one or more neural network models. In at least one embodiment, heterogeneous execution or heterogeneous computing refers to one or more computing processes and / or systems using one or more types of processors and / or cores. In at least one embodiment, OpenVINO provides various software functions for executing a program on one or more devices. In at least one embodiment, OpenVINO provides various software functions for executing a program and / or portions of a program on different devices. In at least one embodiment, OpenVINO provides various software functions, for example, for executing a first portion of the code on a CPU and a second portion of the code on a GPU and / or FPGA.In at least one embodiment, OpenVINO provides various software functions to execute one or more layers of a neural network on one or more devices (e.g., a first set of layers on a first device, such as a GPU, and a second set of layers on a second device, such as a CPU).

[0136] In at least one embodiment, OpenVINO includes various functionalities similar to those associated with a CUDA programming model, for example, various neural network model operations associated with frameworks such as TensorFlow, PyTorch, and / or variants thereof. In at least one embodiment, one or more CUDA programming model operations are performed with OpenVINO. In at least one embodiment, various systems, methods, and / or techniques described herein are implemented using OpenVINO. DATA CENTER

[0137] Fig. 14 shows an example of a data center 1400 in which at least one embodiment may be used. In at least one embodiment, the data center 1400 includes a data center infrastructure layer 1410, a framework layer 1420, a software layer 1430, and an application layer 1440.

[0138] In at least one embodiment, as in Fig. 14, the data center infrastructure layer 1410 may include a resource orchestrator 1412, clustered compute resources 1414, and node compute resources (“Node CRs”) 1416(1)-1416(N), where “N” represents a positive integer (which may be a different integer “N” than used in other figures). In at least one embodiment, the node CRs 1416(1)-1416(N) may include any number of central processing units (“CPUs”) or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc.), storage devices 1418(1)-1418(N) (e.g., dynamic read-only memory, solid-state storage, or hard disk drives), network input / output devices (“NW I / O”), network switches, virtual machines (“VMs”), power modules and cooling modules, etc. In at least one embodiment, one or more node CRs among the node CRs 1416(1)-1416(N) may be a server that has one or more of the computing resources mentioned above.

[0139] In at least one embodiment, the grouped computing resources 1414 may include separate groupings of node CRs housed in one or more racks (not shown) or multiple racks housed in data centers in different geographic locations (also not shown). In at least one embodiment, separate groupings of node CRs within the grouped computing resources 1414 may include grouped computing, networking, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs, including CPUs or processors, may be grouped in one or more racks to provide computing resources to support one or more workloads.In at least one embodiment, one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0140] In at least one embodiment, resource orchestrator 1412 may configure or otherwise control one or more node CRs 1416(1)-1416(N) and / or clustered computing resources 1414. In at least one embodiment, resource orchestrator 1412 may include a software design infrastructure ("SDI") management entity for data center 1400. In at least one embodiment, resource orchestrator 1412 may include hardware, software, or a combination thereof.

[0141] In at least one embodiment, as in Fig. 14, the framework layer 1420 includes a job scheduler 1422, a configuration manager 1424, a resource manager 1426, and a distributed file system 1428. In at least one embodiment, the framework layer 1420 may include a framework for supporting the software 1432 of the software layer 1430 and / or one or more applications 1442 of the application layer 1440. In at least one embodiment, the software 1432 or the application(s) 1442 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 1420 may be, but is not limited to, some type of free and open-source software web application framework such as Apache Spark™ (hereinafter "Spark") that may utilize a distributed file system 1428 for processing large amounts of data (e.g., "Big Data").In at least one embodiment, the job scheduler 1422 may include a Spark driver to facilitate the scheduling of workloads supported by different layers of the data center 1400. In at least one embodiment, the configuration manager 1424 may be capable of configuring different layers, such as the software layer 1430 and the framework layer 1420, which include Spark and the distributed file system 1428 to support processing large amounts of data. In at least one embodiment, the resource manager 1426 may be capable of managing clustered or grouped computing resources allocated to support the distributed file system 1428 and the job scheduler 1422. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 1414 on the data center infrastructure layer 1410.In at least one embodiment, the resource manager 1426 may coordinate with the resource orchestrator 1412 to manage these allocated or assigned computing resources.

[0142] In at least one embodiment, the software 1432 included in software layer 1430 may include software used by at least portions of node CRs 1416(1)-1416(N), clustered computing resources 1414, and / or distributed file system 1428 of framework layer 1420. In at least one embodiment, one or more types of software may include, but are not limited to, Internet website search software, email virus scanning software, database software, and streaming video content software.

[0143] In at least one embodiment, the application(s) 1442 included in the application layer 1440 may include one or more types of applications used by at least portions of the node CRs 1416(1)-1416(N), the clustered computing resources 1414, and / or the distributed file system 1428 of the framework layer 1420. In at least one embodiment, one or more types of applications may include any number of a genomic application, a cognitive computing application, and a machine learning application, including, but not limited to, training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in connection with one or more embodiments.

[0144] In at least one embodiment, configuration manager 1424, resource manager 1426, and resource orchestrator 1412 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically feasible manner. In at least one embodiment, self-modifying actions may relieve a data center operator of data center 1400 from potentially making poor configuration decisions and potentially avoiding underutilized and / or malfunctioning portions of a data center.

[0145] In at least one embodiment, data center 1400 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weighting parameters according to a neural network architecture using software and computational resources described above with respect to data center 1400.In at least one embodiment, trained machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1400 by using weighting parameters calculated by one or more training techniques described herein.

[0146] In at least one embodiment, the data center may use CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, or other hardware to perform training and / or inferencing using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as a service to enable users to train or infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0147] Logic 1215 is used to perform inference and / or training operations in connection with one or more embodiments. Details of logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in data center 1400 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0148] In at least one embodiment, components of Fig. 14 or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig. 14 one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig. 14 one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts. AUTONOMOUS VEHICLE

[0149] Fig. 15A shows an example of an autonomous vehicle 1500 according to at least one embodiment. In at least one embodiment, the autonomous vehicle 1500 (alternatively referred to herein as "vehicle 1500") may be, without limitation, a passenger vehicle, such as a car, a truck, a bus, and / or another type of vehicle that accommodates one or more passengers. In at least one embodiment, the vehicle 1500 may be a semi-trailer truck used for transporting goods. In at least one embodiment, the vehicle 1500 may be an aircraft, a robotic vehicle, or another type of vehicle.

[0150] Autonomous vehicles may be described in terms of levels of automation defined by the National Highway Traffic Safety Administration ("NHTSA"), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers ("SAE") "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (e.g., Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and prior and future versions of this standard). In at least one embodiment, the vehicle 1500 may be capable of performing functions according to one or more of Levels 1 through 5 of the autonomous driving levels. For example, in at least one embodiment, the vehicle 1500 may be capable of conditionally automated (Level 3), highly automated (Level 4), and / or fully automated (Level 5), depending on the embodiment.

[0151] In at least one embodiment, vehicle 1500 may include, without limitation, components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. In at least one embodiment, vehicle 1500 may include, without limitation, a propulsion system 1550, such as an internal combustion engine, a hybrid electric power plant, a pure electric motor, and / or another type of propulsion system. In at least one embodiment, propulsion system 1550 may be connected to a drivetrain of vehicle 1500, which may include, without limitation, a transmission, to facilitate propulsion of vehicle 1500. In at least one embodiment, propulsion system 1550 may be controlled in response to receiving signals from gas pedal / accelerator device(s) 1552.

[0152] In at least one embodiment, a steering system 1554, which may include, without limitation, a steering wheel, is used to steer the vehicle 1500 (e.g., along a desired path or route) when the propulsion system 1550 is operating (e.g., when the vehicle 1500 is in motion). In at least one embodiment, the steering system 1554 may receive signals from the steering actuator(s) 1556. In at least one embodiment, a steering wheel may be optional for full automation (Level 5). In at least one embodiment, a brake sensor system 1546 may be used to apply the vehicle brakes in response to receiving signals from brake actuator(s) 1548 and / or brake sensors.

[0153] In at least one embodiment, controllers 1536, including without limitation one or more system-on-chips (“SoCs”) (in Fig. 15A not shown) and / or graphics processing units ("GPUs"), send signals (e.g., representative of commands) to one or more components and / or systems of the vehicle 1500. In at least one embodiment, the controller 1536 may, for example, send signals to actuate the vehicle brakes via the brake actuator(s) 1548, to actuate the steering system 1554 via the steering actuator(s) 1556, and to actuate the propulsion system 1550 via the gas pedal / accelerator device(s) 1552. In at least one embodiment, the controller(s) 1536 may include one or more on-vehicle (e.g., on-board) computing devices that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 1500.In at least one embodiment, the controller(s) 1536 may include a first controller for autonomous driving functions, a second controller for functional safety functions, a third controller for artificial intelligence functions (e.g., computer vision), a fourth controller for infotainment functions, a fifth controller for emergency redundancy, and / or other controllers. In at least one embodiment, a single controller may perform two or more of the above functions, two or more controllers may perform a single function, and / or any combination thereof.

[0154] In at least one embodiment, the controller(s) 1536 provide signals to control one or more components and / or systems of the vehicle 1500 in response to sensor data received from one or more sensors (e.g., sensor inputs). In at least one embodiment, sensor data may be obtained, for example and without limitation, from Global Navigation Satellite System ("GNSS") sensor(s) 1558 (e.g., Global Positioning System sensor(s)), RADAR sensor(s) 1560, ultrasonic sensor(s) 1562, LIDAR sensor(s) 1564, Inertial Measurement Unit ("IMU") sensor(s) 1566 (e.g., accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s), etc.), microphone(s) 1596, stereo camera(s) 1568, wide-angle camera(s) 1570 (e.g., fisheye cameras), infrared camera(s) 1572, surround camera(s) 1574 (e.g., 360-degree cameras), remote cameras (in Fig. 15A not shown), mid-range camera(s) (in Fig. 15A not shown), speed sensor(s) 1544 (e.g., for measuring the speed of the vehicle 1500), vibration sensor(s) 1542, steering sensor(s) 1540, brake sensor(s) (e.g., as part of the brake sensor system 1546), and / or other types of sensors.

[0155] In at least one embodiment, one or more controllers 1536 may receive inputs (e.g., in the form of input data) from an instrument cluster 1532 of the vehicle 1500 and provide outputs (e.g., in the form of output data, display data, etc.) via a human-machine interface (“HMI”) display 1534, an audible annunciator, a speaker, and / or via other components of the vehicle 1500. In at least one embodiment, the outputs may include information such as vehicle speed, RPM, time, map data (e.g., a high-resolution map (in Fig. 15A not shown)), location data (e.g., the location of the vehicle 1500, as on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by the controller(s) 1536, etc. For example, in at least one embodiment, the HMI display 1534 may display information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing traffic light, etc.) and / or information about maneuvers the vehicle has performed, is performing, or will perform (e.g., lane change now, exit 34B in two miles, etc.).

[0156] In at least one embodiment, the vehicle 1500 further includes a network interface 1524 that may utilize wireless antenna(s) 1526 and / or modem(s) to communicate over one or more networks. For example, in at least one embodiment, the network interface 1524 may be capable of communicating over Long-Term Evolution ("LTE"), Wideband Code Division Multiple Access ("WCDMA"), Universal Mobile Telecommunications System ("UMTS"), Global System for Mobile Communication ("GSM"), IMT-CDMA Multi-Carrier ("CDMA2000") networks, etc. In at least one embodiment, the wireless antenna(s) 1526 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy ("LE"), Z-Wave, ZigBee, etc., and / or Low Power Wide Area Networks ("LPWANs") such as LoRaWAN, SigFox, etc.

[0157] Logic 1215 is used to perform inference and / or training operations in connection with one or more embodiments. Details of logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in vehicle 1500 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0158] In at least one embodiment, components of Fig. 15A or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig. 15A one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating different content of at least one of the two or more different images than the one or more objects. In at least one embodiment, the components of Fig. 15A one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0159] Fig. 15B shows an example of camera positions and fields of view for the autonomous vehicle 1500 of Fig. 15A according to at least one embodiment. In at least one embodiment, the cameras and the respective fields of view represent an exemplary embodiment and are not to be considered limiting. For example, in at least one embodiment, additional and / or alternative cameras may be included and / or cameras may be arranged at different locations of the vehicle 1500.

[0160] In at least one embodiment, the camera types for cameras may include, but are not limited to, digital cameras that may be adapted for use with components and / or systems of the vehicle 1500. In at least one embodiment, the camera(s) may operate at Automotive Safety Integrity Level ("ASIL") B and / or another ASIL. In at least one embodiment, the camera types may achieve any image capture rate, such as 60 frames per second (fps), 1220 fps, 240 fps, etc., depending on the embodiment. In at least one embodiment, the cameras may use rolling shutter, global shutter, another shutter type, or a combination thereof.In at least one embodiment, the color filter array may comprise a red-clear-clear color filter array ("RCCC"), a red-clear-blue color filter array ("RCCB"), a red-blue-green clear color filter array ("RBGC"), a Foveon X3 color filter array, a Bayer sensor color filter array ("RGGB"), a monochrome sensor color filter array, and / or another type of color filter array. In at least one embodiment, clear-pixel cameras, such as cameras with an RCCC, an RCCB, and / or an RBGC color filter array, may be used to increase light sensitivity.

[0161] In at least one embodiment, one or more cameras may be used to perform advanced driver assistance system ("ADAS") functions (e.g., as part of a redundant or fail-safe design). For example, in at least one embodiment, a multi-function mono camera may be installed, including functions such as lane departure warning, traffic sign assist, and intelligent headlight control. In at least one embodiment, one or more of the cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0162] In at least one embodiment, one or more cameras may be mounted in a mounting fixture, such as a custom-designed (three-dimensional ("3D") printed) fixture, to eliminate stray light and reflections within the vehicle 1500 (e.g., reflections from the dashboard reflected in the windshield mirrors) that may impair the camera's ability to capture images. In at least one embodiment, exterior mirror assemblies may be custom 3D printed so that a camera mounting plate conforms to the shape of an exterior mirror. In at least one embodiment, camera(s) may be integrated into the exterior mirrors. In at least one embodiment, for side-facing cameras, the camera(s) may also be integrated into four pillars at each corner of the cabin.

[0163] In at least one embodiment, cameras with a field of view encompassing portions of an environment in front of the vehicle 1500 (e.g., forward-facing cameras) may be used for surround vision to help identify forward paths and obstacles, and to provide information critical to establishing an occupancy grid and / or determining preferred vehicle paths with the assistance of one or more controllers 1536 and / or control SoCs. In at least one embodiment, forward-facing cameras may be used to perform many similar ADAS functions as LIDAR, including, without limitation, emergency braking, pedestrian detection, and collision avoidance.In at least one embodiment, forward-facing cameras may also be used for ADAS features and systems, including, without limitation, lane departure warnings (“LDW”), autonomous cruise control (“ACC”), and / or other features such as traffic sign recognition.

[0164] In at least one embodiment, a plurality of cameras may be used in a forward-facing configuration, including, for example, a monocular camera platform comprising a CMOS (complementary metal oxide semiconductor) color imager. In at least one embodiment, a wide-angle camera 1570 may be used to detect objects entering view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although in Fig. 15B shows only one wide-angle camera 1570, in other embodiments, the vehicle 1500 may include any number (including zero) of wide-angle cameras. In at least one embodiment, any number of long-range cameras 1598 (e.g., a wide-angle stereo camera pair) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. In at least one embodiment, the wide-angle camera(s) 1598 may also be used for object detection and classification, as well as basic object tracking.

[0165] In at least one embodiment, any number of stereo cameras 1568 may also be in a forward-facing configuration. In at least one embodiment, one or more of the stereo cameras 1568 may include an integrated control unit comprising a scalable processing unit that may provide a field-programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. In at least one embodiment, such a unit may be used to create a 3D map of the environment of the vehicle 1500 that includes a distance estimate for all points in an image.In at least one embodiment, one or more of the stereo cameras 1568 may comprise, without limitation, compact stereo vision sensors, which may comprise, without limitation, two camera lenses (one each on the left and right) and an image processing chip that can measure the distance between the vehicle 1500 and the target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. In at least one embodiment, other types of stereo cameras (1568) may be used in addition to or as an alternative to the stereo cameras described herein.

[0166] In at least one embodiment, cameras with a field of view that includes portions of the environment on the sides of the vehicle 1500 (e.g., side cameras) may be used for the environment view and provide information used to create and update an occupancy grid and to generate side impact collision warnings. For example, in at least one embodiment, the environment camera(s) 1574 (e.g., four environment cameras, as in Fig. 15B) may be positioned on the vehicle 1500. In at least one embodiment, the surround view camera(s) 1574 may include, without limitation, any number and combination of wide-angle cameras, fisheye cameras, 360-degree cameras, and / or similar cameras. For example, in at least one embodiment, four fisheye cameras may be positioned on the front, rear, and sides of the vehicle 1500. In at least one embodiment, the vehicle 1500 may utilize three surround view camera(s) 1574 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera.

[0167] In at least one embodiment, cameras with a field of view that includes portions of the environment behind the vehicle 1500 (e.g., rearview cameras) may be used for parking assistance, surround view, rear collision warnings, and creating and updating an occupancy grid. In at least one embodiment, a plurality of cameras may be used, including at least one plurality of cameras that are also suitable as forward-facing cameras (e.g., long-range cameras 1598 and / or medium-range camera(s) 1576, stereo camera(s) 1568, infrared camera(s) 1572, etc.), as described herein.

[0168] In at least one embodiment, components of Fig. 15B or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig. 15B one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating different content of at least one of the two or more different images than the one or more objects. In at least one embodiment, the components of Fig. 15B one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0169] Fig. 15C is a block diagram illustrating an example system architecture for the autonomous vehicle 1500 of Fig. 15A according to at least one embodiment. In at least one embodiment, each of the components, features, and systems of the vehicle 1500 is Fig. 15C as connected via a bus 1502. In at least one embodiment, bus 1502 may include, without limitation, a CAN data interface (alternatively referred to herein as a "CAN bus"). In at least one embodiment, a CAN may be a network within vehicle 1500 used to support the control of various features and functions of vehicle 1500, such as brake application, acceleration, braking, steering, windshield wipers, etc. In at least one embodiment, bus 1502 may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). In at least one embodiment, bus 1502 may be read to determine steering wheel angle, vehicle speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators.In at least one embodiment, bus 1502 may be a CAN bus that is ASIL B compliant.

[0170] In at least one embodiment, FlexRay and / or Ethernet protocols may be used in addition to or alternatively to CAN. In at least one embodiment, there may be any number of buses forming bus 1502, which may include, without limitation, zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses with different protocols. In at least one embodiment, two or more buses may be used to perform different functions and / or may be used for redundancy. For example, a first bus may be used for collision avoidance functionality and a second bus may be used for actuation control.In at least one embodiment, each bus of bus 1502 may communicate with any components of vehicle 1500, and two or more buses of bus 1502 may communicate with corresponding components. In at least one embodiment, each of any number of system(s) on chip(s) ("SoC(s)") 1504 (such as SoC 1504(A) and SoC 1504(B)), each of controllers 1536, and / or each computer within the vehicle may have access to the same input data (e.g., inputs from sensors of vehicle 1500) and be connected to a common bus, such as the CAN bus.

[0171] In at least one embodiment, the vehicle 1500 may include one or more controllers 1536 as described herein with respect to Fig. 15A. In at least one embodiment, the controller(s) 1536 may be used for a variety of functions. In at least one embodiment, the controller(s) 1536 may be coupled to various other components and systems of the vehicle 1500 and used for control of the vehicle 1500, the artificial intelligence of the vehicle 1500, the infotainment for the vehicle 1500, and / or other functions.

[0172] In at least one embodiment, the vehicle 1500 may include any number of SoCs 1504. In at least one embodiment, each of the SoCs 1504 may include, without limitation, central processing units (“CPU(s)”) 1506, graphics processing units (“GPU(s)”) 1508, processor(s) 1510, cache(s) 1512, accelerators 1514, data storage 1516, and / or other components and features not shown. In at least one embodiment, SoC(s) 1504 may be used to control the vehicle 1500 in a variety of platforms and systems. For example, in at least one embodiment, SoC(s) 1504 may be combined in a system (e.g., system of the vehicle 1500) with a high-definition (“HD”) map 1522 that may receive map refreshes and / or updates via the network interface 1524 from one or more servers (in Fig. 15C not shown).

[0173] In at least one embodiment, the CPU(s) 1506 may comprise a CPU cluster or CPU complex (alternatively referred to herein as a "CCPLEX"). In at least one embodiment, the CPU(s) 1506 may comprise multiple cores and / or Level Two ("L2") caches. For example, in at least one embodiment, the CPU(s) 1506 may comprise eight cores in a coherent multiprocessor configuration. In at least one embodiment, the CPU(s) 1506 may comprise four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 megabyte (MB) L2 cache). In at least one embodiment, the CPU(s) 1506 (e.g., CCPLEX) may be configured to support concurrent cluster operations, such that any combination of clusters of CPU(s) 1506 may be active at any given time.

[0174] In at least one embodiment, one or more of the CPU(s) 1506 may implement power management features, including, without limitation, one or more of the following: individual hardware blocks may be automatically clocked when idle to conserve dynamic power; each core clock may be clocked when such core is not actively executing instructions due to the execution of Wait for Interrupt ("WF1") / Wait for Event ("WFE") instructions; each core may be independently power-driven; each core cluster may be independently clock-driven if all cores are clock-driven or power-driven; and / or each core cluster may be independently power-driven if all cores are power-driven.In at least one embodiment, the CPU(s) 1506 may further implement an advanced power state management algorithm, where allowable power states and expected wake-up times are specified, and the hardware / microcode determines which power state is most appropriate for the core, cluster, and CCPLEX. In at least one embodiment, the processor cores may support simplified power state entry sequences in software, offloading the work to the microcode.

[0175] In at least one embodiment, the GPU(s) 1508 may comprise an integrated GPU (alternatively referred to herein as an "iGPU"). In at least one embodiment, the GPU(s) 1508 may be programmable and efficient for parallel workloads. In at least one embodiment, the GPU(s) 1508 may utilize an extended Tensor instruction set. In at least one embodiment, the GPU(s) 1508 may comprise one or more streaming microprocessors, where each streaming microprocessor may comprise a level one ("L1") cache (e.g., an L1 cache having a memory capacity of at least 96 KB), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache having a memory capacity of 512 KB). In at least one embodiment, the GPU(s) 1508 may comprise at least eight streaming microprocessors.In at least one embodiment, GPU(s) 1508 may use one or more application programming interfaces (API(s)) for computation. In at least one embodiment, GPU(s) 1508 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA model).

[0176] In at least one embodiment, one or more of the GPU(s) 1508 may be power-optimized for best performance in automotive and embedded use cases. For example, in at least one embodiment, the GPU(s) 1508 may be fabricated on Fin Field Effect Transistor ("FinFET") circuits. In at least one embodiment, each streaming microprocessor may include a number of mixed-precision compute cores divided into multiple blocks. For example, 64 PF32 cores and 32 FP64 cores could be divided into four processing blocks. In at least one embodiment, each processing block could be assigned 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA mixed-precision Tensor cores for deep learning matrix arithmetic, a level zero ("L0") instruction cache, a scheduler (e.g., warp scheduler) or sequencer, a dispatch unit, and / or a 64 KB register file.In at least one embodiment, streaming microprocessors may include independent parallel integer and floating-point datapaths to enable efficient execution of workloads with a mix of computations and addressing calculations. In at least one embodiment, streaming microprocessors may include independent thread scheduling capabilities to enable finer-grained synchronization and cooperation between parallel threads. In at least one embodiment, streaming microprocessors may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0177] In at least one embodiment, one or more of the GPU(s) 1508 may include high-bandwidth memory ("HBM") and / or a 16 GB HBM2 memory subsystem to provide, in some examples, a peak memory bandwidth of approximately 900 GB / second. In at least one embodiment, in addition to or alternatively to the HBM memory, a synchronous graphics random-access memory ("SGRAM") may be used, such as a synchronous graphics double-data-rate random-access memory type 5 ("GDDR5").

[0178] In at least one embodiment, the GPU(s) 1508 may include unified memory technology. In at least one embodiment, address translation services ("ATS") support may be used to allow the GPU(s) 1508 to directly access page tables of the CPU(s) 1506. In at least one embodiment, an address translation request may be communicated to the CPU(s) 1506 when a GPU of the GPU(s) 1508 memory management unit ("MMU") encounters a fault. In response, the CPU of the CPU(s) 1506 may look up a virtual-to-physical mapping for an address in its page tables and transmit the translation back to the GPU(s) 1508, in at least one embodiment.In at least one embodiment, unified memory technology may enable a single unified virtual address space for the memory of both the CPU(s) 1506 and the GPU(s) 1508, thereby simplifying programming of the GPU(s) 1508 and porting of applications to the GPU(s) 1508.

[0179] In at least one embodiment, the GPU(s) 1508 may include any number of access counters that can track the frequency of access by the GPU(s) 1508 to the memory of other processors. In at least one embodiment, access counters can help ensure that memory pages are moved to the physical memory of a processor that accesses pages most frequently, thereby improving the efficiency of memory regions shared between processors.

[0180] In at least one embodiment, one or more of the SoC(s) 1504 may include any number of cache(s) 1512, including those described herein. For example, in at least one embodiment, the cache(s) 1512 could include a Level 3 ("L3") cache available to both the CPU(s) 1506 and the GPU(s) 1508 (e.g., connected to the CPU(s) 1506 and the GPU(s) 1508). In at least one embodiment, the cache(s) 1512 may include a write-back cache that can track the states of lines, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, an L3 cache may include 4 MB of memory or more, depending on the embodiment, although smaller cache sizes may also be used.

[0181] In at least one embodiment, one or more of the SoC(s) 1504 may include one or more accelerators 1514 (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, the SoC(s) 1504 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. In at least one embodiment, large on-chip memory (e.g., 4 MB of SRAM) may enable a hardware acceleration cluster to accelerate neural networks and other computations. In at least one embodiment, a hardware acceleration cluster may be used to supplement the GPU(s) 1508 and offload some tasks from the GPU(s) 1508 (e.g., to free up more cycles of the GPU(s) 1508 to perform other tasks).In at least one embodiment, the accelerator(s) 1514 could be used for targeted workloads (e.g., perception, convolutional neural networks ("CNNs"), recurrent neural networks ("RNNs"), etc.) that are robust enough to be suitable for acceleration. In at least one embodiment, a CNN may include a region-based or regional neural network ("RCNNs") and fast RCNNs (e.g., for object detection), or another type of CNN.

[0182] In at least one embodiment, the accelerator(s) 1514 (e.g., hardware acceleration clusters) may include one or more deep learning accelerators ("DLAs"). In at least one embodiment, the DLA(s) may include, without limitation, one or more tensor processing units ("TPUs") that may be configured to provide an additional tens of trillion operations per second for deep learning applications and inferencing. In at least one embodiment, the TPUs may be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). In at least one embodiment, the DLA(s) may be further optimized for a particular set of neural network types and floating-point operations, as well as for inferencing.In at least one embodiment, the design of DLA(s) can provide more performance per millimeter than a typical general-purpose GPU, typically far exceeding the performance of a CPU. In at least one embodiment, the TPU(s) can perform multiple functions, including a single-instance convolution function, supporting, for example, INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.In at least one embodiment, DLA(s) can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for a variety of functions, including, for example and without limitation: a CNN for object identification and recognition using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification and recognition using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety-relevant and / or security-related events.

[0183] In at least one embodiment, the DLA(s) may perform any function of the GPU(s) 1508, and by using an inference accelerator, a developer may, for example, dedicate either the DLA(s) or the GPU(s) 1508 to each function. For example, in at least one embodiment, a developer may focus the processing of CNNs and floating-point operations on the DLA(s) and leave other functions to the GPU(s) 1508 and / or accelerator(s) 1514.

[0184] In at least one embodiment, the accelerator(s) 1514 may comprise a programmable image processing accelerator ("PVA"), which may also be referred to herein as a computer vision accelerator. In at least one embodiment, the PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems ("ADAS") 1538, autonomous driving, augmented reality ("AR"), and / or virtual reality ("VR") applications. In at least one embodiment, the PVA may provide a balance between performance and flexibility. In at least one embodiment, each PVA may comprise, for example and without limitation, any number of reduced instruction set ("RISC") cores, direct memory access ("DMA") cores, and / or any number of vector processors.

[0185] In at least one embodiment, RISC cores may interact with image sensors (e.g., image sensors of cameras described herein), image signal processors, etc. In at least one embodiment, each RISC core may include any amount of memory. In at least one embodiment, the RISC cores may use any number of protocols, depending on the embodiment. In at least one embodiment, RISC cores may execute a real-time operating system ("RTOS"). In at least one embodiment, RISC cores may be implemented with one or more integrated circuits, application-specific integrated circuits ("ASICs"), and / or memory devices. In at least one embodiment, RISC cores could, for example, include an instruction cache and / or tightly coupled RAM.

[0186] In at least one embodiment, DMA may enable components of the PVA to access system memory independently of the CPU(s) 1506. In at least one embodiment, DMA may support any number of features used to optimize a PVA, including, but not limited to, supporting multi-dimensional addressing and / or circular addressing. In at least one embodiment, DMA may support up to six or more dimensions of addressing, which may include, without limitation, block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0187] In at least one embodiment, vector processors may be programmable processors that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing functions. In at least one embodiment, a PVA may include a PVA core and two vector processing subsystem partitions. In at least one embodiment, a PVA core may include a processor subsystem, DMA engine(s) (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, a vector processing subsystem may operate as the primary processing unit of a PVA and may include a vector processing unit (“VPU”), an instruction cache, and / or a vector memory (e.g., “VMEM”).In at least one embodiment, the VPU core may include a digital signal processor, such as a single instruction multiple data ("SIMD") very long instruction word ("VLIW") digital signal processor. In at least one embodiment, a combination of SIMD and VLIW may increase throughput and speed.

[0188] In at least one embodiment, each of the vector processors may include an instruction cache and be connected to dedicated memory. Therefore, in at least one embodiment, each of the vector processors may be configured to operate independently of other vector processors. In at least one embodiment, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA may execute a common computer vision algorithm, but on different regions of an image. In at least one embodiment, vector processors included in a particular PVA may concurrently execute different image processing algorithms for an image, or even different algorithms for successive images or portions of an image.In at least one embodiment, among other things, any number of PVAs may be included in a hardware acceleration cluster, and each PVA may include any number of vector processors. In at least one embodiment, the PVA may include additional error correction code ("ECC") memory to increase the security of the overall system.

[0189] In at least one embodiment, the accelerator(s) 1514 may include an on-chip computer vision network and static random access memory ("SRAM") to provide high-bandwidth, low-latency SRAM for the accelerator(s) 1514. In at least one embodiment, the on-chip memory may include at least 4 MB of SRAM, for example, and without limitation, eight field-configurable memory blocks accessible by both a PVA and a DLA. In at least one embodiment, each pair of memory blocks may include an extended peripheral bus ("APB") interface, configuration circuitry, a controller, and a multiplexer. In at least one embodiment, any type of memory may be used.In at least one embodiment, a PVA and a DLA may access the memory via a backbone that provides high-speed access to the memory for a PVA and a DLA. In at least one embodiment, a backbone may include an on-chip computer vision network connecting a PVA and a DLA to the memory (e.g., using APB).

[0190] In at least one embodiment, an on-chip computer vision network may include an interface that determines that both a PVA and a DLA are providing ready and valid signals before transmitting control signals / addresses / data. In at least one embodiment, an interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. In at least one embodiment, an interface may conform to International Organization for Standardization ("ISO") 26262 or International Electrotechnical Commission ("IEC") 61508 standards, although other standards and protocols may be used.

[0191] In at least one embodiment, one or more of the SoC(s) 1504 may include a real-time ray tracing hardware accelerator. In at least one embodiment, the real-time ray tracing hardware accelerator may be used to quickly and efficiently determine positions and extents of objects (e.g., within a world model) to generate real-time visualization simulations, for radar signal interpretation, for sound propagation synthesis and / or analysis, for the simulation of sonar systems, for general wave propagation simulation, for comparison with lidar data for localization, and / or for other functions, and / or for other purposes.

[0192] In at least one embodiment, the accelerator(s) 1514 may have a wide range of uses for autonomous driving. In at least one embodiment, a PVA may be used for critical processing steps in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of a PVA are well suited to algorithmic domains that require predictable, low-power, and low-latency processing. In other words, a PVA is well suited for semi-dense or dense regular computations, even on small datasets, that require predictable, low-latency, and low-power runtimes. In at least one embodiment, such as in the vehicle 1500, PVAs may be designed to execute classical computer vision algorithms because of their efficiency in object detection and processing integer mathematical data.

[0193] For example, according to at least one embodiment of the technology, a PVA is used to perform computer stereo vision. In at least one embodiment, a semi-global matching-based algorithm may be used in some examples, although this is not intended to be limiting. In at least one embodiment, on-the-fly motion estimation / stereo matching is used in Level 3-5 autonomous driving applications (e.g., structure from motion, pedestrian detection, lane detection, etc.). In at least one embodiment, a PVA may perform computer stereo vision functions on inputs from two monocular cameras.

[0194] In at least one embodiment, a PVA may be used to perform dense optical flow. In at least one embodiment, a PVA could, for example, process raw radar data (e.g., using a 4D Fast Fourier Transform) to provide processed radar data. In at least one embodiment, a PVA is used for depth-of-flight processing by processing raw time-of-flight data to provide, for example, processed time-of-flight data.

[0195] In at least one embodiment, a DLA may be used to power any type of network to improve control and driving safety, including, for example and without limitation, a neural network that outputs a confidence measure for each object detection. In at least one embodiment, confidence may be represented or interpreted as a probability, or as the relative "weight" of each detection compared to other detections. In at least one embodiment, a confidence measure allows the system to make further decisions about which detections should be considered true positives and which should be considered false positives. In at least one embodiment, a system may set a threshold for the confidence measure and consider only detections that exceed the threshold to be true positives.In an embodiment using an automatic emergency braking ("AEB") system, false positive detections would cause the vehicle to automatically perform emergency braking, which is clearly undesirable. In at least one embodiment, highly confident detections can be considered as triggers for AEB. In at least one embodiment, a DLA can employ a neural network to regress the confidence value.In at least one embodiment, the neural network may use as input at least a subset of parameters, such as the dimensions of the bounding box, the ground plane estimate obtained (e.g., from another subsystem), the output of the IMU sensor(s) 1566 correlated with the orientation of the vehicle 1500, the distance, the 3D position estimates of the object obtained from the neural network and / or other sensors (e.g., LIDAR sensor(s) 1564 or RADAR sensor(s) 1560), and others.

[0196] In at least one embodiment, one or more SoC(s) 1504 may include one or more data stores 1516 (e.g., memories). In at least one embodiment, the data stores 1516 may be on-chip memory of the SoC(s) 1504, which may store neural networks to be executed on the GPU(s) 1508 and / or a DLA. In at least one embodiment, the data stores 1516 may be large enough to store multiple neural network instances for redundancy and security. In at least one embodiment, the data stores 1516 may include L2 or L3 cache(s).

[0197] In at least one embodiment, one or more of the SoC(s) 1504 may include any number of processor(s) 1510 (e.g., embedded processors). In at least one embodiment, the processor(s) 1510 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle boot power and management functions and associated security enforcement. In at least one embodiment, a boot and power management processor may be part of a boot sequence of SoC(s) 1504 and provide runtime power management services.In at least one embodiment, a boot power and management processor may provide clock and voltage programming, support for low-power system transitions, management of SoC(s) 1504 temperatures and temperature sensors, and / or management of SoC(s) 1504 power states. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC(s) 1504 may use ring oscillators to sense the temperatures of CPU(s) 1506, GPU(s) 1508, and / or accelerator(s) 1514.In at least one embodiment, if temperatures are determined to exceed a threshold, a boot and power management processor may enter a temperature fault routine and place the SoC(s) 1504 into a lower power state and / or place the vehicle 1500 into a chauffeur-to-safe stop mode (e.g., bring the vehicle 1500 to a safe stop).

[0198] In at least one embodiment, processor(s) 1510 may further include a set of embedded processors that may serve as an audio processing engine, which may be an audio subsystem enabling full hardware support for multi-channel audio across multiple interfaces, as well as a wide and flexible range of audio I / O interfaces. In at least one embodiment, an audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0199] In at least one embodiment, the processor(s) 1510 may further include an "always on" processor engine that may provide the necessary hardware functions to support low-power sensor management and wake use cases. In at least one embodiment, an "always on" processor engine may include, without limitation, a processor core, tightly coupled memory, supporting peripherals (e.g., timers and interrupt controllers), various I / O control peripherals, and routing logic.

[0200] In at least one embodiment, the processor(s) 1510 may further comprise a security cluster engine, including, without limitation, a dedicated processor subsystem for handling security management for automotive applications. In at least one embodiment, a security cluster engine may include, without limitation, two or more processor cores, tightly coupled memory, supporting peripherals (e.g., timers, an interrupt controller, etc.), and / or routing logic. In a security mode, in at least one embodiment, two or more cores may operate in a lockstep mode, functioning as a single core with comparison logic to detect any differences between their operations.In at least one embodiment, processor(s) 1510 may further include a real-time camera engine, which may include, without limitation, a dedicated processor subsystem for handling real-time camera management. In at least one embodiment, processor(s) 1510 may further include a high dynamic range signal processor, which may include, without limitation, an image signal processor that is a hardware engine that is part of a camera processing pipeline.

[0201] In at least one embodiment, processor(s) 1510 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate a final image for a player window. In at least one embodiment, a video image compositor may perform lens distortion correction on the wide-angle camera(s) 1570, the surround camera(s) 1574, and / or the in-cabin surveillance camera(s) sensors. In at least one embodiment, the in-cabin surveillance camera(s) sensor(s) is / are preferably monitored by a neural network running on another instance of SoC 1504 and configured to detect and respond to events in the cabin.In at least one embodiment, a system inside the vehicle may perform lip reading without limitation to activate cellular service and place a call, dictate emails, change the destination, activate or change the vehicle's infotainment system and settings, or enable voice-activated web browsing. In at least one embodiment, certain functions are available to the driver when a vehicle is operating in an autonomous mode and are disabled otherwise.

[0202] In at least one embodiment, a video image compositor may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in at least one embodiment where motion occurs in a video, the noise reduction appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In at least one embodiment where an image or portion of an image does not include motion, the temporal noise reduction performed by the video image compositor may use information from a previous image to reduce noise in the current image.

[0203] In at least one embodiment, a video image compositor may also be configured to perform stereo rectification on input stereo lens images. In at least one embodiment, a video image compositor may further be used for user interface composition when an operating system desktop is in use and the GPU(s) 1508 are not required to continuously render new surfaces. In at least one embodiment, a video image compositor may be used to offload the GPU(s) 1508 to improve performance and responsiveness when the GPU(s) 1508 are turned on and actively performing 3D rendering.

[0204] In at least one embodiment, one or more of SoC(s) 1504 may further include a MIPI serial camera interface for receiving video and inputs from cameras, a high-speed interface, and / or a video input block that may be used for a camera and associated pixel input functions. In at least one embodiment, one or more of SoC(s) 1504 may further include one or more input / output controllers that may be controlled by software and may be used to receive I / O signals that are not associated with a particular role.

[0205] In at least one embodiment, one or more of SoC(s) 1504 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio encoders / decoders ("codecs"), power management, and / or other devices. In at least one embodiment, SoC(s) 1504 may be used to receive data from cameras (e.g., via Gigabit Multimedia Serial Link and Ethernet channels), sensors (e.g., LIDAR sensor(s) 1564, RADAR sensor(s) 1560, etc., which may be connected via Ethernet channels), data from bus 1502 (e.g., speed of vehicle 1500, steering wheel position, etc.), data from GNSS sensor(s) 1558 (e.g., connected via an Ethernet bus or a CAN bus), etc.In at least one embodiment, one or more of SoC(s) 1504 may further include dedicated high-performance mass storage controllers, which may include their own DMA engines and which may be used to free the CPU(s) 1506 from routine data management tasks.

[0206] In at least one embodiment, the SoC(s) 1504 may be an end-to-end platform with a flexible architecture spanning automation levels 3 through 5, thereby providing a comprehensive functional safety architecture, leveraging computer vision and ADAS techniques for diversity and redundancy, and providing a platform for a flexible, reliable driving software stack along with deep learning tools. In at least one embodiment, the SoC(s) 1504 may be faster, more reliable, and even more power and space efficient than conventional systems. For example, in at least one embodiment, the accelerator(s) 1514, in combination with the CPU(s) 1506, the GPU(s) 1508, and the data memory(s) 1516, may form a fast, efficient platform for Level 3-5 autonomous vehicles.

[0207] In at least one embodiment, computer vision algorithms may be executed on CPUs that can be configured using a high-level language such as C to execute a variety of processing algorithms on a variety of visual data. However, in at least one embodiment, CPUs are often unable to meet the performance requirements of many image processing applications, such as execution time and power consumption. In at least one embodiment, many CPUs are unable to execute complex object detection algorithms in real time, which is used in in-vehicle ADAS applications and in practical Level 3-5 autonomous vehicles.

[0208] The embodiments described herein enable multiple neural networks to be executed simultaneously and / or sequentially and the results to be combined to enable Level 3-5 autonomous driving functionality. For example, in at least one embodiment, a CNN executing on a DLA or a discrete GPU (e.g., GPU(s) 1520) may include text and word recognition that enables the reading and understanding of traffic signs, including signs for which a neural network has not been specifically trained. In at least one embodiment, a DLA may further include a neural network capable of identifying, interpreting, and semantically understanding a sign and passing this semantic understanding to path planning modules running on a CPU complex.

[0209] In at least one embodiment, multiple neural networks may run simultaneously, such as in Level 3, 4, or 5 driving. For example, in at least one embodiment, a warning sign reading "Caution: Flashing lights indicate black ice," along with an electric light, may be interpreted independently or jointly by multiple neural networks. At least in one embodiment, such a warning sign may itself be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing lights indicate black ice" may be interpreted by a second deployed neural network, which informs a vehicle's path planning software (preferably running on a CPU complex) that, when flashing lights are detected, black ice is present.In at least one embodiment, a turn signal may be identified by operating a third neural network over multiple frames, which informs a vehicle's path planning software of the presence (or absence) of turn signals. In at least one embodiment, all three neural networks may run concurrently, for example, within a DLA and / or on GPU(s) 1508.

[0210] In at least one embodiment, a facial recognition and vehicle owner identification CNN may use data from camera sensors to identify the presence of an authorized driver and / or owner of the vehicle 1500. In at least one embodiment, an "always on" sensor processing engine may be used to unlock a vehicle when an owner approaches a driver's door and turns on the lights, and to disable such a vehicle in a security mode when an owner exits such a vehicle. In this way, the SoC(s) 1504 provide security against theft and / or carjacking.

[0211] In at least one embodiment, a CNN for detecting and identifying emergency vehicles may use data from microphones 1596 to detect and identify emergency vehicle sirens. In at least one embodiment, the SoC(s) 1504 use a CNN to classify environmental and urban sounds, as well as to classify visual data. In at least one embodiment, a CNN running on a DLA is trained to detect a relative approach speed of an emergency vehicle (e.g., using a Doppler effect). In at least one embodiment, a CNN may also be trained to identify emergency vehicles specific to a local area in which a vehicle is traveling, as identified by GNSS sensors 1558.In at least one embodiment, when deployed in Europe, a CNN will attempt to detect European sirens, and when deployed in North America, a CNN will attempt to identify only North American sirens. In at least one embodiment, once an emergency vehicle is detected, a control program may be used to execute an emergency vehicle safety routine, slow a vehicle, pull over to the side of the road, park a vehicle, and / or idle a vehicle using ultrasonic sensors 1562 until emergency vehicles pass by.

[0212] In at least one embodiment, the vehicle 1500 may include one or more CPU(s) 1518 (e.g., discrete CPU(s) or dCPU(s)) that may be connected to the SoC(s) 1504 via a high-speed interconnect (e.g., PCIe). In at least one embodiment, the CPU(s) 1518 may include, for example, an x86 processor. The CPU(s) 1518 may be used to perform a variety of functions, including reconciling potentially conflicting results between ADAS sensors and SoC(s) 1504 and / or monitoring the status and health of the controller(s) 1536 and / or an infotainment system on a chip (“infotainment SoC”) 1530, for example. In at least one embodiment, the SoC(s) 1504 includes one or more interconnects, and an interconnect may include a Peripheral Component Interconnect Express (PCIe).

[0213] In at least one embodiment, vehicle 1500 may include GPU(s) 1520 (e.g., discrete GPU(s) or dGPU(s)) that may be coupled to SoC(s) 1504 via a high-speed interconnect (e.g., NVIDIA's NVLINK channel). In at least one embodiment, GPU(s) 1520 may provide additional artificial intelligence functionality, e.g., by executing redundant and / or distinct neural networks, and may be used to train and / or update neural networks based at least in part on input data (e.g., sensor data) from sensors of vehicle 1500.

[0214] In at least one embodiment, the vehicle 1500 may further include a network interface 1524, which may include, without limitation, one or more wireless antennas 1526 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). In at least one embodiment, the network interface 1524 may be used to enable wireless connectivity to internet cloud services (e.g., to servers and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). In at least one embodiment, a direct connection between the vehicle 1500 and another vehicle and / or an indirect connection (e.g., via networks and the internet) may be established to communicate with other vehicles.In at least one embodiment, direct connections may be established via a vehicle-to-vehicle communication link. In at least one embodiment, a vehicle-to-vehicle communication link may provide information to the vehicle 1500 about vehicles in the vicinity of the vehicle 1500 (e.g., vehicles in front of, beside, and / or behind the vehicle 1500). In at least one embodiment, this aforementioned functionality may be part of a cooperative adaptive cruise control function of the vehicle 1500.

[0215] In at least one embodiment, the network interface 1524 may include an SoC that provides modulation and demodulation functions and enables the controller(s) 1536 to communicate over wireless networks. In at least one embodiment, the network interface 1524 may include a radio frequency front end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. In at least one embodiment, the frequency conversions may be performed in any technically feasible manner. For example, frequency conversions may be performed by known methods and / or using superheterodyne techniques. In at least one embodiment, the radio frequency front end functionality may be provided by a separate chip.In at least one embodiment, the network interfaces may include wireless functionality for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0216] In at least one embodiment, the vehicle 1500 may further include one or more data stores 1528, which may include, without limitation, off-chip memory (e.g., off-SoC(s) 1504). In at least one embodiment, the data store(s) 1528 may include, without limitation, one or more memory elements, including RAM, SRAM, dynamic random access memory ("DRAM"), video random access memory ("VRAM"), flash memory, hard drives, and / or other components and / or devices capable of storing at least one bit of data.

[0217] In at least one embodiment, the vehicle 1500 may further include GNSS sensor(s) 1558 (e.g., GPS and / or assisted GPS sensors) to assist with mapping, sensing, occupancy grid generation, and / or path planning. In at least one embodiment, any number of GNSS sensor(s) 1558 may be used, including, for example, and without limitation, a GPS using a USB port with an Ethernet-to-serial bridge (e.g., RS-232).

[0218] In at least one embodiment, the vehicle 1500 may further include RADAR sensors 1560. In at least one embodiment, the RADAR sensor(s) 1560 may be used by the vehicle 1500 for long-range vehicle detection, even in darkness and / or adverse weather conditions. In at least one embodiment, the RADAR functional safety levels may be ASIL B. In at least one embodiment, the RADAR sensor(s) 1560 may use a CAN bus and / or bus 1502 (e.g., for transmitting the data generated by the RADAR sensor(s) 1560) for control and access to object tracking data, with raw data being accessed via Ethernet channels in some examples. In at least one embodiment, a wide range of RADAR sensors may be used.For example, and without limitation, radar sensor(s) 1560 may be suitable for use as front, rear, and side radar. In at least one embodiment, one or more sensors of radar sensor(s) 1560 is a pulse-Doppler radar sensor.

[0219] In at least one embodiment, the RADAR sensor(s) 1560 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In at least one embodiment, long-range RADAR may be used for the adaptive cruise control function. In at least one embodiment, long-range RADAR systems may provide a wide field of view realized by two or more independent scans, for example, within a range of 250 m (meters). In at least one embodiment, RADAR sensor(s) 1560 may assist in distinguishing between static and moving objects and may be used by the ADAS system 1538 for emergency braking assistance and forward collision warning.In at least one embodiment, the sensor(s) 1560 included in a long-range radar system may comprise, without limitation, a monostatic multimodal radar with multiple (e.g., six or more) fixed radar antennas and a high-speed CAN and FlexRay interface. In at least one embodiment with six antennas, four antennas in the center may create a focused beam pattern useful for detecting the vehicle's surroundings at higher speeds with minimal interference from traffic in adjacent lanes. In at least one embodiment, two additional antennas may expand the field of view so that vehicles entering or exiting a lane of vehicle 1500 can be quickly detected.

[0220] For example, in at least one embodiment, medium-range radar systems may include a range of up to 160 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 150 degrees (rear). In at least one embodiment, short-range radar systems may include, without limitation, any number of radar sensors 1560 that may be installed at either end of a rear bumper. In at least one embodiment, a radar sensor system, when installed at either end of a rear bumper, may create two beams that continuously monitor blind spots in a direction rearward and to the side of a vehicle. In at least one embodiment, short-range radar systems may be used in ADAS system 1538 for blind spot detection and / or lane change assistance.

[0221] In at least one embodiment, the vehicle 1500 may further include ultrasonic sensor(s) 1562. In at least one embodiment, the ultrasonic sensor(s) 1562, which may be located at a front, rear, and / or side location of the vehicle 1500, may be used for parking assistance and / or for creating and updating an occupancy grid. In at least one embodiment, a plurality of ultrasonic sensors 1562 may be used, and different ultrasonic sensors 1562 may be used for different detection ranges (e.g., 2.5 m, 4 m). In at least one embodiment, the ultrasonic sensor(s) 1562 may operate at functional safety levels of ASIL B.

[0222] In at least one embodiment, the vehicle 1500 may include one or more LIDAR sensors 1564. In at least one embodiment, the LIDAR sensor(s) 1564 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, the LIDAR sensor(s) 1564 may operate at the ASIL B functional safety level. In at least one embodiment, the vehicle 1500 may include multiple LIDAR sensors 1564 (e.g., two, four, six, etc.) that may use an Ethernet channel (e.g., to provide data to a Gigabit Ethernet switch).

[0223] In at least one embodiment, the LIDAR sensor(s) 1564 may be capable of providing a list of objects and their distances for a 360-degree field of view. In at least one embodiment, the commercially available LIDAR sensor(s) 1564 may have an advertised range of approximately 100 m, with an accuracy of 2 cm to 3 cm, and with support for a 100 Mbps Ethernet connection, for example. In at least one embodiment, one or more non-protruding LIDAR sensors may be used. In such an embodiment, the LIDAR sensor(s) 1564 may comprise a small device that can be embedded in a front, rear, side, and / or corner position of the vehicle 1500.In at least one embodiment, the LIDAR sensor(s) 1564 in such an embodiment may provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees with a range of 200 m, even for objects with low reflectivity. In at least one embodiment, the front-mounted LIDAR sensor(s) 1564 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0224] In at least one embodiment, LIDAR technologies, such as 3D flash LIDAR, may also be used. In at least one embodiment, 3D flash LIDAR uses a laser flash as a transmission source to illuminate the surroundings of the vehicle 1500 up to a distance of approximately 200 m. In at least one embodiment, a flash LIDAR unit includes, without limitation, a receptor that records the time of flight of the laser pulse and the reflected light at each pixel, which in turn corresponds to a distance from the vehicle 1500 to objects. In at least one embodiment, flash LIDAR may enable highly accurate and distortion-free images of the surroundings to be generated with each laser flash. In at least one embodiment, four flash LIDAR sensors may be deployed, one on each side of the vehicle 1500.In at least one embodiment, 3D flash LIDAR systems include, without limitation, a solid-state 3D star array LIDAR camera with no moving parts other than a fan (e.g., a non-scanning LIDAR device). In at least one embodiment, the flash LIDAR device may use a 5-nanosecond Class I (eye-safe) laser pulse per image and collect the reflected laser light as a 3D range point cloud and co-registered intensity data.

[0225] In at least one embodiment, the vehicle 1500 may further include one or more IMU sensors 1566. In at least one embodiment, the IMU sensor(s) 1566 may be located in the center of a rear axle of the vehicle 1500. In at least one embodiment, the IMU sensor(s) 1566 may include, for example, and without limitation, one or more accelerometers, magnetometers, gyroscopes, a magnetic compass, magnetic compasses, and / or other types of sensors. In at least one embodiment, such as in six-axis applications, the IMU sensor(s) 1566 may include, without limitation, accelerometers and gyroscopes. In at least one embodiment, such as in nine-axis applications, the IMU sensor(s) 1566 may include, without limitation, accelerometers, gyroscopes, and magnetometers.

[0226] In at least one embodiment, the IMU sensor(s) 1566 may be implemented as a miniaturized, high-performance GPS-based inertial navigation system ("GPS / INS") that combines microelectromechanical systems ("MEMS") inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and attitude. In at least one embodiment, the IMU sensors 1566 may enable the vehicle 1500 to estimate its heading without requiring input from a magnetic sensor by directly observing velocity changes from a GPS and correlating them with the IMU sensors 1566. In at least one embodiment, the IMU sensor(s) 1566 and GNSS sensor(s) 1558 may be combined into a single integrated unit.

[0227] In at least one embodiment, the vehicle 1500 may include one or more microphones 1596 disposed in and / or around the vehicle 1500.

[0228] In at least one embodiment, the microphone(s) 1596 may be used, among other things, to detect and identify emergency vehicles.

[0229] In at least one embodiment, the vehicle 1500 may further include any number of camera types, including stereo camera(s) 1568, wide-angle camera(s) 1570, infrared camera(s) 1572, surround camera(s) 1574, long-range camera(s) 1598, medium-range camera(s) 1576, and / or other camera types. In at least one embodiment, cameras may be used to capture image data around the entire perimeter of the vehicle 1500. In at least one embodiment, the types of cameras used depend on the vehicle 1500. In at least one embodiment, any combination of camera types may be used to provide the required coverage around the vehicle 1500. In at least one embodiment, the number of cameras employed may vary depending on the embodiment.In at least one embodiment, the vehicle 1500 may include, for example, six cameras, seven cameras, ten cameras, twelve cameras, or another number of cameras. In at least one embodiment, the cameras may support, for example, and without limitation, Gigabit Multimedia Serial Link ("GMSL") and / or Gigabit Ethernet communication. In at least one embodiment, each camera may be configured as previously described in . Fig. 15A and Fig. 15B will be described in more detail.

[0230] In at least one embodiment, the vehicle 1500 may further include one or more vibration sensors 1542. In at least one embodiment, the vibration sensor(s) 1542 may measure vibrations of components of the vehicle 1500, such as the axle(s). For example, in at least one embodiment, changes in vibrations may indicate a change in the road surface. In at least one embodiment, when two or more vibration sensors 1542 are used, differences between vibrations may be used to determine friction or slippage of the road surface (e.g., when there is a difference in vibration between a driven axle and a free-spinning axle).

[0231] In at least one embodiment, the vehicle 1500 may include the ADAS system 1538. In at least one embodiment, the ADAS system 1538 may include, in some examples, without limitation, an SoC. In at least one embodiment, the ADAS system 1538 may include, without limitation, any number and combination of an autonomous / adaptive / automatic cruise control (“ACC”) system, a cooperative adaptive cruise control (“CACC”) system, a forward crash warning (“FCW”) system, an automatic emergency braking (“AEB”) system, a lane departure warning (“LDW”) system, a lane keep assist (“LKA”) system, a blind spot warning (“BSW”) system, a rear cross traffic warning (“RCTW”) system, a forward collision warning (“CW”) system, a lane centering (“LC”) system, and / or other systems, features, and / or functions.

[0232] In at least one embodiment, the ACC system may use RADAR sensor(s) 1560, LIDAR sensor(s) 1564, and / or any number of cameras. In at least one embodiment, the ACC system may include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, a longitudinal ACC system monitors and controls the distance to another vehicle immediately in front of the vehicle 1500 and automatically adjusts the speed of the vehicle 1500 to maintain a safe distance from preceding vehicles. In at least one embodiment, a lateral ACC system provides follow-through and advises the vehicle 1500 to change lanes if necessary. In at least one embodiment, a lateral ACC system is connected to other ADAS applications, such as LC and CW.

[0233] In at least one embodiment, a CACC system utilizes information from other vehicles, which may be received via a network interface 1524 and / or wireless antenna(s) 1526 from other vehicles over a wireless connection or indirectly via a network connection (e.g., over the Internet). In at least one embodiment, direct connections may be provided by a vehicle-to-vehicle ("V2V") communication link, while indirect connections may be provided by an infrastructure-to-vehicle ("I2V") communication link. Generally, V2V communication provides information about immediately preceding vehicles (e.g., vehicles immediately in front of and in the same lane as vehicle 1500), while I2V communication provides information about traffic further ahead.In at least one embodiment, a CACC system may include either one or both I2V and V2V information sources. In at least one embodiment, a CACC system may be more reliable given information about vehicles ahead of vehicle 1500 and has the potential to improve traffic flow and reduce congestion on the road.

[0234] In at least one embodiment, an FCW system is designed to warn a driver of a hazard so that the driver can take corrective action. In at least one embodiment, an FCW system uses a forward-facing camera and / or RADAR sensor(s) 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is / are electrically coupled to provide feedback to the driver, such as a display, speaker, and / or vibrating component. In at least one embodiment, an FCW system may provide a warning, for example, in the form of a sound, a visual warning, a vibration, and / or a rapid braking pulse.

[0235] In at least one embodiment, an AEB system detects an impending forward collision with another vehicle or other object and may automatically apply the brakes if a driver does not take corrective action within a specified time or distance parameter. In at least one embodiment, the AEB system may utilize forward-facing camera(s) and / or RADAR sensor(s) 1560 connected to a dedicated processor, DSP, FPGA, and / or ASIC. In at least one embodiment, when an AEB system detects a hazard, it will typically first alert a driver to take corrective action to avoid a collision, and if that driver does not take corrective action, the AEB system may automatically apply the brakes to prevent or at least mitigate the effects of a predicted collision.In at least one embodiment, an AEB system may include techniques such as dynamic brake assistance and / or crash-imminent braking.

[0236] In at least one embodiment, an LDW system provides visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 1500 crosses lane markings. In at least one embodiment, an LDW system is not activated when a driver indicates an intentional lane departure, for example, by activating a turn signal. In at least one embodiment, an LDW system may utilize forward-facing cameras coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to provide feedback to the driver, for example, via a display, speaker, and / or vibrating component. In at least one embodiment, an LKA system is a variant of an LDW system.In at least one embodiment, an LKA system provides steering intervention or braking to correct the vehicle 1500 when the vehicle 1500 begins to depart from its lane.

[0237] In at least one embodiment, a BSW system detects and warns the driver of vehicles in the vehicle's blind spot. In at least one embodiment, a BSW system may provide a visual, audible, and / or tactile warning to indicate that merging or changing lanes is unsafe. In at least one embodiment, a BSW system may provide an additional warning when a driver activates a turn signal. In at least one embodiment, a BSW system may utilize rear-facing camera(s) and / or RADAR sensor(s) 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibrating component.

[0238] In at least one embodiment, an RCTW system may provide visual, audible, and / or tactile notification when an object is detected outside the range of the rearview camera when the vehicle 1500 is reversing. In at least one embodiment, an RCTW system includes an AEB system to ensure that the vehicle brakes are applied to avoid a crash. In at least one embodiment, an RCTW system may utilize one or more rear-facing RADAR sensors 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to provide feedback to the driver, for example, via a display, speaker, and / or vibrating component.

[0239] In at least one embodiment, conventional ADAS systems may produce false positive results, which may be annoying and distracting for the driver, but are generally not catastrophic, as conventional ADAS systems warn the driver and give them the opportunity to decide whether a safety condition truly exists and act accordingly. In at least one embodiment, in the event of conflicting results, the vehicle 1500 itself decides whether to consider the result of a primary computer or a secondary computer (e.g., a first control unit or a second control unit of the control units 1536). In at least one embodiment, the ADAS system 1538 may, for example, be a backup and / or secondary computer that provides perception information to a rationality module of the backup computer.In at least one embodiment, a backup computer rationality monitor may run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. In at least one embodiment, the outputs of the ADAS system 1538 may be forwarded to a higher-level MCU. In at least one embodiment, if the outputs of a primary computer and the outputs of a secondary computer conflict, a higher-level MCU determines how to resolve the conflict to ensure safe operation.

[0240] In at least one embodiment, a primary computer may be configured to provide a supervising MCU with a confidence score indicating the primary computer's confidence in a selected outcome. In at least one embodiment, if this confidence score exceeds a threshold, the supervising MCU may follow the primary computer's instruction regardless of whether the secondary computer provides a conflicting or inconsistent outcome. In at least one embodiment, in cases where a confidence score does not meet a threshold and where primary and secondary computers indicate different outcomes (e.g., a conflict), a supervising MCU may arbitrate between the computers to determine an appropriate outcome.

[0241] In at least one embodiment, a monitoring MCU may be configured to execute one or more neural networks trained and configured to determine, based at least in part on the outputs of a primary computer and the outputs of a secondary computer, the conditions under which that secondary computer will provide false alarms. In at least one embodiment, the neural network(s) in a monitoring MCU may learn when the output of a secondary computer can and cannot be trusted. In at least one embodiment, when the secondary computer is a radar-based FCW system, one or more neural networks in that parent MCU may learn when an FCW system identifies metallic objects that do not actually pose a threat, such as a drain grate or manhole cover, that trigger an alarm.In at least one embodiment, a neural network in a supervisory MCU may learn to override lane departure warning when a secondary computer is a camera-based lane departure warning system and lane departure is actually the safest maneuver. In at least one embodiment, a supervisory MCU may include at least one DLA or GPU capable of executing neural networks with associated memory. In at least one embodiment, a supervisory MCU may include and / or be included as a component of the SoC(s) 1504.

[0242] In at least one embodiment, ADAS system 1538 may include a secondary computer that performs ADAS functions using conventional computer vision rules. In at least one embodiment, this secondary computer may use classic computer vision (if-then) rules, and the presence of a neural network(s) in a higher-level MCU may improve reliability, safety, and performance. In at least one embodiment, the different implementation and intentional non-identity make the overall system more fault-tolerant, particularly against errors caused by software functions (or software-hardware interfaces).For example, in at least one embodiment, if a software error occurs in the software running on a primary computer and non-identical software code runs on a secondary computer that produces a consistent overall result, then a supervising MCU may have greater confidence that an overall result is correct and an error in the software or hardware on that primary computer does not cause a significant error.

[0243] In at least one embodiment, an output of the ADAS system 1538 may be fed to the perception block of a primary computer and / or the dynamic driving task block of a primary computer. For example, in at least one embodiment, if the ADAS system 1538 indicates a forward crash warning due to an object immediately ahead, a perception block may use this information in identifying objects. In at least one embodiment, a secondary computer may have its own neural network trained, thus reducing the risk of false alarms, as described herein.

[0244] In at least one embodiment, the vehicle 1500 may further include an infotainment SoC 1530 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, in at least one embodiment, the infotainment system SoC 1530 may not be an SoC and may include, without limitation, two or more discrete components. In at least one embodiment, the infotainment SoC 1530 may include, without limitation, a combination of hardware and software that can be used to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.).) and / or information services (e.g., navigation systems, rear parking assistance, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fuel level, oil level, door open / close, air filter information, etc.) to the vehicle 1500. The infotainment SoC 1530 could include, for example, radios, record players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, WiFi, steering wheel audio controls, hands-free calling, a heads-up display (“HUD”), an HMI display 1534, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components.In at least one embodiment, the infotainment SoC 1530 may be further used to provide information (e.g., visual and / or audible) to the user(s) of the vehicle 1500, such as information from the ADAS system 1538, autonomous driving information such as planned vehicle maneuvers, trajectories, environmental information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0245] In at least one embodiment, the infotainment SoC 1530 may include any amount and type of GPU functionality. In at least one embodiment, the infotainment SoC 1530 may communicate with other devices, systems, and / or components of the vehicle 1500 via the bus 1502. In at least one embodiment, the infotainment SoC 1530 may be coupled to a supervisory MCU so that a GPU of an infotainment system may perform some self-driving functions if the primary controller(s) 1536 (e.g., primary and / or backup computers of the vehicle 1500) fail. In at least one embodiment, the infotainment SoC 1530 may place the vehicle 1500 into a chauffeur-to-safe-stop mode, as described herein.

[0246] In at least one embodiment, the vehicle 1500 may further include an instrument cluster 1532 (e.g., a digital dashboard, an electronic instrument cluster, a digital instrument panel, etc.). In at least one embodiment, the instrument cluster 1532 may include, without limitation, a controller and / or a supercomputer (e.g., a discrete controller or a supercomputer). In at least one embodiment, the instrument cluster 1532 may include, without limitation, any number and combination of instruments, such as, but not limited to, a speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine trouble light(s), supplemental restraint system information (e.g., airbags), lighting controls, safety system controls, navigation information, etc.In some examples, information may be displayed and / or shared between infotainment SoC 1530 and instrument cluster 1532. In at least one embodiment, instrument cluster 1532 may comprise a portion of infotainment SoC 1530, or vice versa.

[0247] In at least one embodiment, components of Fig. 15C or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig. 15C one or more circuits to use one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating different content of at least one of the two or more different images than the one or more objects. In at least one embodiment, the components of Fig. 15C one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0248] Fig. 15D is a diagram of a system for communication between one or more cloud-based servers and the autonomous vehicle 1500 of Fig. 15A, according to at least one embodiment. In at least one embodiment, the system may include, without limitation, server(s) 1578, network(s) 1590, and any number and type of vehicles, including vehicle 1500. In at least one embodiment, server(s) 1578 may include, without limitation, a plurality of GPUs 1584(A)-1584(H) (collectively referred to herein as GPUs 1584), PCIe switches 1582(A)-1582(D) (collectively referred to herein as PCIe switches 1582), and / or CPUs 1580(A)-1580(B) (collectively referred to herein as CPUs 1580). In at least one embodiment, the GPUs 1584, the CPUs 1580, and the PCIe switches 1582 may be interconnected with high-speed interconnects, such as, without limitation, the NVLink interfaces 1588 and / or PCIe interconnects 1586 developed by NVIDIA.In at least one embodiment, the GPUs 1584 are connected via an NVLink and / or NVSwitch SoC, and the GPUs 1584 and the PCIe switches 1582 are connected via PCIe connections. Although eight GPUs 1584, two CPUs 1580, and four PCIe switches 1582 are shown, this is not intended to be limiting. In at least one embodiment, each of the servers 1578 may include, without limitation, any number of GPUs 1584, CPUs 1580, and / or PCIe switches 1582 in any combination. For example, in at least one embodiment, the server(s) 1578 could each include eight, sixteen, thirty-two, and / or more GPUs 1584.

[0249] In at least one embodiment, the server(s) 1578 may receive, via the network(s) 1590 and from vehicles, image data representative of images depicting unexpected or changed road conditions, such as recently commenced roadwork. In at least one embodiment, the server(s) 1578 may transmit, via the network(s) 1590 and to the vehicles, updated or other neural network 1592 and / or map information 1594 including, among other things, information about traffic and road conditions. In at least one embodiment, the updates to the map information 1594 may include, without limitation, updates to the HD map 1522, such as information about construction, potholes, detours, flooding, and / or other obstacles.In at least one embodiment, the neural networks 1592 and / or the map information 1594 may result from recent training and / or experience represented in data received from any number of vehicles in an environment and / or may be based at least in part on training performed in a data center (e.g., using server(s) 1578 and / or other servers).

[0250] In at least one embodiment, the server(s) 1578 may be used to train machine learning models (e.g., neural networks) based at least in part on training data. In at least one embodiment, the training data may be generated by vehicles and / or in a simulation (e.g., using a game engine). In at least one embodiment, any amount of training data is labeled (e.g., if the associated neural network benefits from supervised learning) and / or subjected to other preprocessing. In at least one embodiment, any amount of training data is unlabeled and / or preprocessed (e.g., if the associated neural network does not require supervised learning).In at least one embodiment, once machine learning models are trained, machine learning models may be used by vehicles (e.g., transmitted to vehicles via network(s) 1590), and / or machine learning models may be used by server(s) 1578 to remotely monitor vehicles.

[0251] In at least one embodiment, the server(s) 1578 may receive data from vehicles and apply data to real-time neural networks to perform intelligent inferences in real time. In at least one embodiment, the server(s) 1578 may include deep learning supercomputers and / or dedicated AI computers powered by GPU(s) 1584, such as DGX and DGX Station machines developed by NVIDIA. However, in at least one embodiment, the server(s) 1578 may also include a deep learning infrastructure using CPU-powered data centers.

[0252] In at least one embodiment, the deep learning infrastructure of server(s) 1578 may be capable of performing rapid, real-time inferencing and may utilize this capability to evaluate and verify the health of processors, software, and / or associated hardware in vehicle 1500. For example, in at least one embodiment, the deep learning infrastructure may receive periodic updates from vehicle 1500, such as a sequence of images and / or objects that vehicle 1500 has located in that sequence of images (e.g., via computer vision and / or other machine object classification techniques).In at least one embodiment, the deep learning infrastructure may run its own neural network to identify objects and compare them to objects identified by the vehicle 1500, and if the results do not match and the deep learning infrastructure concludes that the AI ​​in the vehicle 1500 is malfunctioning, then the server(s) 1578 may send a signal to the vehicle 1500 instructing a fail-safe computer of the vehicle 1500 to take over control, notify the passengers, and perform a safe parking maneuver.

[0253] In at least one embodiment, the server(s) 1578 may include GPU(s) 1584 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT 3 devices). In at least one embodiment, a combination of GPU-based servers and inference acceleration may enable real-time responsiveness. In at least one embodiment, for example, when performance is less critical, servers with CPUs, FPGAs, and other processors may be used for inference building. In at least one embodiment, hardware structure(s) 1215 are used to perform one or more embodiments. Details of the hardware structure(s) 1215 are described herein in connection with Fig. 12A and / or 12B. COMPUTER SYSTEMS

[0254] Fig. 16 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC), or a combination thereof, formed with a processor that may include execution units for executing an instruction, according to at least one embodiment. In at least one embodiment, a computer system 1600 may include, without limitation, a component such as a processor 1602 for employing execution units including logic for performing algorithms for processing data in accordance with the present disclosure, as in the embodiment described herein.In at least one embodiment, computer system 1600 may include processors such as the PENTIUM® processor family, Xeon™, Itanium®, XScale™ and / or StrongARM™, Intel® Core™, or Intel® Nervana™ microprocessors available from Intel Corporation of Santa Clara, California, although other systems (including personal computers with other microprocessors, technical workstations, set-top boxes, and the like) may be used. In at least one embodiment, computer system 1600 may run a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0255] Embodiments may be used in other devices such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, embedded applications may include a microcontroller, a digital signal processor ("DSP"), a system on a chip, network computers ("NetPCs"), set-top boxes, network hubs, wide area network ("WAN") switches, or any other system capable of executing one or more instructions according to at least one embodiment.

[0256] In at least one embodiment, computer system 1600 may include, without limitation, a processor 1602, which may include, without limitation, one or more execution units 1608 to perform machine learning model training and / or inferencing according to the techniques described herein. In at least one embodiment, computer system 1600 is a desktop or server system having a processor, but in another embodiment, computer system 1600 may be a multiprocessor system. In at least one embodiment, processor 1602 may include, without limitation, a complex instruction set computer (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing a combination of instruction sets, or any other computing device, such as a digital signal processor.In at least one embodiment, the processor 1602 may be connected to a processor bus 1610 that may transmit data signals between the processor 1602 and other components in the computer system 1600.

[0257] In at least one embodiment, processor 1602 may include, without limitation, an internal Level 1 ("L1") cache 1604. In at least one embodiment, processor 1602 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache may be external to processor 1602. Other embodiments may also include a combination of internal and external caches, depending on the particular implementation and needs. In at least one embodiment, a register file 1606 may store different data types in various registers, including, but not limited to, integer registers, floating-point registers, status registers, and an instruction pointer register.

[0258] In at least one embodiment, execution unit 1608, which includes, without limitation, logic for performing integer and floating-point operations, is also located in processor 1602. In at least one embodiment, processor 1602 may also include microcode read-only memory ("ROM") ("ucode") that stores microcode for certain macroinstructions. In at least one embodiment, execution unit 1608 may include logic for handling a packed instruction set 1609. In at least one embodiment, by including a packed instruction set 1609 in an instruction set of a general-purpose processor, along with associated instruction execution circuitry, operations used by many multimedia applications may be performed using packed data in processor 1602.In at least one embodiment, many multimedia applications can be accelerated and executed more efficiently by utilizing the full width of a processor's data bus to perform operations on packed data, thereby eliminating the need to transfer smaller units of data across the processor's data bus to perform one or more operations on one data element at a time.

[0259] In at least one embodiment, execution unit 1608 may also be used in microcontrollers, embedded processors, graphics processors, DSPs, and other types of logic circuits. In at least one embodiment, computer system 1600 may include, without limitation, memory 1620. In at least one embodiment, memory 1620 may be dynamic random access memory ("DRAM"), static random access memory ("SRAM"), flash memory, or other memory. In at least one embodiment, memory 1620 may store instructions 1619 and / or data 1621 represented by data signals that may be executed by processor 1602.

[0260] In at least one embodiment, a system logic chip may be connected to processor bus 1610 and memory 1620. In at least one embodiment, a system logic chip may include, without limitation, a memory control hub ("MCH") 1616, and processor 1602 may communicate with MCH 1616 via processor bus 1610. In at least one embodiment, MCH 1616 may provide a high-bandwidth memory path 1618 to memory 1620 for instruction and data storage, as well as for graphics command, data, and texture storage. In at least one embodiment, MCH 1616 may route data signals between processor 1602, memory 1620, and other components in computer system 1600, and may bridge data signals between processor bus 1610, memory 1620, and a system I / O interface 1622. In at least one embodiment, a system logic chip may provide a graphics port for connection to a graphics controller.In at least one embodiment, the MCH 1616 may be coupled to the memory 1620 via a high-bandwidth memory path 1618, and a graphics / video card 1612 may be coupled to the MCH 1616 via an Accelerated Graphics Port (“AGP”) connection 1614.

[0261] In at least one embodiment, computer system 1600 may use system I / O interface 1622 as a proprietary hub interface bus to connect MCH 1616 to an I / O control hub ("ICH") 1630. In at least one embodiment, ICH 1630 may provide direct connections to some I / O devices via a local I / O bus. In at least one embodiment, a local I / O bus may include, without limitation, a high-speed I / O bus for connecting peripherals to memory 1620, a chipset, and processor 1602. Examples may include, without limitation, an audio controller 1629, a firmware hub (“Flash BIOS”) 1628, a wireless transceiver 1626, a data store 1624, a legacy I / O controller 1623 with user input and keyboard interfaces 1625, a serial expansion port 1627, such as a Universal Serial Bus (“USB”) port, and a network controller 1634.In at least one embodiment, data storage 1624 may include a hard disk drive, a floppy disk drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0262] In at least one embodiment, Fig. 16 a system comprising interconnected hardware devices or “chips”, while in other embodiments Fig. 16 may show an exemplary SoC. In at least one embodiment, the Fig. 16 may be connected using proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of computer system 1600 are interconnected via Compute Express Link (CXL) connections.

[0263] Logic 1215 is used to perform inference and / or training operations associated with one or more embodiments. Details of logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in computer system 1600 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0264] In at least one embodiment, components of Fig. 16 or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig. 16 one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig. 16 one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0265] Fig. 17 is a block diagram illustrating an electronic device 1700 for using a processor 1710 according to at least one embodiment. In at least one embodiment, the electronic device 1700 may be, for example and without limitation, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0266] In at least one embodiment, the electronic device 1700 may include, without limitation, a processor 1710 communicatively connected to any number or type of components, peripherals, modules, or devices. In at least one embodiment, the processor 1710 is coupled via a bus or interface, such as an I2C bus, a System Management Bus ("SMBus"), a Low Pin Count (LPC) bus, a Serial Peripheral Interface ("SPI"), a High Definition Audio (HDA) bus, a Serial Advance Technology Attachment (SATA) bus, a Universal Serial Bus ("USB") (versions 1, 2, 3, etc.), or a Universal Asynchronous Receiver / Transmitter Bus ("UART"). In at least one embodiment, Fig. 17 a system comprising interconnected hardware devices or “chips”, while in other embodiments Fig. 17 may show an exemplary SoC. In at least one embodiment, the Fig. 17 may be interconnected via proprietary connections, standardized connections (e.g., PCIe), or a combination thereof. In at least one embodiment, one or more components of Fig. 17 are connected via Compute Express Link (CXL) connections.

[0267] At least in one embodiment, Fig. 17 a display 1724, a touchscreen 1725, a touchpad 1730, a Near Field Communications unit (“NFC”) 1745, a sensor hub 1740, a thermal sensor 1746, an Express Chipset (“EC”) 1735, a Trusted Platform Module (“TPM”) 1738, BIOS / Firmware / Flash Memory (“BIOS, FW Flash”) 1722, a DSP 1760, a drive 1720 such as a Solid State Disk (“SSD”) or a Hard Drive (“HDD”), a wireless local area network unit (“WLAN”) 1750, a Bluetooth unit 1752, a wireless wide area network unit (“WWAN”) 1756, a Global Positioning System (GPS) unit 1755, a camera (“USB 3.0 Camera”) 1754, such as a USB 3.0 Camera, and / or a Low Power Double Data Rate ("LPDDR") memory unit ("LPDDR3") 1715, implemented, for example, according to an LPDDR3 standard. These components may each be implemented in any suitable manner.

[0268] In at least one embodiment, other components may be communicatively coupled to processor 1710 via components described herein. In at least one embodiment, an accelerometer 1741, an ambient light sensor ("ALS") 1742, a compass 1743, and a gyroscope 1744 may be communicatively coupled to sensor hub 1740. In at least one embodiment, a thermal sensor 1739, a fan 1737, a keyboard 1736, and a touchpad 1730 may be communicatively coupled to EC 1735. In at least one embodiment, speakers 1763, headphones 1764, and a microphone 1765 may be communicatively coupled to an audio unit ("audio codec and class-D amplifier") 1762, which in turn may be communicatively coupled to DSP 1760. In at least one embodiment, the audio unit 1762 may include, for example and without limitation, an audio encoder / decoder ("codec") and a Class D amplifier.In at least one embodiment, a SIM card ("SIM") 1757 may be communicatively coupled to the WWAN unit 1756. In at least one embodiment, components such as the WLAN unit 1750 and the Bluetooth unit 1752, as well as the WWAN unit 1756, may be implemented in a Next Generation Form Factor ("NGFF").

[0269] Logic 1215 is used to perform inference and / or training operations in connection with one or more embodiments. Details of logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in electronic device 1700 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0270] In at least one embodiment, components of Fig. 17 or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig. 17 one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig. 17 one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0271] Fig. Figure 18 illustrates a computer system 1800 according to at least one embodiment. In at least one embodiment, the computer system 1800 is configured to implement various processes and methods described in this disclosure.

[0272] In at least one embodiment, computer system 1800 includes, without limitation, at least one central processing unit ("CPU") 1802 connected to a communications bus 1810 implemented using any suitable protocol, such as PCI ("Peripheral Component Interconnect"), Peripheral Component Interconnect Express ("PCI-Express"), AGP ("Accelerated Graphics Port"), HyperTransport, or any other bus or point-to-point communications protocol(s). In at least one embodiment, computer system 1800 includes, without limitation, main memory 1804 and control logic (e.g., implemented as hardware, software, or a combination thereof), and data is stored in main memory 1804, which may take the form of random access memory ("RAM").In at least one embodiment, a network interface subsystem (“network interface”) 1822 provides an interface to other computing devices and networks to receive data from and transmit data to other systems with the computing system 1800.

[0273] In at least one embodiment, computer system 1800 includes, without limitation, input devices 1808, a parallel processing system 1812, and display devices 1806, which may be implemented with a conventional cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a light-emitting diode ("LED"), a plasma display, or other suitable display technology. In at least one embodiment, user input is received from input devices 1808 such as a keyboard, mouse, touchpad, microphone, etc. In at least one embodiment, each module described herein may be packaged on a single semiconductor platform to form a processing system.

[0274] Logic 1215 is used to perform inference and / or training operations associated with one or more embodiments. Details of inference and / or training logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in computer system 1800 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0275] In at least one embodiment, components of Fig. 18 or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig. 18 one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig. 18 one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0276] Fig. 19 shows a computer system 1900 according to at least one embodiment. In at least one embodiment, the computer system 1900 includes, without limitation, a computer 1910 and a USB flash drive 1920. In at least one embodiment, the computer 1910 may include, without limitation, any number and type of processor(s) (not shown) and memory (not shown). In at least one embodiment, the computer 1910 includes, without limitation, a server, a cloud instance, a laptop, and a desktop computer.

[0277] In at least one embodiment, the USB flash drive 1920 includes, without limitation, a processing unit 1930, a USB interface 1940, and USB interface logic 1950. In at least one embodiment, the processing unit 1930 may be any instruction execution system, device, or apparatus capable of executing instructions. In at least one embodiment, the processing unit 1930 may include, without limitation, any number and type of compute cores (not shown). In at least one embodiment, the processing unit 1930 includes an application-specific integrated circuit ("ASIC") optimized to perform any number and type of machine learning-related operations.For example, in at least one embodiment, processing unit 1930 is a tensor processing unit ("TPC") optimized for performing machine learning inference operations. In at least one embodiment, processing unit 1930 is a video processing unit ("VPU") optimized for performing video processing and machine learning operations.

[0278] In at least one embodiment, the USB interface 1940 may be any type of USB plug or USB receptacle. For example, in at least one embodiment, the USB interface 1940 is a USB 3.0 Type-C receptacle for data and power. In at least one embodiment, the USB interface 1940 is a USB 3.0 Type-A plug. In at least one embodiment, the USB interface logic 1950 may include any amount and type of logic that enables the processing unit 1930 to communicate with devices (e.g., the computer 1910) via the USB port 1940.

[0279] Logic 1215 is used to perform inference and / or training operations in connection with one or more embodiments. Details of logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in computer system 1900 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0280] In at least one embodiment, components of Fig. 19 or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig. 19 one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig. 19 one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0281] Fig. 20A illustrates an example architecture in which a plurality of GPUs 2010(1)-2010(N) are communicatively coupled to a plurality of multi-core processors 2005(1)-2005(M) via high-speed interconnects 2040(1)-2040(N) (e.g., buses, point-to-point interconnects, etc.). In at least one embodiment, the high-speed interconnects 2040(1)-2040(N) support communication throughput of 4 GB / s, 30 GB / s, 80 GB / s, or more. In at least one embodiment, various interconnect protocols may be used, including, but not limited to, PCIe 4.0 or 5.0 and NVLink 2.0. In various figures, "N" and "M" represent positive integers, the values ​​of which may vary from figure to figure. In at least one embodiment, one or more GPUs in a plurality of GPUs 2010(1)-2010(N) includes one or more graphics cores (also referred to simply as “cores”) 2300, as shown in the Fig. 23A and Fig. 23B. In at least one embodiment, one or more graphics cores 2300 may be referred to as streaming multiprocessors ("SMs"), stream processors ("SPs"), stream processing units ("SPUs"), compute units ("CUs"), execution units ("EUs"), and / or slices, where a slice in this context may refer to a portion of the processing resources in a processing unit (e.g., 16 cores, a ray tracing unit, a thread director, or scheduler).

[0282] Additionally, and in at least one embodiment, two or more GPUs 2010 are interconnected via high-speed interconnects 2029(1)-2029(2), which may be implemented using similar or different protocols / connections than those used for high-speed interconnects 2040(1)-2040(N). Similarly, two or more multi-core processors 2005 may be interconnected via a high-speed interconnect 2028, which may be symmetric multiprocessor (SMP) buses operating at 20 GB / s, 30 GB / s, 120 GB / s, or more. Alternatively, all communication between the various Fig. 20A using similar protocols / connections (for example, via a common connection structure).

[0283] In at least one embodiment, each multi-core processor 2005 is communicatively coupled to a processor memory 2001(1)-2001(M) via memory interconnects 2026(1)-2026(M), and each GPU 2010(1)-2010(N) is communicatively coupled to GPU memory 2020(1)-2020(N) via GPU memory interconnects 2050(1)-2050(N). In at least one embodiment, memory interconnects 2026 and 2050 may use similar or different memory access technologies. For example, the processor memories 2001 (1)-2001 (M) and the GPU memories 2020 may be volatile memories, such as dynamic random access memories (DRAMs) (including stacked DRAMs), graphics DDR SDRAM (GDDR) (e.g., GDDR5, GDDR6), or high bandwidth memory (HBM), and / or they may be non-volatile memories, such as 3D XPoint or Nano-Ram.In at least one embodiment, a portion of the processor memory 2001 may be volatile memory and another portion may be non-volatile memory (e.g., using a two-level memory hierarchy (2LM)).

[0284] As described herein, various multi-core processors 2005 and GPUs 2010 may be physically coupled to a particular memory 2001 or 2020, respectively, and / or a unified memory architecture may be implemented in which a virtual system address space (also referred to as "effective address space") is distributed across different physical memories. For example, processor memories 2001(1)-2001(M) may each comprise 64 GB of system address space, and GPU memories 2020(1)-2020(N) may each comprise 32 GB of system address space, resulting in a total of 256 GB of addressable memory when M=2 and N=4. Other values ​​for N and M are possible.

[0285] Fig. 20B shows additional details for a connection between a multi-core processor 2007 and a graphics acceleration module 2046 in accordance with an exemplary embodiment. In at least one embodiment, the graphics acceleration module 2046 may include one or more GPU chips integrated on a line card connected to the processor 2007 via a high-speed interconnect 2040 (e.g., a PCIe bus, NVLink, etc.). Alternatively, in at least one embodiment, the graphics acceleration module 2046 may be integrated on a package or die with the processor 2007.

[0286] In at least one embodiment, processor 2007 includes a plurality of cores 2060A-2060D (which may be referred to as "execution units"), each having a translation lookaside buffer ("TLB") 2061A-2061D and one or more caches 2062A-2062D. In at least one embodiment, cores 2060A-2060D may include various other components for executing instructions and processing data, not shown. In at least one embodiment, caches 2062A-2062D may include Level 1 (L1) and Level 2 (L2) caches. Additionally, one or more shared caches 2056 may be included in caches 2062A-2062D and shared by multiple cores 2060A-2060D. For example, one embodiment of the 2007 processor includes 24 cores, each with its own L1 cache, twelve shared L2 caches, and twelve shared L3 caches.In this embodiment, one or more L2 and L3 caches are shared between two adjacent cores. In at least one embodiment, processor 2007 and graphics acceleration module 2046 are coupled to system memory 2014, which includes processor memories 2001(1)-2001(M). Fig. 20A can be included.

[0287] In at least one embodiment, coherency for data and instructions stored in various caches 2062A-2062D, 2056, and system memory 2014 is maintained via inter-core communication over a coherency bus 2064. For example, in at least one embodiment, each cache may have cache coherency logic / circuitry associated therewith to communicate over the coherency bus 2064 in response to detected reads or writes to particular cache lines. In at least one embodiment, a cache coherency protocol is implemented over the coherency bus 2064 to sniff out cache accesses.

[0288] In at least one embodiment, a proxy circuit 2025 communicatively couples the graphics acceleration module 2046 to the coherence bus 2064, allowing the graphics acceleration module 2046 to participate in a cache coherence protocol as a peer of the cores 2060A-2060D. Specifically, in at least one embodiment, an interface 2035 provides connectivity to the proxy circuit 2025 via the high-speed interconnect 2040, and an interface 2037 connects the graphics acceleration module 2046 to the high-speed interconnect 2040.

[0289] In at least one embodiment, circuitry 2036 provides cache management, memory access, context management, and interrupt management services for a plurality of graphics processing engines 2031(1)-2031(N) of the graphics acceleration module 2046. In at least one embodiment, the graphics processing engines 2031(1)-2031(N) may each comprise a separate graphics processing unit (GPU). In at least one embodiment, a plurality of graphics processing engines 2031(1)-2031(N) of the graphics acceleration module 2046 comprise one or more graphics cores 2300, as described in connection with the Fig. 23A and Fig. 23B. In at least one embodiment, the graphics processing engines 2031(1)-2031(N) may alternatively comprise different types of graphics processing engines within a GPU, such as graphics execution units, media processing engines (e.g., video encoders / decoders), samplers, and blit engines. In at least one embodiment, the graphics acceleration module 2046 may be a GPU with a plurality of graphics processing engines 2031(1)-2031(N), or the graphics processing engines 2031(1)-2031(N) may be individual GPUs integrated on a common package, line card, or die.

[0290] In at least one embodiment, accelerator integration circuitry 2036 includes a memory management unit (MMU) 2039 for performing various memory management functions, such as virtual-to-physical memory translations (also referred to as effective-to-real memory translations) and memory access protocols for accessing system memory 2014. In at least one embodiment, MMU 2039 may also include a translation lookaside buffer (TLB) (not shown) to cache virtual / effective to physical / real address translations. In at least one embodiment, a cache 2038 may store instructions and data for efficient access by graphics processing engines 2031(1)-2031(N).In at least one embodiment, the data stored in cache 2038 and graphics memories 2033(1)-2033(M) is kept coherent with core caches 2062A-2062D, 2056, and system memory 2014, possibly using a fetch unit 2044. As mentioned, this may be done via a proxy circuit 2025 on behalf of cache 2038 and memories 2033(1)-2033(M) (e.g., sending updates to cache 2038 regarding changes / accesses to cache lines in processor caches 2062A-2062D, 2056 and receiving updates from cache 2038).

[0291] In at least one embodiment, a set of registers 2045 stores context data for threads executed by graphics processing engines 2031(1)-2031(N), and context management circuitry 2048 manages thread contexts. For example, context management circuitry 2048 may perform save and restore operations to save and restore the contexts of various threads upon context switches (e.g., when a first thread is saved and a second thread is saved to allow a second thread to be executed by a graphics processing engine). For example, upon a context switch, context management circuitry 2048 may save the current register values ​​to a specific region of memory (e.g., identified by a context pointer). The register values ​​may then be restored upon return to a context.In at least one embodiment, an interrupt management circuit 2047 receives and processes interrupts received from system devices.

[0292] In at least one embodiment, virtual / effective addresses from a graphics processing engine 2031 are translated by the MMU 2039 into real / physical addresses in system memory 2014. In at least one embodiment, the accelerator integration circuit 2036 supports multiple (e.g., 4, 8, 16) graphics acceleration modules 2046 and / or other acceleration devices. In at least one embodiment, the graphics acceleration module 2046 may be dedicated to a single application executing on the processor 2007 or may be shared among multiple applications. In at least one embodiment, a virtualized graphics execution environment is illustrated in which the resources of the graphics processing engines 2031(1)-2031(N) are shared among multiple applications or virtual machines (VMs).In at least one embodiment, the resources may be divided into "slices" that are allocated to different VMs and / or applications based on processing requirements and priorities associated with VMs and / or applications.

[0293] In at least one embodiment, accelerator integration circuitry 2036 acts as a bridge to a system for graphics acceleration module 2046 and provides address translation and system memory caching services. Furthermore, in at least one embodiment, accelerator integration circuitry 2036 may provide virtualization facilities for a host processor to manage the virtualization of graphics processing engines 2031(1)-2031(N), interrupts, and memory management.

[0294] Because, in at least one embodiment, the hardware resources of graphics processing engines 2031(1)-2031(N) are explicitly mapped to a real address space seen by host processor 2007, each host processor can directly address these resources via an effective address value. In at least one embodiment, a function of accelerator integration circuitry 2036 is to physically separate graphics processing engines 2031(1)-2031(N) so that they appear to a system as independent entities.

[0295] In at least one embodiment, one or more graphics memories 2033(1)-2033(M) are coupled to each of the graphics processing engines 2031(1)-2031(N), where N=M. In at least one embodiment, the graphics memories 2033(1)-2033(M) store instructions and data processed by each of the graphics processing engines 2031(1)-2031(N). In at least one embodiment, the graphics memories 2033(1)-2033(M) may be volatile memories such as DRAMs (including stacked DRAMs), GDDR memories (e.g., GDDR5, GDDR6), or HBM, and / or they may be non-volatile memories such as 3D XPoint or Nano-Ram.

[0296] In at least one embodiment, to reduce data traffic over high-speed interconnect 2040, bias techniques may be used to ensure that the data stored in graphics memories 2033(1)-2033(M) is data most frequently used by graphics processing engines 2031(1)-2031(N) and preferably not used (at least not frequently) by cores 2060A-2060D. Similarly, in at least one embodiment, a bias mechanism attempts to keep data needed by cores (and preferably not used by graphics processing engines 2031(1)-2031(N)) in caches 2062A-2062D, 2056, and system memory 2014.

[0297] Fig. 20C shows another exemplary embodiment in which the accelerator integration circuit 2036 is integrated into the processor 2007. In this embodiment, the graphics processing engines 2031(1)-2031(N) communicate directly over the high-speed interconnect 2040 with the accelerator integration circuit 2036 via the interface 2037 and the interface 2035 (which may again be any form of bus or interface protocol). In at least one embodiment, the accelerator integration circuit 2036 may perform similar operations as described in Fig. 20B, but possibly with higher throughput due to its proximity to the coherence bus 2064 and caches 2062A-2062D, 2056. In at least one embodiment, an accelerator integration circuit supports different programming models, including a dedicated process programming model (without graphics acceleration module virtualization) and shared programming models (with virtualization), which may include programming models controlled by the accelerator integration circuit 2036 and programming models controlled by the graphics acceleration module 2046.

[0298] In at least one embodiment, the graphics processing engines 2031(1)-2031(N) are dedicated to a single application or process under a single operating system. In at least one embodiment, a single application can forward other application requests to the graphics processing engines 2031(1)-2031(N), thereby enabling virtualization within a VM / partition.

[0299] In at least one embodiment, the graphics processing engines 2031(1)-2031(N) may be shared between multiple VM / application partitions. In at least one embodiment, shared models may use a system hypervisor to virtualize the graphics processing engines 2031(1)-2031(N) to enable access by any operating system. In at least one embodiment, for systems with a partition without a hypervisor, the graphics processing engines 2031(1)-2031(N) are owned by an operating system. In at least one embodiment, an operating system may virtualize the graphics processing engines 2031(1)-2031(N) to enable access by any process or application.

[0300] In at least one embodiment, the graphics acceleration module 2046 or an individual graphics processing engine 2031(1)-2031(N) selects a process element using a process handle. In at least one embodiment, the process elements are stored in system memory 2014 and are addressable using an effective address to real address translation technique described herein. In at least one embodiment, a process handle may be an implementation-specific value provided to a host process when it registers its context with the graphics processing engine 2031(1)-2031(N) (i.e., when it calls system software to add a process element to a linked process element list). In at least one embodiment, the lower 16 bits of a process handle may be an offset of a process element within a linked process element list.

[0301] Fig. 20D shows an example accelerator integration slice 2090. In at least one embodiment, a "slice" comprises a particular portion of the processing resources of accelerator integration circuitry 2036. In at least one embodiment, an application stores process elements 2083 in the effective address space 2082 of system memory 2014. In at least one embodiment, process elements 2083 are stored in response to GPU calls 2081 from applications 2080 executing on processor 2007. In at least one embodiment, a process element 2083 contains process state for the corresponding application 2080. In at least one embodiment, a work descriptor (WD) 2084 contained in process element 2083 may be a single job requested by an application or may contain a pointer to a queue of jobs.In at least one embodiment, the WD 2084 is a pointer to a job request queue in the effective address space 2082 of an application.

[0302] In at least one embodiment, the graphics acceleration module 2046 and / or individual graphics processing engines 2031(1)-2031(N) may be shared by all or a subset of the processes in a system. In at least one embodiment, an infrastructure for establishing process states and sending a WD 2084 to a graphics acceleration module 2046 to start a job in a virtualized environment may be included.

[0303] In at least one embodiment, a programming model for dedicated processes is implementation-specific. In at least one embodiment, in this model, a single process owns the graphics acceleration module 2046 or an individual graphics processing engine 2031. In at least one embodiment, when the graphics acceleration module 2046 is owned by a single process, a hypervisor initializes the accelerator integration circuit 2036 for an owning partition, and an operating system initializes the accelerator integration circuit 2036 for an owning process when the graphics acceleration module 2046 is allocated.

[0304] In at least one embodiment, a WD fetch unit 2091 in accelerator integration slice 2090 operatively fetches the next WD 2084, which includes an indication of the work to be performed by one or more graphics processing engines of graphics acceleration module 2046. In at least one embodiment, the data from WD 2084 may be stored in registers 2045 and used by MMU 2039, interrupt management circuitry 2047, and / or context management circuitry 2048, as shown. For example, one embodiment of MMU 2039 includes segment / page browsing circuitry for accessing segment / page tables 2086 within an operating system virtual address space 2085. In at least one embodiment, circuitry 2047 may process interrupt events 2092 received from graphics acceleration module 2046.In at least one embodiment, when performing graphics operations, an effective address 2093 generated by a graphics processing engine 2031(1)-2031(N) is translated into a real address by the MMU 2039.

[0305] In at least one embodiment, registers 2045 are duplicated for each graphics processing engine 2031(1)-2031(N) and / or each graphics acceleration module 2046 and may be initialized by a hypervisor or an operating system. In at least one embodiment, each of these duplicated registers may comprise an accelerator integration slice 2090. Example registers that may be initialized by a hypervisor are shown in Table 1. Table 1 - Initialized hypervisor registers Register # Beschreibung 1 Slice-Steuerregister (Slice-Steuerregister) 2 Reale Adresse (RA) Zeiger auf den Bereich für geplante Prozesse 3 Authority mask override register 4 Interrupt vector table entry offset 5 Interrupt vector table entry boundary 6 Condition register 7 Logical partition ID 8 Real Address (RA) pointer to Hypervisor Accelerator Utilization Record 9 Memory description register

[0306] Example registers that can be initialized by an operating system are shown in Table 2. Table 2 - Initialized operating system registers Register # Description 1 Process and thread identification 2 Effective Address (EA) Context Store / Restore Pointer 3 Virtual Address (VA) pointer for the accelerator utilization set 4 Virtual Address (VA) Pointer to the memory segment table 5 Authority mask 6 Job description

[0307] In at least one embodiment, each WD 2084 is specific to a particular graphics acceleration module 2046 and / or graphics processing engines 2031(1)-2031(N). In at least one embodiment, it contains all the information needed by a graphics processing engine 2031(1)-2031(N) to perform work, or it may be a pointer to a memory location where an application has established a command queue of work to be performed.

[0308] Fig.20E shows additional details for an exemplary embodiment of a joint model. This embodiment includes a real hypervisor address space 2098 in which a process element list 2099 is stored. In at least one embodiment, the real hypervisor address space 2098 is accessible via a hypervisor 2096 that virtualizes graphics acceleration engine engines for the operating system 2095.

[0309] In at least one embodiment, shared programming models allow all or a subset of processes from all or a subset of partitions in a system to use a graphics acceleration module 2046. In at least one embodiment, there are two programming models in which the graphics acceleration module 2046 is shared among multiple processes and partitions: time-sliced ​​sharing and graphics-directed sharing.

[0310] In this model, in at least one embodiment, the system hypervisor 2096 owns the graphics acceleration module 2046 and provides its functionality to all operating systems 2095. In at least one embodiment, a graphics acceleration module 2046 may meet certain requirements to support virtualization by the system hypervisor 2096, such as (1) the job request of an application must be autonomous (i.e.the state does not need to be maintained between jobs), or the graphics acceleration module 2046 must provide a mechanism for saving and restoring the context, (2) the graphics acceleration module 2046 guarantees that an application's job request will be completed in a specified amount of time, including any translation errors, or the graphics acceleration module 2046 provides the ability to preempt the processing of a job, and (3) the graphics acceleration module 2046 must be guaranteed fairness between processes when operating in a directed joint programming model.

[0311] In at least one embodiment, application 2080 must make a system call to operating system 2095 with a graphics acceleration module type, a work description (WD), an authority mask register (AMR) value, and a context save / restore pointer (CSRP). In at least one embodiment, the graphics acceleration module type describes a targeted acceleration function for a system call. In at least one embodiment, the graphics acceleration module type may be a system-specific value. In at least one embodiment, WD is formatted specifically for graphics acceleration module 2046 and may be in the form of a graphics acceleration module 2046 instruction, a pointer to the effective address of a user-defined structure, a pointer to the effective address of an instruction queue, or another data structure describing the work to be performed by graphics acceleration module 2046.

[0312] In at least one embodiment, an AMR value is an AMR state to be used for a current process. In at least one embodiment, a value passed to an operating system is similar to an application setting an AMR. In at least one embodiment, if accelerator integration circuitry 2036 (not shown) and graphics acceleration module 2046 do not support an authority mask override register (UAMOR), an operating system may apply a current UAMOR value to an AMR value before passing an AMR in a hypervisor call. In at least one embodiment, hypervisor 2096 may optionally apply a current authority mask override register (AMOR) value before placing an AMR into process element 2083.In at least one embodiment, CSRP is one of the registers 2045 that contains an effective address of a region in an application's effective address space 2082 for the graphics acceleration module 2046 to save and restore context state. In at least one embodiment, this pointer is optional if no state needs to be saved between jobs or if a job terminates prematurely. In at least one embodiment, the context save / restore region may be located in system memory.

[0313] Upon receiving a system call, the operating system 2095 may verify whether the application 2080 has and has been granted permission to use the graphics acceleration module 2046. In at least one embodiment, the operating system 2095 then invokes the hypervisor 2096 with the information listed in Table 3. Table 3 - Parameters for calling the operating system to the hypervisor Parameters # Description 1 A job description (WD) 2 An Authority Mask Register (AMR) value (possibly masked) 3 An effective address (EA) Context save / restore pointer (CSRP) 4 A process ID (PID) and optionally a thread ID (TID) 5 A virtual address (VA) accelerator utilization set pointer (AURP) 6 Virtual address of the pointer to the memory segment table (SSTP) 7 A logical interrupt service number (LISN)

[0314] In at least one embodiment, upon receiving a hypervisor call, the hypervisor 2096 checks whether the operating system 2095 has and receives permission to use the graphics acceleration module 2046. In at least one embodiment, the hypervisor 2096 then places the process element 2083 in a process element list for a corresponding type of graphics acceleration module 2046. In at least one embodiment, a process element may include the information shown in Table 4. Table 4 - Process element information Item # Description 1 A job description (WD) 2 An Authority Mask Register (AMR) value (possibly masked). 3 An effective address (EA) Context save / restore pointer (CSRP) 4 A process ID (PID) and optionally a thread ID (TID) 5 A virtual address (VA) accelerator utilization set pointer (AURP) 6 Virtual address of the pointer to the memory segment table (SSTP) 7 A logical interrupt service number (LISN) 8 Interrupt vector table derived from hypervisor call parameters 9 A status register value (SR) 10 A logical partition ID (LPID) 11 A pointer to the hypervisor's accelerator utilization set with real address (RA) 12 Memory Description Register (SDR)

[0315] In at least one embodiment, the hypervisor initializes a plurality of registers 2045 for accelerator integration slices 2090.

[0316] As in Fig.20F, in at least one embodiment, a unified memory addressable via a common virtual memory address space is used to access physical processor memories 2001(1)-2001(N) and GPU memories 2020(1)-2020(N). In this implementation, operations performed on GPUs 2010(1)-2010(N) use the same virtual / effective address space to access processor memories 2001(1)-2001(M) and vice versa, which simplifies programmability. In at least one embodiment, a first portion of a virtual / effective address space is assigned to processor memory 2001(1), a second portion is assigned to second processor memory 2001(N), a third portion is assigned to GPU memory 2020(1), and so on.In at least one embodiment, this distributes an entire virtual / effective memory space (sometimes referred to as effective address space) across each of the processor memories 2001 and the GPU memories 2020, such that each processor or GPU can access each physical memory with a virtual address associated with that memory.

[0317] In at least one embodiment, the bias / coherence management circuitry 2094A-2094E within one or more MMUs 2039A-2039E ensures cache coherence between the caches of one or more host processors (e.g., 2005) and GPUs 2010 and implements bias techniques that indicate in which physical memories certain data types should be stored. In at least one embodiment, while multiple instances of the bias / coherence management circuitry 2094A-2094E in Fig.20F, the bias / coherence circuit may be implemented within an MMU of one or more host processors 2005 and / or within the accelerator integration circuit 2036.

[0318] In one embodiment, GPU memory 2020 may be mapped as part of system memory and accessed using shared virtual memory (SVM) technology, but without incurring the performance penalties associated with full system cache coherence. In at least one embodiment, the ability to access GPU memory 2020 as system memory without burdensome cache coherence overhead provides a favorable operating environment for GPU offload. In at least one embodiment, this arrangement allows the software of host processor 2005 to set operands and access computation results without the overhead of traditional I / O DMA data copies. In at least one embodiment, such traditional copies involve driver calls, interrupts, and memory-mapped I / O (MMIO) accesses, all of which are inefficient compared to simple memory accesses.In at least one embodiment, the ability to access GPU memory 2020 without cache coherence overheads may be critical to the execution time of an offloaded computation. For example, in at least one embodiment, cache coherence overhead may significantly reduce the effective write bandwidth of a GPU 2010 in cases with significant streaming write memory traffic. In at least one embodiment, operand construction efficiency, result access efficiency, and GPU computation efficiency may play a role in determining the effectiveness of a GPU offload.

[0319] In at least one embodiment, the selection of GPU bias and host processor bias is controlled by a bias tracker data structure. For example, in at least one embodiment, a bias table may be used, which may be a page-granular structure (e.g., controlled at the granularity of a memory page) comprising 1 or 2 bits per GPU-attached memory page. In at least one embodiment, a bias table may be implemented in a stolen memory region of one or more GPU memories 2020, with or without a bias cache in a GPU 2010 (e.g., to cache frequently / recently used entries of a bias table). Alternatively, in at least one embodiment, an entire bias table may be maintained in a GPU.

[0320] In at least one embodiment, a bias table entry associated with each access to GPU-attached memory 2020 is accessed prior to the actual GPU memory access, causing the following operations. In at least one embodiment, local requests from a GPU 2010 that find their page in the GPU bias are forwarded directly to a corresponding GPU memory 2020. In at least one embodiment, local requests from a GPU that find their page in the host processor's bias are forwarded to the processor 2005 (e.g., over a high-speed connection, as described herein). In at least one embodiment, requests from the processor 2005 that find a requested page in the host processor's bias complete a request like a normal memory read.Alternatively, requests for a GPU-biased page may be forwarded to a GPU 2010. In at least one embodiment, a GPU may then transition a page to a host processor bias if it is not currently using the page. In at least one embodiment, the bias state of a page may be changed either by a software-based mechanism, a hardware-assisted software-based mechanism, or, for a limited number of cases, by a purely hardware-based mechanism.

[0321] In at least one embodiment, a mechanism for changing bias state uses an API call (e.g., OpenCL), which in turn calls a graphics processor's device driver, which in turn sends a message to a graphics processor (or queues a command descriptor) to instruct it to change a bias state and, on some transitions, perform a cache flush operation in a host. In at least one embodiment, a cache flush operation is used for a transition from host processor 2005 bias to GPU bias, but not for an opposite transition.

[0322] In at least one embodiment, cache coherence is maintained by temporarily making GPU-biased pages uncacheable by host processor 2005. In at least one embodiment, to access these pages, processor 2005 may request access from GPU 2010, which may or may not grant access immediately. Therefore, in at least one embodiment, to reduce communication between processor 2005 and GPU 2010, it is advantageous to ensure that GPU-biased pages are those required by a GPU but not by host processor 2005, and vice versa.

[0323] Hardware structure(s) 1215 are used to perform one or more embodiments. Details of a hardware structure (or more hardware structures) 1215 may be described herein in connection with Fig. 12A and / or 12B must be specified.

[0324] Fig.Figure 21 illustrates exemplary integrated circuits and associated graphics processors that may be fabricated using one or more IP cores according to various embodiments described herein. In addition to the illustrated embodiments, other logic and circuitry may also be included in at least one embodiment, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0325] Fig.21 is a block diagram illustrating an exemplary integrated circuit 2100 that may be fabricated using one or more IP cores according to at least one embodiment. In at least one embodiment, the integrated circuit 2100 includes one or more application processors 2105 (e.g., CPUs), at least one graphics processor 2110, and may additionally include an image processor 2115 and / or a video processor 2120, each of which may be a modular IP core. In at least one embodiment, the integrated circuit 2100 includes peripheral or bus logic, including a USB controller 2125, a UART controller 2130, an SPI / SDIO controller 2135, and an I22S / I22C controller 2140.In at least one embodiment, integrated circuit 2100 may include a display device 2145 coupled to one or more of the following interfaces: a High-Definition Multimedia Interface (HDMI) controller 2150 and a Mobile Industry Processor Interface (MIPI) interface 2155. In at least one embodiment, memory may be provided by a flash memory subsystem 2160 comprising flash memory and a flash memory controller. In at least one embodiment, a memory interface may be provided via a memory controller 2165 for accessing SDRAM or SRAM memory devices. In at least one embodiment, some integrated circuits additionally include an embedded security engine 2170.

[0326] Logic 1215 is used to perform inference and / or training operations in connection with one or more embodiments. Details of logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in integrated circuit 2100 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0327] In at least one embodiment, components of Fig. 21 or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig.21 one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating different content of at least one of the two or more different images than the one or more objects. In at least one embodiment, the components of Fig. 21 one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0328] Fig.22A-22B illustrate exemplary integrated circuits and associated graphics processors that may be fabricated using one or more IP cores according to various embodiments described herein. In addition to the embodiments shown, other logic and circuitry may be included in at least one embodiment, including additional graphics processors / cores, peripheral interface controllers, or general-purpose processor cores.

[0329] Fig. 22A-22B are block diagrams illustrating example graphics processors for use in an SoC according to the embodiments described herein. Fig. 22A shows an exemplary integrated circuit graphics processor 2210 for a system on a chip that may be manufactured using one or more IP cores in accordance with at least one embodiment. Fig.22B shows another exemplary integrated circuit graphics processor 2240 that may be fabricated with one or more IP cores in accordance with at least one embodiment. In at least one embodiment, the graphics processor 2210 is Fig. 22A is a low-power graphics processor core. In at least one embodiment, the graphics processor 2240 is Fig. 22B, a higher performance graphics processor core. In at least one embodiment, each of the graphics processors 2210, 2240 may be a variant of the graphics processor 2110 of Fig. be 21.

[0330] In at least one embodiment, graphics processor 2210 includes a vertex processor 2205 and one or more fragment processors 2215A-2215N (e.g., 2215A, 2215B, 2215C, 2215D, through 2215N-1, and 2215N). In at least one embodiment, graphics processor 2210 may execute different shader programs via separate logic, such that vertex processor 2205 is optimized to perform operations for vertex shader programs, while one or more fragment processors 2215A-2215N perform fragment (e.g., pixel) shading operations for fragment or pixel shader programs. In at least one embodiment, the vertex processor 2205 executes a vertex processing stage of a 3D graphics pipeline and generates primitives and vertex data.In at least one embodiment, fragment processor(s) 2215A-2215N use the primitive and vertex data generated by vertex processor 2205 to generate a framebuffer displayed on a display device. In at least one embodiment, fragment processor(s) 2215A-2215N are optimized for executing fragment shader programs as provided in an OpenGL API, which can be used to perform similar operations as a pixel shader program as provided in a Direct 3D API.

[0331] In at least one embodiment, graphics processor 2210 additionally includes one or more memory management units (MMUs) 2220A-2220B, cache(s) 2225A-2225B, and circuit interconnect(s) 2230A-2230B. In at least one embodiment, one or more MMU(s) 2220A-2220B provide virtual to physical address mapping for graphics processor 2210, including vertex processor 2205 and / or fragment processor(s) 2215A-2215N, which may reference vertex or image / texture data stored in memory, in addition to the vertex or image / texture data stored in one or more cache(s) 2225A-2225B. In at least one embodiment, one or more MMU(s) 2220A-2220B may be synchronized with other MMUs within a system, including one or more MMUs associated with one or more application processor(s) 2105, image processor(s) 2115, and / or video processor(s) 2120 of Fig.21, so that each processor 2105-2120 can participate in a shared or unified virtual memory system. In at least one embodiment, one or more circuit interconnects 2230A-2230B enable the graphics processor 2210 to interface with other IP cores within the SoC, either via an internal bus of the SoC or via a direct connection.

[0332] In at least one embodiment, the graphics processor 2240 includes one or more shader cores 2255A-2255N (e.g., 2255A, 2255B, 2255C, 2255D, 2255E, 2255F through 2255N-1 and 2255N), as shown in Fig.22B, which provides a unified shader core architecture in which a single core or type of core can execute all types of programmable shader code, including shader program code for implementing vertex shaders, fragment shaders, and / or compute shaders. In at least one embodiment, the number of shader cores can vary. In at least one embodiment, the graphics processor 2240 includes an inter-core task manager 2245 acting as a thread dispatcher to distribute execution threads to one or more shader cores 2255A-2255N, and a tiling unit 2258 to accelerate tiling operations for tile-based rendering, in which rendering operations for a scene are partitioned into image space, for example, to exploit local spatial coherence within a scene or to optimize the use of internal caches.

[0333] Logic 1215 is used to perform inference and / or training operations in connection with one or more embodiments. Details of logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in graphics processor 2210 and / or 2240 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0334] In at least one embodiment, components of the Fig. 22A-22B or with components, processes and / or a combination thereof from the Fig. 1-11 to generate output images. In at least one embodiment, the components of the Fig.22A-22B one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig. 22A-22B one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0335] Fig.23A-23B illustrate additional exemplary graphics processor logic according to the embodiments described herein. In at least one embodiment, the Fig. 23A-23B are integrated into a single system, such as a graphics processing unit (GPU), an SoC, or other type of processor. Fig. 23A shows a graphics core 2300 that, in at least one embodiment, includes the graphics processor 2110 of Fig. 21 and in at least one embodiment, a unified shader core 2255A-2255N as in Fig. 22B can be. Fig.23B illustrates a highly parallel general-purpose graphics processing unit ("GPGPU," which may also be referred to as a "graphics processing unit") 2330, which in at least one embodiment is suitable for use on a multi-chip module. In at least one embodiment, the graphics processing unit 2330 is a GPGPU that includes a graphics processor. In at least one embodiment, the integrated circuit 2100 includes the graphics core 2300, for example, to form an integrated circuit and / or an SoC, such an integrated circuit and / or SoC performing the operations described herein.

[0336] In at least one embodiment, the graphics core 2300 includes a shared instruction cache 2302, a texture unit 2318, and a cache / shared memory 2320 (e.g., L1, L2, L3, last level cache, or other caches) common to the execution resources within the graphics core 2300. In at least one embodiment, the graphics core 2300 may include multiple slices 2301A-2301N or a partition for each core, and a graphics processor may include multiple instances of the graphics core 2300. In at least one embodiment, each slice 2301A-2301N relates to the graphics core 2300. In at least one embodiment, the slices 2301A-2301N include sub-slices that are part of a slice 2301A-2301N. In at least one embodiment, slices 2301A-2301N are independent of other slices or dependent on other slices.In at least one embodiment, slices 2301A-2301N may include support logic including a local instruction cache 2304A-2304N, a thread scheduler (sequencer) 2306A-2306N, a thread dispatcher 2308A-2308N, and a set of registers 2310A-2310N. In at least one embodiment, slices 2301A-2301N may include a set of additional functional units (AFUs 2312A-2312N), floating-point units (FPUs 2314A-2314N), integer arithmetic logic units (ALUs 2316A-2316N), address calculation units (ACUs 2313A-2313N), double-precision floating-point units (DPFPUs 2315A-2315N), and matrix processing units (MPUs 2317A-2317N). In at least one embodiment, MPUs 2317A-2317N are referred to as matrix engines.

[0337] In at least one embodiment, each slice 2301A-2301N includes one or more engines for floating-point and integer vector operations and one or more engines for accelerating convolution and matrix operations in AI, machine learning, or large datasets. In at least one embodiment, one or more slices 2301A-2301N include one or more vector engines for computing a vector (e.g., computing mathematical operations on vectors). In at least one embodiment, a vector engine can compute a vector operation in 16-bit floating point (also referred to as "FP16"), 32-bit floating point (also referred to as "FP32"), or 64-bit floating point (also referred to as "FP64").In at least one embodiment, one or more slices 2301A-2301N comprise 16 vector engines paired with 16 matrix math units to compute matrix / tensor operations, where the vector engines and the math units are accessible via matrix extensions. In at least one embodiment, a slice comprises a particular portion of the processing resources of a processing unit, for example, 16 cores and a ray tracing unit or 8 cores, a thread scheduler, a thread dispatcher, and additional functional units for a processor. In at least one embodiment, the graphics core 2300 comprises one or more matrix engines for computing matrix operations, for example, when computing tensor operations.

[0338] In at least one embodiment, one or more slices 2301A-2301N include one or more ray tracing units for computing ray tracing operations (e.g., 16 ray tracing units per slice 2301A-2301N). In at least one embodiment, a ray tracing unit computes ray crossings, triangle intersections, bounding box intersections, or other ray tracing operations.

[0339] In at least one embodiment, one or more slices 2301A-2301N comprise a media slice that encodes, decodes, and / or transcodes data, scales and / or formats data, and / or performs video quality operations on video data.

[0340] In at least one embodiment, one or more slices 2301A-2301N are connected to L2 cache and memory structure, interconnect ports, HBM stacks (e.g., HBM2e, HDM3), and a media engine. In at least one embodiment, one or more slices 2301A-2301N include multiple cores (e.g., 16 cores) and multiple ray tracing units (e.g., 16) coupled to each core. In at least one embodiment, one or more slices 2301A-2301N include one or more L1 caches. In at least one embodiment, one or more slices 2301A-2301N include one or more vector engines; one or more instruction caches for storing instructions; one or more L1 caches for caching data; one or more shared local memories (SLMs) for storing data, e.g., corresponding to instructions; one or more samplers for sampling data;one or more ray tracing units for performing ray tracing operations; one or more geometries for performing operations in geometry pipelines and / or applying geometric transformations to vertices or polygons; one or more rasterizers for describing an image in a vector graphics format (e.g., shape) and converting it into a raster image (e.g., a series of pixels, points, or lines that, when displayed, combine to form an image represented by shapes); one or more hierarchical depth buffers (Hiz) for buffering data; and / or one or more pixel backends. In at least one embodiment, a slice 2301A-2301N includes a memory structure, such as an L2 cache.;

[0341] In at least one embodiment, the FPUs 2314A-2314N can perform single-precision (32-bit) and half-precision (16-bit) floating-point operations, while the DPFPUs 2315A-2315N can perform double-precision (64-bit) floating-point operations. In at least one embodiment, the ALUs 2316A-2316N can perform variable-precision integer operations at 8-bit, 16-bit, and 32-bit precision and can be configured for mixed-precision operations. In at least one embodiment, the MPUs 2317A-2317N can also be configured for mixed-precision matrix operations, including half-precision floating-point and 8-bit integer operations. In at least one embodiment, MPUs 2317-2317N may perform a variety of matrix operations to accelerate machine learning application frameworks, including support for accelerated generalized matrix-matrix multiplication (GEMM).In at least one embodiment, AFUs 2312A-2312N may perform additional logical operations not supported by floating point or integer units, including trigonometric operations (e.g., sine, cosine, etc.).

[0342] Logic 1215 is used to perform inference and / or training operations in connection with one or more embodiments. Details of logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in graphics core 2300 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0343] In at least one embodiment, the graphics core 2300 includes an interconnect and a link fabric sublayer connected to a switch and a GPU-GPU bridge that enables multiple graphics processors 2300 (e.g., 8) to be interconnected without glue, with load / store units (LSUs), data transfer units, and synchronization semantics across multiple graphics processors 2300. In at least one embodiment, the interconnects include standardized interconnects (e.g., PCIe) or a combination thereof.

[0344] In at least one embodiment, the graphics core 2300 includes multiple tiles. In at least one embodiment, a tile is a single die or one or more dies, where individual dies may be connected to an interconnect (e.g., Embedded Multi-Die Interconnect Bridge (EMIB)). In at least one embodiment, the graphics core 2300 includes a compute tile, a memory tile (e.g., where a memory tile can be accessed exclusively by different tiles or different chipsets, such as a Rambo tile), a substrate tile, a base tile, an HMB tile, a link tile, and an EMIB tile, all of which tiles are packaged together in the graphics core 2300 as part of a GPU. In at least one embodiment, the graphics core 2300 may include multiple tiles in a single package (also referred to as a "multi-tile package").In at least one embodiment, a compute tile may include 8 graphics cores 2300, an L1 cache, and a base tile; a host interface with PCIe 5.0, HBM2e, MDFI, and EMIB; and a link tile with 8 links, 8 ports, and an embedded switch. In at least one embodiment, the tiles are connected using face-to-face (F2F) chip-on-chip bonding via finely pitched 36-micrometer microbumps (e.g., copper pillars). In at least one embodiment, the graphics core 2300 includes a memory structure including memory and a tile accessible by multiple tiles. In at least one embodiment, the graphics core 2300 stores, accesses, or loads its own hardware contexts in memory. A hardware context is a set of data loaded from registers prior to resuming a process, and a hardware context may indicate a state of the hardware (e.g., the state of a GPU).

[0345] In at least one embodiment, the graphics core 2300 includes a serialization / deserialization circuit (SERDES) that converts a serial data stream to a parallel data stream or a parallel data stream to a serial data stream.

[0346] In at least one embodiment, the graphics core 2300 includes a coherent high-speed unified fabric (GPU to GPU), load / store units, bulk data transfer and sync semantics, and GPUs connected via an embedded switch, with a GPU-GPU bridge controlled by a controller.

[0347] In at least one embodiment, the graphics core 2300 executes an API, where the API abstracts the hardware of the graphics core 2300 and accesses libraries of instructions for performing mathematical operations (e.g., math core library), deep neural network operations (e.g., deep neural network library), vector operations, collective communication, threading building blocks, video processing, data analysis library, and / or ray tracing operations.

[0348] In at least one embodiment, components of the Fig. 22A-22B or with components, processes and / or a combination thereof from the Fig. 1-11 to generate output images. In at least one embodiment, the components of the Fig.22A-22B one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig. 22A-22B one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0349] Fig.Figure 23B illustrates GPGPU 2330, which may be configured to enable highly parallel computational operations by an array of graphics processing units, in at least one embodiment. In at least one embodiment, GPGPU 2330 may be directly connected to other instances of GPGPU 2330 to form a multi-GPU cluster and improve training speed for deep neural networks. In at least one embodiment, GPGPU 2330 includes a host interface 2332 to enable connection to a host processor. In at least one embodiment, host interface 2332 is a PCI Express interface. In at least one embodiment, host interface 2332 may be a vendor-specific interface or communication fabric.In at least one embodiment, the GPGPU 2330 receives instructions from a host processor and uses a global scheduler 2334 (which may be referred to as a thread sequencer and / or asynchronous compute engine) to distribute the execution threads associated with those instructions among a number of compute clusters 2336A-2336H. In at least one embodiment, the compute clusters 2336A-2336H share a cache 2338. In at least one embodiment, the cache 2338 may serve as a parent cache for caches within compute clusters 2336A-2336H. In at least one embodiment, the compute clusters 2336A-2336H comprise a slice or are referred to as "slices." In at least one embodiment, the GPGPU 2330 is part of an SoC, such as part of the integrated circuit 2100 (. Fig. 21).

[0350] In at least one embodiment, GPGPU 2330 includes memory 2344A-2344B coupled to compute clusters 2336A-2336H via a series of memory controllers 2342A-2342B (e.g., one or more controllers for HBM2e). In at least one embodiment, memory 2344A-2344B may include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate memory (GDDR).

[0351] In at least one embodiment, the compute clusters 2336A-2336H each comprise a set of graphics cores, such as the graphics core 2300 of Fig.23A, which may include multiple types of integer and floating-point logic units capable of performing computational operations at a range of precisions, including those suitable for machine learning computations. For example, in at least one embodiment, at least a subset of floating-point units in each of compute clusters 2336A-2336H may be configured to perform 16-bit or 32-bit floating-point operations, while a different subset of floating-point units may be configured to perform 64-bit floating-point operations.

[0352] In at least one embodiment, multiple instances of GPGPU 2330 can be configured to operate as a compute cluster. In at least one embodiment, the communication used by compute clusters 2336A-2336H for synchronization and data exchange varies between embodiments. In at least one embodiment, multiple instances of GPGPU 2330 communicate via host interface 2332. In at least one embodiment, GPGPU 2330 includes an I / O hub 2339 that couples GPGPU 2330 to a GPU interconnect 2340 that enables direct connection to other instances of GPGPU 2330. In at least one embodiment, GPU interconnect 2340 is connected to a dedicated GPU-to-GPU bridge that enables communication and synchronization between multiple instances of GPGPU 2330.In at least one embodiment, GPU interconnect 2340 is coupled to a high-speed interconnect for sending and receiving data to other GPGPUs or parallel processors. In at least one embodiment, multiple GPGPU instances 2330 are located in separate computing systems and communicate via a network interface accessible via host interface 2332. In at least one embodiment, GPU interconnect 2340 may be configured to enable connection to a host processor in addition to, or alternatively to, host interface 2332.

[0353] In at least one embodiment, GPGPU 2330 may be configured to train neural networks. In at least one embodiment, GPGPU 2330 may be used within an inferencing platform. In at least one embodiment where GPGPU 2330 is used for inferencing, GPGPU 2330 may include fewer compute clusters 2336A-2336H than when GPGPU 2330 is used for training a neural network. In at least one embodiment, the memory technology associated with memory 2344A-2344B may differ between inference and training configurations, with higher bandwidth memory technologies being allocated to training configurations. In at least one embodiment, an inferencing configuration of GPGPU 2330 may support inferencing-specific instructions.For example, in at least one embodiment, an inference configuration may provide support for one or more 8-bit integer dot product instructions that may be used during inference operations for deployed neural networks.

[0354] Logic 1215 is used to perform inference and / or training operations in connection with one or more embodiments. Details of logic 1215 are described herein in connection with Fig. 12A and / or 12B. In at least one embodiment, logic 1215 in GPGPU 2330 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0355] In at least one embodiment, components of the Fig. 23A-23B or with components, processes and / or a combination thereof from the Fig. 1-11 to generate output images. In at least one embodiment, the components of the Fig. 23A-23B one or more circuits for using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig.23A-23B one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts.

[0356] Fig.24 is a block diagram illustrating a computer system 2400 according to at least one embodiment. In at least one embodiment, the computer system 2400 includes a processing subsystem 2401 having one or more processors 2402 and a system memory 2404 communicating via an interconnect path that may include a memory hub 2405. In at least one embodiment, the memory hub 2405 may be a separate component within a chipset component or integrated with one or more processors 2402. In at least one embodiment, the memory hub 2405 is connected to an I / O subsystem 2411 via a communications link 2406. In at least one embodiment, the I / O subsystem 2411 includes an I / O hub 2407 that may enable the computer system 2400 to receive input from one or more input devices 2408.In at least one embodiment, the I / O hub 2407 may enable a display controller, which may include one or more processors 2402, to provide outputs to one or more display devices 2410A. In at least one embodiment, one or more display devices 2410A coupled to the I / O hub 2407 may include a local, internal, or embedded display device.

[0357] In at least one embodiment, the processing subsystem 2401 includes one or more parallel processors 2412 connected to the storage hub 2405 via a bus or other communication link 2413. In at least one embodiment, the communication link 2413 may use any number of standards-based communication link technologies or protocols, such as, but not limited to, PCI Express, or may be a vendor-specific communication interface or communication structure. In at least one embodiment, one or more parallel processors 2412 form a compute-centric parallel or vector processing system, which may include a large number of compute cores and / or processing clusters, such as a Many Integrated Core (MIC) processor.In at least one embodiment, some or all of the parallel processors 2412 form a graphics processing subsystem that can output pixels to one or more display devices 2410A coupled via the I / O hub 2407. In at least one embodiment, the parallel processor(s) 2412 may also include a display controller and a display interface (not shown) to enable direct connection to one or more display devices 2410B. In at least one embodiment, the parallel processor(s) 2412 include one or more cores, such as the graphics cores 2300 discussed herein.

[0358] In at least one embodiment, a system storage unit 2414 may be coupled to the I / O hub 2407 to provide a storage mechanism for the computer system 2400. In at least one embodiment, an I / O switch 2416 may be used to provide an interface enabling connections between the I / O hub 2407 and other components, such as a network adapter 2418 and / or a wireless network adapter 2419 that may be integrated into the platform, and various other devices that may be added via one or more add-in devices 2420. In at least one embodiment, the network adapter 2418 may be an Ethernet adapter or other wired network adapter.In at least one embodiment, the wireless network adapter 2419 may include one or more of the following devices: Wi-Fi, Bluetooth, Near Field Communication (NFC), or another network device that includes one or more wireless radios.

[0359] In at least one embodiment, computer system 2400 may include other components not explicitly shown, including USB or other connectors, optical storage drives, video capture devices, and the like, which may also be connected to I / O hub 2407. In at least one embodiment, communication paths connecting various components in Fig.24 interconnection, be implemented using any suitable protocols, such as PCI (Peripheral Component Interconnect) based protocols (e.g. PCI Express) or other bus or point-to-point communication interfaces and / or protocols, such as NV-Link high-speed interconnect or interconnection protocols.

[0360] In at least one embodiment, the parallel processor(s) 2412 include circuitry optimized for graphics and video processing, such as video output circuitry, and form a graphics processing unit (GPU). For example, the parallel processor(s) 2412 includes the graphics core 2300. In at least one embodiment, the parallel processor(s) 2412 include circuitry optimized for general processing. In at least one embodiment, the components of the computer system 2400 may be integrated with one or more other system elements on a single integrated circuit. For example, in at least one embodiment, the parallel processor(s) 2412, the memory hub 2405, the processor(s) 2402, and the I / O hub 2407 may be integrated into a system-on-chip (SoC) integrated circuit.In at least one embodiment, the components of computer system 2400 may be integrated into a single package to form a system-in-package (SIP) configuration. In at least one embodiment, at least a portion of the components of computer system 2400 may be integrated into a multi-chip module (MCM), which may be interconnected with other multi-chip modules to form a modular computer system.

[0361] Logic 1215 is used to perform inference and / or training operations associated with one or more embodiments. Details of logic 1215 are described herein in connection with Fig.12A and / or 12B. In at least one embodiment, logic 1215 in computer system 2400 may be used for inference or prediction operations based at least in part on weighting parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0362] In at least one embodiment, components of Fig. 24 or with components, processes and / or a combination thereof from Fig. 1-11 to generate output images. In at least one embodiment, the components of Fig.24 one or more circuits to use one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating the content of at least one of the two or more different images that is not the one or more objects. In at least one embodiment, the components of Fig. 24 one or more circuits for using one or more neural networks to generate two or more images depicting a same subject in different settings based at least in part on two or more different input text prompts. PROCESSORS

[0363] Fig.25A shows a parallel processor 2500 according to at least one embodiment. In at least one embodiment, various components of the parallel processor 2500 may be implemented using one or more integrated circuits, such as programmable processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In at least one embodiment, the parallel processor 2500 shown is a variant of one or more parallel processors 2412 described in Fig. 24 according to an exemplary embodiment. In at least one embodiment, a parallel processor 2500 includes one or more graphics cores 2300.

[0364] In at least one embodiment, parallel processor 2500 includes a parallel processing unit 2502. In at least one embodiment, parallel processing unit 2502 includes an I / O unit 2504 that enables communication with other devices, including other instances of parallel processing unit 2502. In at least one embodiment, I / O unit 2504 may be directly connected to other devices. In at least one embodiment, I / O unit 2504 is connected to other devices via a hub or switch interface, such as a storage hub 2505. In at least one embodiment, the connections between storage hub 2505 and I / O unit 2504 form a communication link 2513.In at least one embodiment, the I / O unit 2504 is coupled to a host interface 2506 and a memory crossbar 2516, where the host interface 2506 receives commands to perform processing operations and the memory crossbar 2516 receives commands to perform memory operations.

[0365] In at least one embodiment, when the host interface 2506 receives a command buffer via the I / O unit 2504, the host interface 2506 may forward work operations to a front end 2508 for execution of those commands. In at least one embodiment, the front end 2508 is coupled to a scheduler 2510 (which may also be referred to as a sequencer) configured to dispatch commands or other work items to a processing cluster array 2512. In at least one embodiment, the scheduler 2510 ensures that the processing array 2512 is properly configured and in a valid state before dispatching tasks to a cluster of the processing array 2512. In at least one embodiment, the scheduler 2510 is implemented via firmware logic executing on a microcontroller.In at least one embodiment, the microcontroller-implemented scheduler 2510 is configurable to perform complex scheduling and work distribution operations at coarse and fine granularity, enabling fast preemption and context switching of threads executing on the processing array 2512. In at least one embodiment, host software may allocate workloads for scheduling on the processing cluster array 2512 via one of several graphics processing paths. In at least one embodiment, the workloads may then be automatically distributed across the processing array cluster 2512 by the logic of the scheduler 2510 within a microcontroller that includes the scheduler 2510.

[0366] In at least one embodiment, processing array 2512 may include up to "N" processing clusters (e.g., cluster 2514A, cluster 2514B, through cluster 2514N), where "N" represents a positive integer (which may be a different integer "N" than used in other figures). In at least one embodiment, each cluster 2514A-2514N of processing array 2512 may execute a large number of concurrent threads. In at least one embodiment, scheduler 2510 may allocate work to clusters 2514A-2514N of processing array 2512 using various scheduling and / or work distribution algorithms, which m...

Claims

[1] A processor comprising: one or more circuits to use one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating content of at least one of the two or more different images different from the one or more objects. [2] The processor of claim 1, wherein the one or more objects in the two or more different images comprise a same object. [3] The processor of claim 1, wherein the one or more neural networks comprise one or more layers that denoise images to generate the two or more different images. [4] The processor of claim 1, wherein the one or more neural networks comprise a diffusion neural network. [5] The processor of claim 1, wherein the one or more indications comprise one or more input text prompts, the text prompts being generated by one or more users. [6] The processor of claim 1, wherein the one or more neural networks receive only text prompts as inputs. [7] The processor of claim 1, wherein the one or more objects are the same objects that are in different poses in the two or more different images. [8] A system comprising: one or more processors to use one or more neural networks to generate one or more objects within two or more different images based, at least in part, on one or more indications from one or more users indicating different content of at least one of the two or more different images than the one or more objects. [9] The system of claim 8, wherein the one or more objects in the two or more different images comprise the same item. [10] The system of claim 8, wherein the one or more neural networks comprise one or more layers that denoise the two or more different images. [11] The system of claim 8, wherein the one or more neural networks comprise a diffusion neural network. [12] The system of claim 8, wherein the one or more indications comprise one or more input text prompts. [13] The system of claim 8, wherein the one or more neural networks receive only text prompts as inputs. [14] The system of claim 8, wherein the one or more objects in the two or more different images are in different poses. [15] A method comprising using one or more neural networks to generate one or more objects within two or more different images based at least in part on one or more indications from one or more users indicating content other than the one or more objects in at least one of the two or more different images. [16] The method of claim 15, wherein the one or more objects within the two or more different images comprise the same subject matter. [17] The method of claim 15, wherein the one or more neural networks comprise one or more layers that denoise the two or more different images. [18] The method of claim 15, wherein the one or more neural networks comprise a diffusion neural network. [19] The method of claim 15, wherein the one or more statements comprise one or more input text prompts. [20] The method of claim 15, wherein the one or more neural networks receive only text prompts as inputs.

Images