Shape fusion for image analysis

Abstract

Multiple types of image analysis benefit from a multi-stream architecture that allows analysis to consider shape data. The shape stream may process image data in parallel with the main stream, wherein data from a network layer in the main stream is provided as an input to a network of the shape stream. The shape data may be fused with primary analysis data to produce a more accurate output, such as to produce accurate boundary information when the shape data is used with semantic segmentation data produced by the mainstream. A gating structure may be used for an intermediate layer connecting the main flow and the shape flow, gating lower levels of activations in the shape flow using higher levels of activations. Such a gating structure may help concentrate the shape flow on the relevant information and reduce any additional weight of the shape flow.

Description

[0001] This invention patent application is a divisional application of an invention patent application with an application date of March 21, 2020, application number 202080021225.9, and invention name “Shape Fusion for Image Analysis”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to (a) U.S. Provisional Patent Application Serial No. 62 / 822,685, filed on March 22, 2019, entitled “Gated Shape CNNs for Semantic Segmentation” and (b) U.S. Provisional Patent Application Serial No. 62 / 872,650, filed on July 10, 2019, entitled “Gated Shape CNNs for Semantic Segmentation” and (c) U.S. Non-Provisional Application Serial No. 16 / 825,192, filed on March 20, 2020, entitled “Shape Fusion for Image Analysis”, each of which is incorporated herein by reference in its entirety and for all purposes Background Art

[0004] Computer vision is being used for an increasing variety of tasks that come with ever-increasing performance demands. For applications such as autonomous or assisted driving or robotic control, accurate object classification is crucial. Conventional methods do not always provide the level of accuracy required for these and other such applications. For example, traditional image segmentation methods form a dense image representation where color, shape, and texture information are all processed together within a deep neural network, which may not be ideal because these parameters contain very different amounts of information relevant to recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0006] Figure 1 shows a set of vehicle cameras that may be used according to at least one embodiment;

[0007] Figure 2A , Figure 2B , Figure 2C and Figure 2D An image showing objects in a nearby environment that may be analyzed according to at least one embodiment;

[0008] Figure 3An image processing pipeline according to at least one embodiment is shown;

[0009] Figure 4 data showing various states of a pipeline through an image processing pipeline according to at least one embodiment;

[0010] Figure 5 shows a shape flow output for an input image according to at least one embodiment;

[0011] Fig. 6A and Figure 6B illustrates semantic segmentation for different views of an image in accordance with at least one embodiment;

[0012] Figure 7 illustrates a set of boundary outputs for an input image according to at least one embodiment;

[0013] Figure 8 Components of an automated or semi-automated device according to at least one embodiment are shown;

[0014] Fig. 9 A process for determining an action to take based in part on a boundary determined from image data in accordance with at least one embodiment is shown;

[0015] Fig.10 A process for analyzing image data according to at least one embodiment is shown;

[0016] Fig.11A Inference and / or training logic according to at least one embodiment is shown;

[0017] Fig. 11B Inference and / or training logic according to at least one embodiment is shown;

[0018] Fig.12 An example data center system is shown in accordance with at least one embodiment;

[0019] Fig.13 A computer system according to at least one embodiment is shown;

[0020] Fig.14 A computer system according to at least one embodiment is shown;

[0021] Fig.15 and Fig.16 illustrates at least a portion of a graphics processor according to one or more embodiments;

[0022] Fig.17A An example of an autonomous vehicle is shown in accordance with at least one embodiment;

[0023] Fig. 17B According to at least one embodiment, Fig.17A An example system architecture for an autonomous vehicle in FIG. 1 ; and

[0024] Fig. 17C One or more cloud-based servers and Fig.17A Communication system between autonomous vehicles in a vehicle. DETAILED DESCRIPTION

[0025] Computer vision generally involves one or more computing devices analyzing image data (e.g., one or more images or video content) in an attempt to determine or extract information about objects represented in the image data, such as classifying something about these objects. This can include, for example, analyzing high-dimensional data extracted from the captured image data in an attempt to identify objects or types of objects represented in the image data, and generating human-readable descriptions or labels that identify these objects. Various methods can utilize trained deep neural networks to reason about specific aspects of the captured image data, such as the semantic boundaries of objects represented in the image data. The boundaries of these objects are referred to herein as semantic boundaries because each boundary can be associated with a semantic label or other semantic data, such as identifying the type of object defined by a given boundary. This approach can help identify objects in image data, as well as the shape, location, and potential distance to these objects, which is beneficial for applications involving tasks such as navigation or component manipulation.

[0026] Figure 1 An example vehicle 100 is shown, which can utilize various aspects of various embodiments to achieve such a purpose. As shown, the vehicle 100 includes multiple cameras or imaging sensors at different locations of the vehicle to capture image data representing the environment in which the vehicle is located. These can include at least two stereo camera assemblies, including front and rear stereo cameras 168. The stereo data captured by these cameras can be used to identify objects in front and behind the vehicle 100, where these objects can include other vehicles, pedestrians, road signs, etc. Although the image data from any camera can be analyzed using computer vision to identify the type of these objects, the stereo aspect of the data enables the distance of the vehicle 100 to those objects to be determined. This is important for tasks or applications such as pedestrian detection and collision avoidance. In this example, a dedicated stereo camera can support disparity estimation, while any of these cameras can support optical flow or general computer vision determination, which may include object recognition.

[0027] Figure 2A and Figure 2BA left image 202 and a corresponding right image 204 of a stereo image pair captured from a front-facing stereo camera of a vehicle are shown. A single image may be analyzed separately (or only a single image may be captured and analyzed) to identify objects in the image, such as other vehicles, lane markings, and street signs. Where distance information is required, the difference in position of these objects in image 202, image 204, or the distance from the front-facing camera to these objects may be used to determine parallax. In some embodiments, data from at least one of these images may be used to generate a Figure 2C The semantic map 206 shown includes identification of objects identified in the image data, and an indication of the type of each object. In embodiments where distance information is also required, disparity information may be used with semantic segmentation to create a depth map 208, such as Figure 2D As shown in Figure 1, the distance of various identified objects can be represented by different colors or shades, in this example, lighter objects are closer to the camera than darker objects. When combined with computer vision data, this depth data can provide not only the identity of nearby objects, but also the relative distance to those objects. By monitoring this information over time, other information can also be determined, such as relative speed and heading, which is important for tasks such as navigation.

[0028] As described above, processes such as image segmentation can be processed together using a network that represents color, shape, and texture information, such as a deep convolutional neural network (CNN), but this approach may not be ideal because they contain very different types of information related to recognition. A method consistent with various embodiments instead utilizes a multi-stream CNN architecture for image analysis (e.g., semantic segmentation), which determines shape information as a separate processing branch or "shape stream" that can process information in parallel with the main stream, such as a conventionally trained neural network that can be used to perform image analysis. In at least one embodiment, higher-level activations in the main stream can be used to gate lower-level activations in the shape stream. This helps remove noise and enables the shape stream to focus on processing relevant boundary-related information. In at least one embodiment, a very shallow architecture can be used for a shape stream that operates at image-level resolution, producing clear predictions around object boundaries and significantly improving performance on thinner and smaller objects relative to methods that do not use shape streams.

[0029] In at least one embodiment, the mainstream can utilize a convolutional neural network (CNN), for example to perform image semantic segmentation. In such a network, the fully connected layers of a traditional CNN can be converted to convolutional layers. There may be several disadvantages to using a classification architecture alone for dense pixel prediction, which may be related to the loss of output spatial resolution due to the use of pooling layers. A specialized CNN can then be used to help restore the spatial resolution of the network output. In order to obtain the desired accuracy in the generated results, it may be beneficial to view complete and detailed object boundaries to obtain discriminative encodings of shapes, while colors and textures may contain fairly low-level information. Incorporating additional connections helps different types of information flow between network depths of different scales.

[0030] Methods according to various embodiments may utilize a multi-stream (e.g., dual-stream) CNN architecture 300 for semantic segmentation, such as Figure 3 As shown in . This architecture 300 can explicitly connect shape information into a separate processing branch. In particular, the architecture can include a mainstream 320, which utilizes an image analysis network, such as a classic CNN useful for semantic segmentation. The architecture 300 can also include a separate parallel processing branch, referred to herein as a shape stream 304. In at least one embodiment, the mainstream 302 and the shape stream 304 can operate in parallel, and the information from these streams is not fused until the top layer. In at least one embodiment, the architecture 300 uses a gating mechanism to connect the intermediate layers of the parallel streams. It should be understood that the number of layers shown in the shape stream 304 is only an example, and different numbers of layers can be used in other embodiments. The fusion module 306 is used to perform information fusion between streams 302, 304 near the pipeline. In order to predict high-quality boundaries, a loss function can be used in training to encourage the predicted semantic segmentation mask to align with the ground truth boundary. In this example, the data provided as input to the shape stream 304 comes from the continuous convolutional layers of the shape stream. The data in at least one embodiment will include features representing the image encoded at the corresponding layer. Feature data from consecutive layers helps to identify and remove image parts that do not correspond to boundaries, as data from the mainstream can be used as a series of attention maps.

[0031] In at least one embodiment, a specific type of gate can be used to allow parallel branches to interact. This can enable higher-level information contained in the classical stream to be used to denoise activations in the shape stream at a very early stage of processing. By doing so, the shape stream 304 can focus on processing relevant information. To ensure that shape information is directed to the desired stream, a semantic boundary loss can be used to supervise the network. A second loss function can be used to encourage the predicted semantic segmentation to align correctly with the ground truth semantic boundaries, which further encourages the fusion module 306 (or one or more layers) to utilize information from the shape stream 304.

[0032] In at least one embodiment, the mainstream 302 of the network can be any backbone architecture, such as a conventional segmented CNN. The shape stream 304 can process shape information in the form of semantic boundaries. Through a set of residual blocks, and the use of at least one gated convolutional layer (GCL), and local supervision, the shape stream can process only information related to the boundary. In at least one embodiment, the semantic region features from the mainstream and the boundary features from the shape stream can be fused in the fusion module 306 to produce a refined segmentation result, especially around the boundary. The fusion module 306 in at least one embodiment can use, for example, a dilated spatial convolutional pyramid pooling module (ASPP) to combine information from the two streams in a multi-scale manner. For example, a dual-task regularizer can be used to ensure high-quality boundaries on the segmentation mask.

[0033] In at least one embodiment, the main stream 302, represented as R_θ(I) with parameters θ, takes as input an image I∈R^3xHxW with height H and width W, and produces dense pixel features. In at least one embodiment, the main stream 302 can be any feed-forward fully convolutional network, such as a ResNet-based or VGG-based semantic segmentation network. In one embodiment, ResNet-like architectures (such as ResNet-101 and WideResNet) can be used for the main stream. The output feature representation of the main stream can be given by:

[0034]

[0035] where m is the stride of the main flow.

[0036] In at least one embodiment, shape flow 304 may be represented as The parameters are And the image gradient can be The shape stream is fed with the output of the first convolutional layer of the main stream as input. The shape stream can then produce semantic boundaries as output. In at least one embodiment, the architecture consists of several residual blocks (such as Res 1 in the figure) interleaved with gated convolutional layers (GCL). The gated convolutional layers can help ensure that the shape stream only processes information related to boundaries. The output boundary map of the shape stream can be expressed as Since the ground truth (GT) binary edges can be obtained from the GT semantic segmentation masks, a supervised binary cross entropy loss can be used on the output boundaries to supervise the shape flow.

[0037] In at least one embodiment, the fusion module 306 can be represented as With parameter γ, a dense feature representation r from the rule branch can be taken as input and fused with the boundary map s output by the shape branch in a way that preserves multi-scale context information. Regional features can be combined with boundary features and refined semantic segmentation output. In at least one embodiment, for segmentation prediction of K semantic classes, the output classification distribution can be given by:

[0038]

[0039] It represents the probability that a pixel belongs to each of the K classes. In at least one embodiment, the boundary map s can be merged with the dense feature representation r using ASPP. As previously described, this allows multi-scale context information to be retained.

[0040] Since tasks such as estimating semantic segmentation and semantic boundaries are closely related, gated convolutional layers (GCL) can be utilized to facilitate the flow of information from the main stream 302 into the shape stream 304. GCL can help ensure that the shape stream 304 only processes relevant information by filtering out information determined to be irrelevant to shapes or boundaries. It should be noted that in this architecture, the shape stream does not merge features from the main stream, but instead utilizes GCL to deactivate its activations that are not relevant to the higher-level information contained in the main stream. This can be thought of as a collaboration between two parallel streams, where the more powerful stream forms a higher-level semantic understanding of the scene, helping the other stream to focus only on the relevant parts. This approach allows the shape stream to adopt an effective shallow architecture to process images at very high resolutions.

[0041] GCL can be applied to multiple positions between the main stream 302 and the shape stream 304. For example, let m denote the number of positions, let t∈0,1,...,m be a running index, where rt and st denote the intermediate representations of the corresponding regular stream and shape stream processed using GCL. To apply GCL, rt and st are connected to a normalized 1x1 convolutional layer C 1x1 , and then connect a sigmoid function σ to get an attention map As shown below:

[0042] α t =σ(C 1x1 (s t ||r t ))

[0043] Where / / represents the connection of feature maps. Given the attention map α t , GCL is applied as element-wise product in s t Above, the attention map α is followed by a residual connection and kernel w t Channel-by-channel weighting. Each pixel (i, j) GCL At , GCL can be calculated as shown below:

[0044]

[0045] In at least one embodiment, then Passed to the next layer in the shape stream for further processing. Both the attention map computation and the gated convolution are differentiable, so backpropagation can be performed end-to-end. Intuitively, α can also be viewed as an attention map that weights regions with important boundary information. In at least one embodiment, three GCLs connected to the third, fourth, and last layers of the mainstream can be used. If upsampling of the feature map from the mainstream is required, bilinear interpolation can be used.

[0046] In at least one embodiment, the main stream and shape stream can be jointly trained in an end-to-end manner along with a fusion module. During training, segmentation and boundary map prediction can be jointly supervised. In this case, the boundary map can be a binary representation of the outlines of all objects and object classes in the scene, e.g. Figure 5 As shown in . The binary cross entropy (BCE) loss can be used with the predicted boundary map s, and the cross entropy (CE) loss can be used with the predicted semantic segmentation f, which can be given by:

[0047]

[0048] in represents the GT boundary, Represents the GT semantic label. Here, λ1,λ2 are two hyperparameters that control the weights between losses.

[0049] like Figure 3 As shown, BCE supervision is performed on the boundary maps s before feeding these maps to the fusion module 306. Therefore, the BCE loss Update the parameters of the main and shape streams. The final classification distribution f of the semantic class can be obtained using the final CE loss Perform supervision and update all network parameters. In the case of BCE on the boundary, the β coefficient can be used to explain the high imbalance between boundary and non-boundary pixels.

[0050] As mentioned above, It can represent the classification distribution output of the fusion module. Let ζ = R HxW is the potential energy representing whether a particular pixel belongs to a semantic boundary in the input image I. This value can be calculated by taking a spatial derivative of the segmentation output as follows:

[0051]

[0052] Where G represents a Gaussian filter. If we assume that ζ^ is a GT binary mask computed from the GT semantic label f^ in the same way, we can use the loss function as follows:

[0053]

[0054] Among them (p + ) contains ζ and The set of all non-zero pixel coordinates in . When there is a mismatch with the GT boundary, it is necessary to ensure that the boundary pixels are penalized and avoid non-boundary pixels dominating the loss function. It can be noted that the above regularized loss function exploits the duality between boundary prediction and semantic segmentation in the boundary space.

[0055] In at least one embodiment, the shape stream may be used The boundary prediction of s is used to ensure the consistency between the binary boundary prediction s and the predicted semantics p(y|r,s), as can be given by the following formula:

[0056]

[0057] where p and k operate on all image pixels and semantic classes respectively. In this case, The corresponding indicator function is thrs, which is the confidence threshold, for example 0.8. Therefore, the total dual-task regularizer loss function can be given by:

[0058]

[0059] In this case, λ3,λ4 are two hyperparameters that control the weights of the regularizer.

[0060] In order to back-propagate through the above loss function, the spatial derivative gradient of the segmentation output can be determined. For example, let g = ∥.∥, the partial derivative with respect to a given parameter η can be given by:

[0061]

[0062] Since argmax is not a differentiable function that can use the Gumbel softmax trick. During backpropagation, the argmax operator can be approximated as a softmax approximation with temperature τ, which is given by:

[0063]

[0064] where g j ~Gumbel(0,I) and τ is a hyperparameter. Operator It can be calculated by filtering with a Sobel kernel.

[0065] In at least one embodiment, the Deeplab v3+ approach can be used, but with a simpler encoder. For example, the network can be implemented using PyTorch. At least one embodiment can utilize 800x 800 as the training resolution and synchronized batch specification. Training can be performed on a workstation, such as an NVIDIA DGX station with a total batch size of 16 using 8 GPUs. For Cityscapes, a learning rate of 1e-2 can be used with a polynomial decay strategy. For example, a training process can run 100 single training iterations for ablation. For the joint loss, values ​​such as τ1=20, τ2=1, τ3=1, and τ4=1 can be set, where τ=1 for Gumbel softmax.

[0066] Figure 4 The state of data passing through the analysis pipeline 400 according to at least one embodiment is shown. In this example, an image 402 including a vehicle representation is first passed to the mainstream 30 for analysis as input to the pipeline, for example for determining semantic segmentation using a deep CNN. In this example, data 404, 406 from the layers of the mainstream network are provided as input to the network of the shape stream 304. It can be seen that the data 404, 406 received from different layers of the mainstream network will be different and can be analyzed together to produce shape data 408, which is a relatively accurate representation of the shape of the object in the input image, as shown by the accurate shape of the vehicle in the shape data 408. The shape data 408 is combined with the mainstream output of the fusion network or fusion network layer, which can produce a semantic segmentation mask 410, from which boundary data 412 can be generated. As shown, the semantic segmentation mask 410 retains semantic information, indicating the type of object recognized from the image (or at least distinguishing different objects if not recognized). As previously described, a dual-task regularizer can be used, and the dual-task loss function encourages the predicted semantic segmentation mask to align with the true boundary to ensure high-quality boundary prediction. In at least one embodiment, this enables semantic boundaries to affect semantic masks, and semantic masks to affect semantic boundaries.

[0067] Figure 5 An example 500 of shape data output by a shape stream of a pair of input images is shown. As shown, the input images have various objects of different sizes, shapes, and positions. Each shape stream output produces highly accurate shape data that distinguishes different objects and is able to determine the external shapes of these objects without generating erroneous object shapes due to features or patterns in or on the surfaces of these objects. Fig. 6AAn input image 600 is shown, which may take into account multiple different parts, such as possibly based on different image crops, to help obtain more accurate data for different distance ranges. As shown, each crop region corresponds to a different distance range, such as by making the point no closer than a determinable distance. Figure 6B As shown in view 650 of FIG. 6A , this helps to obtain a higher level of detail for objects that are farther away, because a semantic segmentation mask with a crop factor of 0 would make it difficult to determine distant objects, but by setting the crop factor to 400, it can more easily determine that there is a person behind a distant vehicle. Therefore, different crops can be used for distance-based evaluation, resulting in different predictions at different crop factors. The use of shape data has been shown to produce more accurate results in these different crop factors. Figure 7 A view 700 of the resulting boundary output data for three different input images is shown. As shown, the boundary data provides an accurate overview of the objects and features contained in the input images. In at least one embodiment, the boundary data can be color coded or otherwise labeled to retain semantic information determined from the mainstream and that may have been otherwise represented in the semantic segmentation.

[0068] Figure 8An example system architecture 800 that can be used according to at least one embodiment is shown. In this example, the architecture is used as part of an automated device 802. This can be a fully automated device, such as an autonomous vehicle or unmanned robot, or a semi-automated vehicle, such as a vehicle with driver assistance. In this example, at least one camera 804 can be used to capture image data of the environment in which the automated device 802 is located. The image data can be provided to an image processing subsystem 806, which can include a preprocessing module 808 to perform any desired or necessary preprocessing of the image data. The image data can then be passed to a perception module 810, which can be based on the multi-stream architecture discussed herein. This can include processing the data with a mainstream and a shape stream, and then fusing the information in at least one embodiment to generate semantic boundary data so that the boundary determination takes into account the shapes discussed herein. The generated semantic boundary data can be provided to a control subsystem 812 of the automated device 802. In this subsystem, a state analysis module 814 can accept boundary data as information about the current state of the environment, and can also accept state information of the automated device 802, which can be obtained from a set of sensors through a state monitoring subsystem. The state information can include any relevant information, such as position, direction, motion, temperature, pressure, etc. The state analysis module 814 may then determine one or more actions to take, such as adjusting the navigation path based on the current position and orientation of the device and objects in the environment detected using the semantic boundary information. The state analysis module 814 may then provide instructions to the control system 816, which may generate and provide actionable instructions to one or more operable components 820, which, for a vehicle, may include steering, acceleration, and braking components or assemblies, among other such options.

[0069] Fig. 9 An example process 900 for determining boundaries from input image data that can be utilized in accordance with various embodiments is shown. It should be understood that for this process and other processes discussed herein, unless otherwise stated, there can be additional, alternative, or fewer steps performed in a similar or alternative order or in parallel within the scope of various embodiments. In at least one embodiment, such a process can be implemented using a two-stream CNN architecture for semantic segmentation that connects shapes as separate parallel streams, uses a gating mechanism to connect intermediate layers, and uses a loss function that exploits the duality between the semantic segmentation task and the semantic boundary prediction task. Such an architecture can produce sharp predictions around object boundaries and can significantly improve performance for thinner and smaller objects.

[0070] In this example, an image 902 captured for the environment is received. This can include a single still image, an image of a group or series of images, or a frame of video or streaming image data, as well as other such options. In this example, at least some preprocessing 904 can be performed, such as removing noise, adjusting resolution, adjusting image quality parameters, performing lens error correction, or adjusting color depth, as well as other such options. After any preprocessing is completed, the image 906 can be provided as input to the semantic segmentation network. It should be understood that for ease of explanation, the term "image" is used in many examples, but image data can also be used in at least some embodiments and situations, which may not be a complete image in a standard image format. The image can be processed by the main branch of the architecture, such as a deep convolutional neural network trained for semantic segmentation. The data 908 of the semantic segmentation network layer can be provided as input to the shape determination network of the shape branch of the architecture. As previously described, the data can be processed in parallel by the shape determination network and the semantic segmentation network. The output 910 of the segmentation and shape determination network can then be provided as input to a fusion module, such as an ASPP module for fusion. The shape data 912 can then be combined or fused with the semantic segmentation data in a multi-scale manner to generate a semantic prediction, such as a semantic segmentation mask. Based at least in part on the semantic prediction data, a set of high-quality boundaries 914 is generated as an input image. These boundary data 916 can then be provided for use in determining one or more actions to be taken. For an autonomous vehicle, this may involve indicating a navigation action, while for a robot, this may involve an arm manipulation action, or for safety, it may involve triggering an alarm, among other such options.

[0071] Fig.10 An example process 1000 for predicting object boundaries of an image is shown, which can be utilized according to at least one embodiment. In this example, an image including a representation of one or more objects is received 1002. The image is processed 1004 to infer segmentation data of the image. At the same time, the image data can be processed 1006 to infer shape data of one or more objects. In this example, the image data is received from a network layer that processes the image to infer the segmentation. The shape data and the segmentation data are fused 1008 or combined or analyzed together to infer the boundaries of one or more objects in the image. In at least one embodiment, the fusion is performed using layers of a trained fusion network. In at least one embodiment, the boundaries will include semantic information determined for one or more objects, such as information about the type of object or the identity of those objects.

[0072] Reasoning and training logic

[0073] Fig.11AInference and / or training logic 1115 is shown for performing reasoning and / or training operations associated with one or more embodiments. Fig.11A and / or Fig. 11B Provide details about the inference and / or training logic 1115.

[0074] In at least one embodiment, the inference and / or training logic 1115 may include, but is not limited to, code and / or data storage 1101 to store forward and / or output weights and / or input / output data corresponding to other parameter configuration neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the training logic 1115 may include or be coupled with code and / or data storage 1101 to store graph code or other software to control the timing and / or sequence in which weights and / or other parameter information will be loaded to configure logic, including integer and / or floating point units (collectively referred to as arithmetic logic units (ALUs)). In at least one embodiment, code such as graph code loads weights or other parameter information into an architecture-based processor ALU. In at least one embodiment, code and / or data storage 1101 stores input / output data during training and / or inference using aspects of one or more embodiments and / or weight parameters during forward propagation of each layer of a neural network trained or used in conjunction with one or more embodiments. In at least one embodiment, any portion of code and / or data storage 1101 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.

[0075] In at least one embodiment, any portion of code and / or data storage 1101 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or code and / or data storage 1101 may be cache memory, dynamic random addressable memory ("DRAM"), static random addressable memory ("SRAM"), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether code and / or code and / or data storage 1101 is internal or external to a processor, e.g., or composed of DRAM, SRAM, flash memory, or some other storage type, may depend on the available storage space on-chip versus off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of data used in inference and / or training of a neural network, or some combination of these factors.

[0076] In at least one embodiment, the inference and / or training logic 1115 may include, but is not limited to, code and / or data storage 1105 to store backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, during training and / or inference using aspects of one or more embodiments, the code and / or data storage 1105 stores weight parameters and / or input / output data for each layer of a neural network trained or used in conjunction with one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 1115 may include or be coupled to the code and / or data storage 1105 to store graphics code or other software to control the timing and / or sequence in which weights and / or other parameter information will be loaded to configure logic, including integer and / or floating point units (collectively referred to as arithmetic logic units (ALUs)). In at least one embodiment, code, such as graphics code, loads weights or other parameter information into a processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, any portion of code and / or data storage 1105 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and / or data storage 1105 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 1105 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether code and / or data storage 1105 is internal or external to the processor, e.g., composed of DRAM, SRAM, flash memory, or some other storage type, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of data used in the inference and / or training of the neural network, or some combination of these factors.

[0077] In at least one embodiment, code and / or data store 1101 and code and / or data store 1105 may be separate storage structures. In at least one embodiment, code and / or data store 1101 and code and / or data store 1105 may be the same storage structure. In at least one embodiment, code and / or data store 1101 and data store 1105 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of code and / or data store 1101 and code and / or data store 1105 may be included with other on-chip or off-chip data stores, including the processor's L1, L2, or L3 cache or system memory.

[0078] In at least one embodiment, the inference and / or training logic 1115 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 1110, including integer and / or floating point units, for performing logical and / or mathematical operations based at least in part on or as directed by training and / or inference code (e.g., graphics code), the results of which may result in activations (e.g., output values ​​from a layer or neuron within a neural network) being stored in activation storage 1120, the activations being a function of input / output and / or weight parameter data stored in code and / or data storage 1101 and code and / or data storage 1105. In at least one embodiment, in response to executing instructions or other code, activations stored in activation storage 1120 are generated based on linear algebra and / or matrix-based mathematics performed by ALU 1110, wherein weight values ​​stored in code and / or data storage 1105 and / or code and / or data storage 1101 are used as operators with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data storage 1105 or code and / or data storage 1101 or other on-chip or off-chip storage.

[0079] In at least one embodiment, one or more ALUs 1110 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment, one or more ALUs 1110 may be outside of a processor or other hardware logic device or circuit that uses them (e.g., a coprocessor). In at least one embodiment, one or more ALUs 1110 may be included within an execution unit of a processor, or otherwise included in a group of ALUs that may be accessed by an execution unit of a processor, which may be within the same processor or distributed between different processors of different types (e.g., a central processing unit, a graphics processing unit, a fixed function unit, etc.). In at least one embodiment, code and / or data storage 1101, code and / or data storage 1105, and activation storage 1120 may be on the same processor or other hardware logic device or circuit, while in another embodiment, they may be in different processors or other hardware logic devices or circuits or some combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 1120 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Additionally, inference and / or training code may be stored with other code that is accessible to a processor or other hardware logic or circuitry and may be retrieved and / or processed using the processor's fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0080] In at least one embodiment, activation storage 1120 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 1120 may be completely or partially internal or external to one or more processors or other logic circuits. In at least one embodiment, whether activation storage 1120 is internal or external to the processor, for example, or includes DRAM, SRAM, flash memory, or other storage types, may be selected depending on the storage available on-chip versus off-chip, the latency requirements for performing training and / or inference functions, the batch size of data used in inferencing and / or training neural networks, or some combination of these factors. In at least one embodiment, Fig.11A The inference and / or training logic 1115 shown in FIG. 1 may be used in conjunction with an application specific integrated circuit (“ASIC”), such as the ASIC from Google. Processing unit from Graphcore TM Inference Processing Unit (IPU) from Intel Corp (e.g., "Lake Crest") processor. In at least one embodiment, Fig.11A The illustrated inference and / or training logic 1115 may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware such as a field programmable gate array (“FPGA”).

[0081] Fig. 11B Inference and / or training logic 1115 is shown in accordance with at least one or more various embodiments. In at least one embodiment, the reasoning and / or training logic 1115 may include, but is not limited to, hardware logic, where computing resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Fig. 11B The inference and / or training logic 1115 shown in FIG. 1 may be used in conjunction with an application specific integrated circuit (ASIC), such as the ASIC from Google. Processing unit from Graphcore TM Inference Processing Unit (IPU) from Intel Corp (e.g., "Lake Crest") processor. In at least one embodiment, Fig. 11BThe inference and / or training logic 1115 shown in can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware (e.g., field programmable gate array (FPGA)). In at least one embodiment, the inference and / or training logic 1115 includes, but is not limited to, code and / or data storage 1101 and code and / or data storage 1105, which can be used to store code (e.g., graphics code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Fig. 11B In at least one embodiment shown in , each of code and / or data storage 1101 and code and / or data storage 1105 is associated with a dedicated computing resource (e.g., computing hardware 1102 and computing hardware 1106), respectively. In at least one embodiment, each of computing hardware 1102 and computing hardware 1106 includes one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) solely on the information stored in code and / or data storage 1101 and code and / or data storage 1105, respectively, and the results of performing the functions are stored in activation storage 1120.

[0082] In at least one embodiment, each of the code and / or data stores 1101 and 1105 and the corresponding computing hardware 1102 and 1106 corresponds to a different layer of the neural network, such that activations obtained from one "storage / compute pair 1101 / 1102" of the code and / or data store 1101 and computing hardware 1102 are provided as inputs to a "storage / compute pair 1105 / 1106" of the code and / or data store 1105 and computing hardware 1106 to reflect the conceptual organization of the neural network. In at least one embodiment, each storage / compute pair 1101 / 1102 and 1105 / 1106 can correspond to more than one neural network layer. In at least one embodiment, additional storage / compute pairs (not shown) can be included in the inference and / or training logic 1115 after or in parallel with the storage / compute pairs 1101 / 1102 and 1105 / 1106.

[0083] Data Center

[0084] Fig.12 An example data center 1200 is shown in which at least one embodiment may be used. In at least one embodiment, the data center 1200 includes a data center infrastructure layer 1210 , a framework layer 1220 , a software layer 1230 , and an application layer 1240 .

[0085] In at least one embodiment, Fig.12As shown, the data center infrastructure layer 1210 may include a resource coordinator 1212, grouped computing resources 1214, and node computing resources ("node CRs") 1216(1)-1216(N), where "N" represents any complete positive integer. In at least one embodiment, the node CRs 1216(1)-1216(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state drives or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules and cooling modules, etc. In at least one embodiment, one or more of the node CRs 1216(1)-1216(N) may be a server having one or more of the above computing resources.

[0086] In at least one embodiment, the grouped computing resources 1214 may include separate groups of node CRs housed in one or more racks (not shown), or many racks (also not shown) housed in data centers at various geographic locations. The separate groups of node CRs within the grouped computing resources 1214 may include computing, network, memory, or storage resources that can be configured or allocated to support groupings of one or more workloads. In at least one embodiment, several node CRs including a CPU or processor may be grouped in one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0087] In at least one embodiment, resource coordinator 1212 may configure or otherwise control one or more nodes CR 1216(1)-1216(N) and / or grouped computing resources 1214. In at least one embodiment, resource coordinator 1212 may include a software design infrastructure ("SDI") management entity for data center 1200. In at least one embodiment, resource coordinator 1212 may include hardware, software, or some combination thereof.

[0088] In at least one embodiment, Fig.12As shown, the framework layer 1220 includes a job scheduler 1222, a configuration manager 1224, a resource manager 1226, and a distributed file system 1228. In at least one embodiment, the framework layer 1220 may include a framework that supports software 1232 of the software layer 1230 and / or one or more applications 1242 of the application layer 1240. In at least one embodiment, the software 1232 or the application 1242 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 1220 may be, but is not limited to, a free and open source software web application framework, such as Apache Spark, which may utilize the distributed file system 1228 for large-scale data processing (e.g., "big data"). TM (hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 1222 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 1200. In at least one embodiment, the configuration manager 1224 may be able to configure different layers, such as the software layer 1230 and the framework layer 1220 including Spark and a distributed file system 1228 for supporting large-scale data processing. In at least one embodiment, the resource manager 1226 can manage cluster or group computing resources mapped to or allocated to support the distributed file system 1228 and the job scheduler 1222. In at least one embodiment, the cluster or group computing resources may include grouped computing resources 1214 on the data center infrastructure layer 1210. In at least one embodiment, the resource manager 1226 may coordinate with the resource coordinator 1212 to manage these mapped or allocated computing resources.

[0089] In at least one embodiment, software 1232 included in software layer 1230 may include software used by at least a portion of node CRs 1216(1)-1216(N), grouped computing resources 1214, and / or distributed file system 1228 of framework layer 1220. One or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0090] In at least one embodiment, one or more applications 1242 included in application layer 1240 may include one or more types of applications used by at least a portion of node CRs 1216(1)-1216(N), grouped computing resources 1214, and / or distributed file system 1228 of framework layer 1220. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0091] In at least one embodiment, any of the configuration manager 1224, resource manager 1226, and resource coordinator 1212 can implement any number and type of self-modification actions based on any number and type of data acquired in any technically feasible manner. In at least one embodiment, the self-modification actions can relieve a data center operator of the data center 1200 from making potentially bad configuration decisions and can avoid underutilized and / or poorly performing portions of the data center.

[0092] In at least one embodiment, data center 1200 may include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to data center 1200. In at least one embodiment, by using weight parameters calculated by one or more training techniques described herein, information may be inferred or predicted using trained machine learning models corresponding to one or more neural networks using the resources described above with respect to data center 1200.

[0093] In at least one embodiment, the data center can use a CPU, an application-specific integrated circuit (ASIC), a GPU, an FPGA, or other hardware to use the above resources to perform training and / or reasoning. In addition, one or more of the above software and / or hardware resources can be configured as a service to allow users to train or perform information reasoning, such as image recognition, speech recognition, or other artificial intelligence services.

[0094] The reasoning and / or training logic 1115 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig.11A and / or Fig. 11BProvides details about the reasoning and / or training logic 1115. In at least one embodiment, the reasoning and / or training logic 1115 may be in the system Fig.12 In some embodiments, the present invention provides a method for performing inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0095] In at least one embodiment, such a component can be used to perform classification or segmentation of one or more objects represented in the image data. This includes using a shape stream in parallel with the main image analysis stream to force the main stream to take into account the shape information of the object.

[0096] Computer Systems

[0097] According to at least one embodiment, Fig.13 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system on a chip (SOC), or some combination thereof 1300, formed together with a processor, which may include an execution unit to execute instructions. In at least one embodiment, the computer system 1300 may include, but is not limited to, components such as those described in the embodiments described herein, such as a processor 1302, to execute algorithms for process data using execution units including logic. In at least one embodiment, the computer system 1300 may include a processor, such as Processor series, Xeon TM , XScale TM and / or StrongARM TM , Core TM , or from Intel Corporation in Santa Clara, California Nervana TM microprocessor, although other systems (including PCs with other microprocessors, engineering workstations, set-top boxes, etc.) may also be used. In at least one embodiment, computer system 1300 may execute a version of the WINDOWS operating system available from Microsoft Corporation of Redmond, Washington, although other operating systems (e.g., UNIX and Linux), embedded software, and / or graphical user interfaces may also be used.

[0098] Embodiments may be used in other devices, such as handheld devices and embedded applications. Some examples of handheld devices include cellular phones, Internet protocol devices, digital cameras, personal digital assistants ("PDAs"), and handheld PCs. In at least one embodiment, the embedded application may include a microcontroller, a digital signal processor ("DSP"), a single chip system, a network computer ("NetPC"), a set-top box, a network hub, a wide area network ("WAN") switch, or any other system that can execute one or more instructions according to at least one embodiment.

[0099] In at least one embodiment, computer system 1300 may include, but is not limited to, processor 1302, which may include, but is not limited to, one or more execution units 1308 to perform machine learning model training and / or reasoning according to the techniques described herein. In at least one embodiment, computer system 1300 is a single-processor desktop or server system, but in another embodiment, computer system 1300 may be a multi-processor system. In at least one embodiment, processor 1302 may include, but is not limited to, a complex instruction set computer ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, a processor that implements a combination of instruction sets, or any other processor device, such as a digital signal processor. In at least one embodiment, processor 1302 may be coupled to a processor bus 1310, which may transmit data signals between processor 1302 and other components in computer system 1300.

[0100] In at least one embodiment, processor 1302 may include, but is not limited to, a level 1 ("L1") internal cache memory ("cache") 1304. In at least one embodiment, processor 1302 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, cache memory may reside external to processor 1302. Other embodiments may also include a combination of internal and external caches, depending on the particular implementation and requirements. In at least one embodiment, register file 1306 may store different types of data in various registers, including, but not limited to, integer registers, floating point registers, status registers, and instruction pointer registers.

[0101] In at least one embodiment, an execution unit 1308, including but not limited to logic to perform integer and floating point operations, also resides in the processor 1302. In at least one embodiment, the processor 1302 may also include a microcode ("ucode") read-only memory ("ROM") for storing microcode for certain macroinstructions. In at least one embodiment, the execution unit 1308 may include logic to process a packed instruction set 1309. In at least one embodiment, by including the packed instruction set 1309 in the instruction set of the general purpose processor 1302, along with associated circuitry to execute the instructions, operations used by many multimedia applications may be performed using packed data in the general purpose processor 1302. In one or more embodiments, many multimedia applications may be executed faster and more efficiently by using the full width of the processor's data bus for performing operations on packed data, which may eliminate the need to transfer smaller units of data across the processor's data bus to perform one or more operations one data element at a time.

[0102] In at least one embodiment, execution unit 1308 may also be used in microcontrollers, embedded processors, graphics devices, data signal processing, and other types of logic circuits. In at least one embodiment, computer system 1300 may include, but is not limited to, memory 1320. In at least one embodiment, memory 1320 may be implemented as a dynamic random access memory ("DRAM") device, a static random access memory ("SRAM") device, a flash memory device, or other storage device. In at least one embodiment, memory 1320 may store instructions 1319 and / or data 1321 represented by data signals that may be executed by processor 1302.

[0103] In at least one embodiment, the system logic chip can be coupled to the processor bus 1310 and the memory 1320. In at least one embodiment, the system logic chip can include, but is not limited to, a memory controller hub ("MCH") 1316, and the processor 1302 can communicate with the MCH 1316 via the processor bus 1310. In at least one embodiment, the MCH 1316 can provide a high-bandwidth memory path 1318 to the memory 1320 for instruction and data storage and for storage of graphics commands, data, and textures. In at least one embodiment, the MCH 1316 can direct data signals between the processor 1302, the memory 1320, and other components in the computer system 1300 and bridge data signals between the processor bus 1310, the memory 1320, and the system I / O 1322. In at least one embodiment, the system logic chip can provide a graphics port for coupling to a graphics controller. In at least one embodiment, the MCH 1316 may be coupled to the memory 1320 via a high bandwidth memory path 1318 and the graphics / video card 1312 may be coupled to the MCH 1316 via an accelerated graphics port (“AGP”) interconnect 1314 .

[0104] In at least one embodiment, the computer system 1300 may use a system I / O 1322 that is a proprietary hub interface bus to couple the MCH 1316 to an I / O controller hub ("ICH") 1330. In at least one embodiment, the ICH 1330 may provide direct connection to certain I / O devices through a local I / O bus. In at least one embodiment, the local I / O bus may include, but is not limited to, a high-speed I / O bus used to connect peripheral devices to the memory 1320, the chipset, and the processor 1302. Examples may include, but are not limited to, an audio controller 1329, a firmware hub ("flash BIOS") 1328, a wireless transceiver 1326, a data store 1324, a traditional I / O controller 1323 including a user input and keyboard interface 1325, a serial expansion port 1327, such as a universal serial bus ("USB"), and a network controller 1334. The data store 1324 may include a hard drive, a floppy drive, a CD-ROM device, a flash memory device, or other mass storage device.

[0105] In at least one embodiment, Fig.13 A system comprising interconnected hardware devices or "chips" is shown, while in other embodiments, Fig.13 An exemplary system on a chip ("SoC") is shown. In at least one embodiment, the devices can be interconnected with a proprietary interconnect, a standardized interconnect (e.g., PCIe), or some combination thereof. In at least one embodiment, one or more components of computer system 1300 are interconnected using a Compute Express Link (CXL) interconnect.

[0106] The reasoning and / or training logic 1115 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig.11A and / or Fig. 11B Provides details about the reasoning and / or training logic 1115. In at least one embodiment, the reasoning and / or training logic 1115 may be used in the system Fig.13 For inferring or predicting operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network operations described herein.

[0107] In at least one embodiment, such a component may be used to perform classification or segmentation of one or more objects represented in the image data. This may include using a shape stream in parallel with the main image analysis stream to force the main stream to consider shape information of the one or more objects.

[0108] Fig.14 1 is a block diagram illustrating an electronic device 1400 for utilizing a processor 1410 according to at least one embodiment. In at least one embodiment, the electronic device 1400 may be, for example, but not limited to, a notebook, a tower server, a rack server, a blade server, a laptop, a desktop, a tablet, a mobile device, a phone, an embedded computer, or any other suitable electronic device.

[0109] In at least one embodiment, system 1400 may include, but is not limited to, processor 1410, which is communicatively coupled to any suitable number or kind of components, peripherals, modules, or devices. In at least one embodiment, processor 1410 is coupled using a bus or interface, such as an I2C bus, a system management bus ("SMBus"), a low pin count (LPC) bus, a serial peripheral interface ("SPI"), a high-definition audio ("HDA") bus, a serial advanced technology attachment ("SATA") bus, a universal serial bus ("USB") (versions 1, 2, 3), or a universal asynchronous receiver / transmitter ("UART") bus. In at least one embodiment, Fig.14 A system is shown that includes interconnected hardware devices or "chips", while in other embodiments, Fig.14 An exemplary system on a chip ("SoC") may be shown. In at least one embodiment, Fig.14 The devices shown in can be interconnected with a proprietary interconnect, a standardized interconnect (e.g., PCIe), or some combination thereof. In at least one embodiment, Fig.14 One or more components of the system are interconnected by one or more components using a Compute Express Link (CXL) interconnect.

[0110] In at least one embodiment, Fig.14 It may include a display 1424, a touch screen 1425, a touch pad 1430, a near field communication unit (“NFC”) 1445, a sensor hub 1440, a thermal sensor 1446, a high-speed chipset (“EC”) 1435, a trusted platform module (“TPM”) 1438, a BIOS / firmware / flash (“BIOS, FW flash”) 1422, a DSP 1460, a drive 1420 such as a solid state disk (“SSD”) or a hard disk drive (“HDD”), a wireless local area network unit (“WLAN”) 1450, a Bluetooth unit 1452, a wireless wide area network unit (“WWAN”) 1456, a global positioning system (GPS) 1455, a camera (“USB 3.0 camera”) 1454 (e.g., a USB 3.0 camera), and / or a low power double data rate (“LPDDR”) storage unit (“LPDDR3”) 1415, for example, implemented in the LPDDR3 standard. Each of these components may be implemented in any suitable way.

[0111] In at least one embodiment, other components may be communicatively coupled to the processor 1410 via the components discussed above. In at least one embodiment, an accelerometer 1441, an ambient light sensor (“ALS”) 1442, a compass 1443, and a gyroscope 1444 may be communicatively coupled to the sensor hub 1440. In at least one embodiment, a thermal sensor 1439, a fan 1437, a keyboard 1446, and a touchpad 1430 may be communicatively coupled to the EC 1435. In at least one embodiment, a speaker 1463, an earphone 1464, and a microphone (“mic”) 1465 may be communicatively coupled to an audio unit (“audio codec and class-d amplifier”) 1462, which in turn may be communicatively coupled to the DSP 1460. In at least one embodiment, the audio unit 1464 may include, for example, but not limited to, an audio encoder / decoder (“codec”) and a class-D amplifier. In at least one embodiment, a SIM card (“SIM”) 1457 may be communicatively coupled to the WWAN unit 1456. In at least one embodiment, components such as the WLAN unit 1450 and the Bluetooth unit 1452, as well as the WWAN unit 1456 may be implemented in a next generation form factor ("NGFF").

[0112] The reasoning and / or training logic 1115 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig.11A and Fig. 11B Details regarding the inference and / or training logic 1115 are provided in the following. In at least one embodiment, the inference and / or training logic 1115 may be used in the system Fig.14In the present invention, an inference or prediction operation is performed based at least in part on weight parameters calculated using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0113] In at least one embodiment, such a component may be used to perform classification or segmentation of one or more objects represented in the image data. This may include using a shape stream in parallel with the main image analysis stream to force the main stream to take into account shape information of the object.

[0114] Fig.15 is a block diagram of a processing system according to at least one embodiment. In at least one embodiment, system 1500 includes one or more processors 1502 and one or more graphics processors 1508, and can be a single-processor desktop system, a multi-processor workstation system, or a server system with a large number of processors 1502 or processor cores 1507. In at least one embodiment, system 1500 is a processing platform integrated in a system-on-chip (SoC) integrated circuit for use in a mobile, handheld, or embedded device.

[0115] In at least one embodiment, system 1500 may include or be incorporated into a server-based gaming platform, and gaming consoles include gaming and media consoles, mobile gaming consoles, handheld gaming consoles, or online gaming consoles. In at least one embodiment, system 1500 is a mobile phone, a smart phone, a tablet device, or a mobile internet device. In at least one embodiment, processing system 1500 may also include, be coupled to, or be integrated into a wearable device, such as a smart watch wearable device, a smart glasses device, an augmented reality device, or a virtual reality device. In at least one embodiment, processing system 1500 is a television or set-top box device having one or more processors 1502 and a graphical interface generated by one or more graphics processors 1508.

[0116] In at least one embodiment, one or more processors 1502 each include one or more processor cores 1507 to process instructions, which, when executed, perform operations of the system and user software. In at least one embodiment, each of the one or more processor cores 1507 is configured to process a specific instruction set 1509. In at least one embodiment, the instruction set 1509 can facilitate complex instruction set computing (CISC), reduced instruction set computing (RISC), or computing through very long instruction words (VLIW). In at least one embodiment, the processor cores 1507 can each process different instruction sets 1509, which may include instructions that facilitate emulation of other instruction sets. In at least one embodiment, the processor cores 1507 may also include other processing devices, such as digital signal processors (DSPs).

[0117] In at least one embodiment, the processor 1502 includes a cache memory 1504. In at least one embodiment, the processor 1502 may have a single internal cache or multiple levels of internal cache. In at least one embodiment, the cache memory is shared between various components of the processor 1502. In at least one embodiment, the processor 1502 also uses an external cache (e.g., a level 3 (L3) cache or a last level cache (LLC)) (not shown), which can be shared between the processor cores 1507 using known cache coherence techniques. In at least one embodiment, the processor 1502 additionally includes a register file 1506, which may include different types of registers (e.g., integer registers, floating point registers, status registers, and instruction pointer registers) for storing different types of data. In at least one embodiment, the register file 1506 may include general registers or other registers.

[0118] In at least one embodiment, one or more processors 1502 are coupled to one or more interface buses 1510 to transmit communication signals such as addresses, data, or control signals between the processor 1502 and other components in the system 1500. In at least one embodiment, the interface bus 1510 may be a processor bus in one embodiment, such as a version of a direct media interface (DMI) bus. In at least one embodiment, the interface 1510 is not limited to a DMI bus, and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), a memory bus, or other types of interface buses. In at least one embodiment, the processor 1502 includes an integrated memory controller 1516 and a platform controller hub 1530. In at least one embodiment, the memory controller 1516 facilitates communication between memory devices and other components of the system 1500, while the platform controller hub (PCH) 1530 provides connections to I / O devices through a local I / O bus.

[0119] In at least one embodiment, the memory device 1520 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase change memory device, or other memory device with suitable performance as process memory. In at least one embodiment, the memory device 1520 may operate as system memory of the system 1500 to store data 1522 and instructions 1521 for use when one or more processors 1502 execute applications or processes. In at least one embodiment, the integrated memory controller 1516 is also coupled to an optional external graphics processor 1512, which may communicate with one or more graphics processors 1508 in the processor 1502 to perform graphics and media operations. In at least one embodiment, a display device 1511 may be connected to the processor 1502. In at least one embodiment, the display device 1511 may include one or more internal display devices, such as in a mobile electronic device or a laptop device, or an external display device attached via a display interface (e.g., DisplayPort, etc.). In at least one embodiment, the display device 1511 may include a head-mounted display (HMD), such as a stereoscopic display device used in virtual reality (VR) applications or augmented reality (AR) applications.

[0120] In at least one embodiment, the platform controller hub 1530 enables peripheral devices to be connected to the memory device 1520 and the processor 1502 via a high-speed I / O bus. In at least one embodiment, the I / O peripherals include, but are not limited to, an audio controller 1546, a network controller 1534, a firmware interface 1528, a wireless transceiver 1526, a touch sensor 1525, and a data storage device 1524 (e.g., a hard drive, flash memory, etc.). In at least one embodiment, the data storage device 1524 can be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCIExpress). In at least one embodiment, the touch sensor 1525 may include a touch screen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 1526 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver such as a 3G, 4G, or long-term evolution (LTE) transceiver. In at least one embodiment, the firmware interface 1528 implements communication with the system firmware, such as a unified extensible firmware interface (UEFI). In at least one embodiment, the network controller 1534 can implement a network connection to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to the interface bus 1510. In at least one embodiment, the audio controller 1546 is a multi-channel high-definition audio controller. In at least one embodiment, the system 1500 includes an optional traditional I / O controller 1540 for coupling traditional (e.g., personal system 2 (PS / 2)) devices to the system. In at least one embodiment, the platform controller hub 1530 can also be connected to one or more universal serial bus (USB) controllers 1542 to connect input devices, such as a keyboard and mouse 1543 combination, a camera 1544, or other USB input devices.

[0121] In at least one embodiment, instances of memory controller 1516 and platform controller hub 1530 may be integrated into a discrete external graphics processor, such as external graphics processor 1512. In at least one embodiment, platform controller hub 1530 and / or integrated memory controller 1516 may be external to one or more processors 1502. For example, in at least one embodiment, system 1500 may include an external integrated memory controller 1516 and platform controller hub 1530, which may be configured as a memory controller hub and a peripheral controller hub within a system chipset that communicates with processor 1502.

[0122] The reasoning and / or training logic 1115 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig.11A and Fig. 11B1500. In at least one embodiment, some or all of the inference and / or training logic 1115 may be incorporated into the graphics processor 1500. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs included in the graphics processor 1512. In addition, in at least one embodiment, the inference and / or training operations described herein may use processors other than the graphics processor 1512. Fig.11A or Fig. 11B In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) to configure the ALU of graphics processor 1500 to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0123] In at least one embodiment, such a component may be used to perform classification or segmentation of one or more objects represented in the image data. This may include using a shape stream in parallel with the main image analysis stream to force the main stream to consider shape information of the one or more objects.

[0124] Fig.16 1 is a block diagram of a processor 1600 having one or more processor cores 1602A-1602N, an integrated memory controller 1614, and an integrated graphics processor 1608 according to at least one embodiment. In at least one embodiment, the processor 1600 may include additional cores, up to and including the additional core 1602N represented by the dashed box. In at least one embodiment, each of the processor cores 1602A-1602N includes one or more internal cache units 1604A-1604N. In at least one embodiment, each processor core may also have access to one or more shared cache units 1606.

[0125] In at least one embodiment, the internal cache units 1604A-1604N and the shared cache unit 1606 represent a cache memory hierarchy within the processor 1600. In at least one embodiment, the cache memory units 1604A-1604N may include at least one level of instruction and data cache in each processor core and one or more shared mid-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, with the highest level of cache before external memory being classified as LLC. In at least one embodiment, cache coherence logic maintains coherence between the various cache units 1606 and 1604A-1604N.

[0126] In at least one embodiment, the processor 1600 may also include a set of one or more bus controller units 1616 and a system agent core 1610. In at least one embodiment, the one or more bus controller units 1616 manage a set of peripheral buses, such as one or more PCI or PCI Express buses. In at least one embodiment, the system agent core 1610 provides management functions for various processor components. In at least one embodiment, the system agent core 1610 includes one or more integrated memory controllers 1614 to manage access to various external memory devices (not shown).

[0127] In at least one embodiment, one or more processor cores 1602A-1602N include support for simultaneous multithreading. In at least one embodiment, system agent core 1610 includes components for coordinating and operating cores 1602A-1602N during multithreaded processing. In at least one embodiment, system agent core 1610 may additionally include a power control unit (PCU) that includes logic and components for regulating one or more power states of processor cores 1602A-1602N and graphics processor 1608.

[0128] In at least one embodiment, the processor 1600 additionally includes a graphics processor 1608 to perform graphics processing operations. In at least one embodiment, the graphics processor 1608 is coupled to the shared cache unit 1606 and the system agent core 1610, including one or more integrated memory controllers 1614. In at least one embodiment, the system agent core 1610 also includes a display controller 1611 to drive the graphics processor output to one or more coupled displays. In at least one embodiment, the display controller 1611 can also be a separate module coupled to the graphics processor 1608 via at least one interconnect, or can be integrated within the graphics processor 1608.

[0129] In at least one embodiment, a ring-based interconnect unit 1612 is used to couple the internal components of the processor 1600. In at least one embodiment, alternative interconnect units may be used, such as point-to-point interconnects, switched interconnects, or other technologies. In at least one embodiment, the graphics processor 1608 is coupled to the ring interconnect 1612 via an I / O link 1613.

[0130] In at least one embodiment, I / O link 1613 represents at least one of a variety of I / O interconnects, including package I / O interconnect 1618 that facilitates communication between various processor components and high-performance embedded memory modules, such as eDRAM modules. In at least one embodiment, each processor core 1602A-1602N and graphics processor 1608 uses embedded memory module 1618 as a shared last-level cache.

[0131] In at least one embodiment, processor cores 1602A-1602N are isomorphic cores that execute a common instruction set architecture. In at least one embodiment, processor cores 1602A-1602N are heterogeneous in terms of instruction set architecture (ISA), wherein one or more processor cores 1602A-1602N execute a common instruction set, and one or more other processor cores 1602A-16-02N execute a subset of a common instruction set or a different instruction set. In at least one embodiment, processor cores 1602A-1602N are heterogeneous in terms of microarchitecture, wherein one or more cores with relatively high power consumption are coupled with one or more cores with lower power consumption. In at least one embodiment, processor 1600 can be implemented on one or more chips or as a SoC integrated circuit.

[0132] The reasoning and / or training logic 1115 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig.11A and / or Fig. 11B 1600. In at least one embodiment, a portion or all of the inference and / or training logic 1115 may be incorporated into the processor 1600. For example, in at least one embodiment, the training and / or inference techniques described herein may use one or more ALUs, graphics cores 1602A-1602N, or other processors included in the graphics processor 1512. Fig.16 In addition, in at least one embodiment, the reasoning and / or training operations described herein may use Fig.11A and Fig. 11B In at least one embodiment, the weight parameters may be stored in on-chip or off-chip memory and / or registers (shown or not shown) to configure the ALU of graphics processor 1600 to execute one or more machine learning algorithms, neural network architectures, use cases, or training techniques described herein.

[0133] In at least one embodiment, such a component may be used to perform classification or segmentation of one or more objects represented in the image data. This may include using a shape stream in parallel with the main image analysis stream to force the main stream to consider shape information of the one or more objects.

[0134] Autonomous Vehicles

[0135] Fig.17A An example of an autonomous vehicle 1700 according to at least one embodiment is shown. In at least one embodiment, autonomous vehicle 1700 (alternatively referred to herein as "vehicle 1700") can be, but is not limited to, a passenger vehicle, such as a car, truck, bus, and / or another type of vehicle that can accommodate one or more passengers. In at least one embodiment, vehicle 1a00 can be a semi-tractor-trailer for hauling cargo. In at least one embodiment, vehicle 1a00 can be an aircraft, a robotic vehicle, or another type of vehicle.

[0136] An autonomous vehicle may be described according to the automation levels "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (e.g., Standard No. J3016-201806 published on June 15, 2018, Standard No. J3016-201609 published on September 30, 2016, and previous and future versions of the standard) defined by the National Highway Traffic Safety Administration ("NHTSA") and the Society of Automotive Engineers ("SAE") under the U.S. Department of Transportation. In one or more embodiments, the vehicle 1700 may have one or more functions of Level 1 to Level 5 according to the level of autonomous driving. For example, in at least one embodiment, according to an embodiment, the vehicle 1700 may be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0137] In at least one embodiment, the vehicle 1700 may include, but is not limited to, components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. In at least one embodiment, the vehicle 1700 may include, but is not limited to, a propulsion system 1750, such as an internal combustion engine, a hybrid power plant, an all-electric engine, and / or another type of propulsion system. In at least one embodiment, the propulsion system 1750 may be connected to a drive train of the vehicle 1700, which may include, but is not limited to, a transmission, so that the vehicle 1700 can be propelled. In at least one embodiment, the propulsion system 1750 may be controlled in response to receiving a signal from a throttle / accelerator 1752.

[0138] In at least one embodiment, when propulsion system 1750 is operating (e.g., when the vehicle is traveling), steering system 1754 (which may include, but is not limited to, a steering wheel) is used to steer vehicle 1700 (e.g., along a desired path or route). In at least one embodiment, steering system 1754 may receive signals from steering actuator 1756. The steering wheel may be optional for full automation (Level 5) functionality. In at least one embodiment, brake sensor system 1746 may be used to operate vehicle brakes in response to signals received from brake actuator 1748 and / or brake sensors.

[0139] In at least one embodiment, controller 1736 may include, but is not limited to, one or more system on a chip (“SoC”) ( Fig.17A ) and / or a graphics processing unit (“GPU”) that provides signals (e.g., representing commands) to one or more components and / or systems of the vehicle 1700. For example, in at least one embodiment, the controller 1736 can send signals to operate vehicle brakes via brake actuator 1748, operate steering system 1754 via one or more steering actuators 1756, and / or operate propulsion system 1750 via one or more throttles / accelerators 1752. The one or more controllers 1736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a driver in driving the vehicle 1700. In at least one embodiment, the one or more controllers 1736 may include a first controller 1736 for autonomous driving functions, a second controller 1736 for functional safety functions, a third controller 1736 for artificial intelligence functions (e.g., computer vision), a fourth controller 1736 for infotainment functions, a fifth controller 1736 for redundancy in emergency situations, and / or other controllers. In at least one embodiment, a single controller 1736 may handle two or more of the above functions, two or more controllers 1736 may handle a single function, and / or any combination thereof.

[0140] In at least one embodiment, the one or more controllers 1736 provide signals for controlling one or more components and / or systems of the vehicle 1700 in response to sensor data received from one or more sensors (e.g., sensor inputs). In at least one embodiment, the sensor data can be received from sensors of a type such as, but not limited to, one or more global navigation satellite system ("GNSS") sensors 1758 (e.g., one or more global positioning system sensors), one or more RADAR sensors 1760, one or more ultrasonic sensors 1762, one or more LIDAR sensors 1764, one or more inertial measurement unit (IMU) sensors 1766 (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetic compasses, one or more magnetometers, etc.), one or more microphones 1796, one or more stereo cameras 1768, one or more wide angle cameras 1770 (e.g., fisheye cameras), one or more infrared cameras 1772, one or more surround cameras 1774 (e.g., 360 degree cameras), telephoto cameras (e.g., infrared cameras), and / or a plurality of other sensors. Fig.17A Not shown), mid-range camera ( Fig.17A 1700), one or more speed sensors 1744 (e.g., for measuring the speed of the vehicle 1700), one or more vibration sensors 1742, one or more steering sensors 1740, one or more brake sensors (e.g., as part of a brake sensor system 1746), and / or other sensor types are received.

[0141] In at least one embodiment, one or more controllers 1736 may receive input (e.g., represented by input data) from a dashboard 1732 of the vehicle 1700 and provide output (e.g., represented by output data, display data, etc.) via a human machine interface ("HMI") display 1734, an audible annunciator, a speaker, and / or other components of the vehicle 1700. In at least one embodiment, the output may include information such as vehicle speed, velocity, time, map data (e.g., high definition map ( Fig.17A 1700 ), location data (e.g., the location of the vehicle 1700 , such as on a map), directions, locations of other vehicles (e.g., occupancy barriers), information about objects and states of objects sensed by one or more controllers 1736 , etc. For example, in at least one embodiment, the HMI display 1734 can display information about the presence of one or more objects (e.g., road signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers the vehicle has, is, or will make (e.g., changing lanes now, taking exit 34B in two miles, etc.).

[0142] In at least one embodiment, the vehicle 1700 further includes a network interface 1724, which can use one or more wireless antennas 1726 and / or one or more modems to communicate through one or more networks. For example, in at least one embodiment, the network interface 1724 may be able to communicate through Long Term Evolution ("LTE"), Wideband Code Division Multiple Access ("WCDMA"), Universal Mobile Telecommunications System ("UMTS"), Global System for Mobile Communications ("GSM"), IMT-CDMA Multi-Carrier ("CDMA2000"), etc. In at least one embodiment, the one or more wireless antennas 1726 can also use one or more local area networks (e.g., Bluetooth, low energy Bluetooth (LE), Z-Wave, ZigBee, etc.) and / or one or more low power wide area networks (hereinafter referred to as "LPWAN") (e.g., LoRaWAN, SigFox, etc.) to enable communication between objects in the environment (e.g., vehicles, mobile devices).

[0143] The reasoning and / or training logic 1115 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig.11A and / or Fig. 11B Provides details about the reasoning and / or training logic 1115. In at least one embodiment, the reasoning and / or training logic 1115 may be in the system Fig.17A for performing inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0144] In at least one embodiment, such a component can be used to perform classification or segmentation of one or more objects represented in the image data. This can include using a shape stream in parallel with the main image analysis stream to force the main stream to consider shape information of the object.

[0145] Refer back Figure 1 , according to at least one embodiment Fig.17A 1700. In at least one embodiment, the cameras and respective fields of view are an example embodiment and are not intended to be limiting. For example, in at least one embodiment, additional and / or alternative cameras may be included and / or the cameras may be located at different locations on the vehicle 1700.

[0146] In at least one embodiment, the camera type for the camera may include, but is not limited to, a digital camera that may be suitable for use with components and / or systems of the vehicle 1700. In at least one embodiment, one or more cameras may operate at an automotive safety integrity level ("ASIL") B and / or other ASILs. In at least one embodiment, the camera type may have any image capture rate, such as 60 frames per second (fps), 120fps, 240fps, etc., depending on the embodiment. In at least one embodiment, the camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In at least one embodiment, the color filter array may include a red transparent transparent ("RCCC") filter array, a red transparent transparent blue ("RCCB") filter array, a red blue green transparent ("RBGC") filter array, a Foveon X3 filter array, a Bayer sensor ("RGGB") filter array, a monochrome sensor filter array, and / or other types of filter arrays. In at least one embodiment, a clear pixel camera, such as one having an RCCC, RCCB, and / or RBGC color filter array, may be used in an effort to increase photosensitivity.

[0147] In at least one embodiment, one or more cameras can be used to perform advanced driver assistance system ("ADAS") functions (e.g., as part of a redundant or fail-safe design). For example, in at least one embodiment, a multi-function mono camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. In at least one embodiment, one or more cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0148] In at least one embodiment, one or more cameras can be mounted in a mounting assembly, such as a custom designed (three-dimensional ("3D") printed) assembly, so as to cut out stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. With respect to the rearview mirror mounting assembly, in at least one embodiment, the rearview mirror assembly can be 3D printed custom so that the camera mounting plate matches the shape of the rearview mirror. In at least one embodiment, one or more cameras can be integrated into the rearview mirror. For side-view cameras, in at least one embodiment, one or more cameras can also be integrated within four pillars at each corner of the cabin.

[0149] In at least one embodiment, a camera (e.g., a forward-facing camera) having a field of view that includes a portion of the environment in front of the vehicle 1700 can be used for surround vision, as well as to help identify the forward path and obstacles with the assistance of one or more controllers 1736 and / or control SoCs, thereby providing information critical to generating an occupancy grid and / or determining a preferred vehicle path. In at least one embodiment, the forward-facing camera can be used to perform many of the same ADAS functions as LIDAR, including but not limited to emergency braking, pedestrian detection, and collision avoidance. In at least one embodiment, the forward-facing camera can also be used for ADAS functions and systems, including but not limited to lane departure warning ("LDW"), automatic cruise control ("ACC"), and / or other functions (e.g., traffic sign recognition).

[0150] In at least one embodiment, a variety of cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform including a CMOS ("complementary metal oxide semiconductor") color imager. In at least one embodiment, a wide-angle camera 1770 can be used to sense objects entering from the periphery (e.g., pedestrians, people crossing the street, or bicycles). Although in Figure 1 Only one wide-angle camera 1770 is shown in FIG. 1 , however, in other embodiments, there may be any number (including zero) of wide-angle cameras 1770 on the vehicle 1700. In at least one embodiment, any number of remote cameras 1798 (e.g., a remote stereo camera pair) may be used for depth-based object detection, particularly for neural networks that have not been trained for the object. In at least one embodiment, the remote cameras 1798 may also be used for object detection and classification and basic object tracking.

[0151] In at least one embodiment, any number of stereo cameras 1768 may also be included in the forward configuration. In at least one embodiment, one or more stereo cameras 1768 may include an integrated control unit including a scalable processing unit that may provide programmable logic ("FPGA") and a multi-core microprocessor with a controller area network ("CAN") or Ethernet interface integrated on a single chip. In at least one embodiment, such a unit may be used to generate a 3D map of the environment of the vehicle 1700, including distance estimates for all points in the image. In at least one embodiment, one or more stereo cameras 1768 may include, but are not limited to, a compact stereo vision sensor, which may include, but are not limited to, two camera lenses (one on each side) and an image processing chip that may measure the distance from the vehicle 1700 to the target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. In at least one embodiment, other types of stereo cameras 1768 may be used in addition to those described herein.

[0152] In at least one embodiment, a camera having a view of a portion of the environment including the sides of the vehicle 1700 (e.g., a side view camera) can be used for surround viewing, thereby providing information for creating and updating an occupancy grid, and generating side collision warnings. For example, in at least one embodiment, one or more surround cameras 1774 (e.g., such as Figure 1 Four surround cameras 1774 (shown) may be positioned on the vehicle 1700. In at least one embodiment, the one or more surround cameras 1774 may include, but are not limited to, any number and combination of one or more wide angle cameras 1770, one or more fisheye lenses, one or more 360 ​​degree cameras, and / or the like. For example, in at least one embodiment, four fisheye lens cameras may be located at the front, rear, and sides of the vehicle 1700. In at least one embodiment, the vehicle 1700 may use three surround cameras 1774 (e.g., left, right, and rear), and may utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround camera.

[0153] In at least one embodiment, a camera having a field of view that includes a portion of the environment behind the vehicle 1700 (e.g., a rear view camera) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. In at least one embodiment, a variety of cameras can be used, including but not limited to cameras that are also suitable as one or more forward-facing cameras (e.g., long-range camera 1798 and / or one or more mid-range cameras 1776, one or more stereo cameras 1768, one or more infrared cameras 1772, etc.), as described herein.

[0154] Reasoning and / or training logic 1115 is used to perform reasoning and / or training operations related to one or more embodiments. Fig.11A and / or Fig. 11B Provides details about the reasoning and / or training logic 1115. In at least one embodiment, the reasoning and / or training logic 1115 can be used to Figure 1 In a system of the present invention, an operation can be performed based at least in part on weight parameters calculated using the neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0155] In at least one embodiment, such a component can be used to perform classification or segmentation of one or more objects represented in the image data. This can include using a shape stream in parallel with the main image analysis stream to force the main stream to consider shape information of the object.

[0156] Fig. 17B According to at least one embodiment, Fig.17AA block diagram of an example system architecture of an autonomous vehicle 1700. In at least one embodiment, Fig. 17B Each of one or more components, one or more features, and one or more systems of the vehicle 1700 in FIG. 1 is shown as being connected via a bus 1702. In at least one embodiment, the bus 1702 may include, but is not limited to, a CAN data interface (referred to herein alternatively as a "CAN bus"). In at least one embodiment, the CAN may be a network inside the vehicle 1700 that helps control various features and functions of the vehicle 1700, such as actuation of brakes, acceleration, braking, steering, wipers, etc. In one embodiment, the bus 1702 may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). In at least one embodiment, the bus 1702 may be read to find steering wheel angle, ground speed, engine revolutions per minute ("RPM"), button position, and / or other vehicle status indicators. In at least one embodiment, the bus 1702 may be a CAN bus that complies with ASIL B.

[0157] In at least one embodiment, in addition to or from CAN, FlexRay and / or Ethernet can be used. In at least one embodiment, there can be any number of buses 1702, which can include but are not limited to zero or more CAN buses, zero or more FlexRay buses, zero or more Ethernet buses, and / or zero or more other types of buses using other protocols. In at least one embodiment, two or more buses 1702 can be used to perform different functions, and / or can be used for redundancy. For example, the first bus 1702 can be used for collision avoidance functions, and the second bus 1702 can be used for actuation control. In at least one embodiment, each bus 1702 can communicate with any component of the vehicle 1700, and two or more buses 1702 can communicate with the same component. In at least one embodiment, each of any number of systems on a chip ("SoC") 1704, each of one or more controllers 1736, and / or each computer within the vehicle can access the same input data (e.g., input from sensors of the vehicle 1700) and can be connected to a common bus, such as a CAN bus.

[0158] In at least one embodiment, the vehicle 1700 may include one or more controllers 1736, such as those described herein with respect to Fig.17AThose described. Controller 1736 can be used for a variety of functions. In at least one embodiment, controller 1736 can be coupled to any of the various other components and systems of vehicle 1700 and can be used to control vehicle 1700, artificial intelligence of vehicle 1700, infotainment of vehicle 1700, etc.

[0159] In at least one embodiment, the vehicle 1700 may include any number of SoCs 1704. Each of the SoCs 1704 may include, but is not limited to, a central processing unit ("CPU(s)") 1706, a graphics processing unit ("GPU(s")) 1708, one or more processors 1710, one or more caches 1712, one or more accelerators 1714, one or more data stores 1716, and / or other components and features not shown. In at least one embodiment, the one or more SoCs 1704 may be used to control the vehicle 1700 in a variety of platforms and systems. For example, in at least one embodiment, the one or more SoCs 1704 may be combined with a high-definition ("HD") map 1722 in a system (e.g., a system of the vehicle 1700), which may be downloaded from one or more servers (e.g., a system of the vehicle 1700) via a network interface 1724. Fig. 17B ) to obtain map refreshes and / or updates.

[0160] In at least one embodiment, one or more CPUs 1706 may include a CPU cluster or CPU complex (referred to herein alternatively as a "CCPLEX"). In at least one embodiment, one or more CPUs 1706 may include multiple cores and / or a secondary ("L2") cache. For example, in at least one embodiment, one or more CPUs 1706 may include eight cores in a coherent multiprocessor configuration. In at least one embodiment, one or more CPUs 1706 may include four dual-core clusters, each with a dedicated L2 cache (e.g., a 2MB L2 cache). In at least one embodiment, one or more CPUs 1706 (e.g., CCPLEX) may be configured to support simultaneous cluster operations such that any combination of clusters of one or more CPUs 1706 may be active at any given time.

[0161] In at least one embodiment, one or more CPUs 1706 may implement power management functions, including but not limited to one or more of the following features: each hardware module may be automatically clock gated when idle to save dynamic power; each core clock may be gated when the core is not actively executing instructions due to executing a wait for interrupt ("WFI") / wait for event ("WFE") instruction; each core may be independently powered; each core cluster may be independently clock gated when all cores are clock gated or power gated; and / or each core cluster may be independently power gated when all cores are power gated. In at least one embodiment, one or more CPUs 1706 may further implement an enhanced algorithm for managing power states, wherein allowed power states and expected wake-up times are specified, and hardware / microcode determines the optimal power state for cores, clusters, and CCPLEX inputs. In at least one embodiment, the processing core may support a simplified power state entry sequence in software, wherein the work is offloaded to the microcode.

[0162] In at least one embodiment, one or more GPUs 1708 may include an integrated GPU (or referred to herein as an "iGPU"). In at least one embodiment, one or more GPUs 1708 may be programmable and may be effective for parallel workloads. In at least one embodiment, one or more GPUs 1708, in at least one embodiment, may use an enhanced tensor instruction set. In at least one embodiment, one or more GPUs 1708 may include one or more streaming microprocessors, wherein each streaming microprocessor may include a level 1 ("L1") cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with a storage capacity of 512KB). In at least one embodiment, one or more GPUs 1708 may include at least eight streaming microprocessors. In at least one embodiment, one or more GPUs 1708 may use a computing application programming interface (API). In at least one embodiment, one or more GPUs 1708 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0163] In at least one embodiment, one or more GPUs 1708 may be optimized for power consumption to achieve optimal performance in automotive and embedded use cases. For example, in one embodiment, one or more GPUs 1708 may be manufactured on fin field effect transistors ("FinFETs"). In at least one embodiment, each streaming microprocessor may include multiple mixed precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In at least one embodiment, 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed precision NVIDIATENSOR COREs for deep learning matrix arithmetic, a zero-level ("L0") instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64KB register file may be allocated to each processing block. In at least one embodiment, the streaming microprocessor may include independent parallel integer and floating point data paths to provide efficient execution of workloads that mix computation and addressing operations. In at least one embodiment, the streaming microprocessor may include independent thread scheduling capabilities to achieve finer-grained synchronization and collaboration between parallel threads. In at least one embodiment, a streaming microprocessor may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0164] In at least one embodiment, one or more GPUs 1708 may include high bandwidth memory ("HBM") and / or a 16GB HBM2 memory subsystem to provide peak memory bandwidth of about 900GB / sec in some examples. In at least one embodiment, synchronous graphics random access memory ("SGRAM"), such as graphics double data rate type five synchronous random access memory ("GDDR5"), may be used in addition to or in lieu of HBM memory.

[0165] In at least one embodiment, one or more GPUs 1708 may include unified memory technology. In at least one embodiment, address translation service (“ATS”) support may be used to allow one or more GPUs 1708 to directly access one or more CPU 1706 page tables. In at least one embodiment, when one or more GPU 1708 memory management units (“MMUs”) experience a miss, an address translation request may be sent to one or more CPUs 1706. In response, in at least one embodiment, one or more CPUs 1706 may look up the virtual-to-physical mapping of the address in its page table and transmit the translation back to one or more GPUs 1708. In at least one embodiment, unified memory technology may allow a single unified virtual address space to be used for memory of both one or more CPUs 1706 and one or more GPUs 1708, thereby simplifying programming of one or more GPUs 1708 and porting applications to one or more GPUs 1708.

[0166] In at least one embodiment, one or more GPUs 1708 may include any number of access counters that may track the frequency of accesses to other processors' memory by one or more GPUs 1708. In at least one embodiment, the one or more access counters may help ensure that memory pages are moved to the physical memory of the processor that accesses the page most frequently, thereby improving the efficiency of memory ranges shared between processors.

[0167] In at least one embodiment, one or more SoCs 1704 may include any number of caches 1712, including those described herein. For example, in at least one embodiment, one or more caches 1712 may include a level 3 ("L3") cache that may be used for one or more CPUs 1706 and one or more GPUs 1708 (e.g., connecting both CPUs 1706 and GPUs 1708). In at least one embodiment, one or more caches 1712 may include a write-back cache that may track the state of a line, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). In at least one embodiment, although smaller cache sizes may be used, the L3 cache may include 4MB or more, depending on the embodiment.

[0168] In at least one embodiment, one or more SoCs 1704 may include one or more accelerators 1714 (e.g., hardware accelerators, software accelerators, or a combination thereof). In at least one embodiment, one or more SoCs 1704 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. In at least one embodiment, large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other calculations. In at least one embodiment, the hardware acceleration cluster may be used to supplement one or more GPUs 1708 and unload some tasks of one or more GPUs 1708 (e.g., freeing up more cycles of one or more GPUs 1708 to perform other tasks). In at least one embodiment, one or more accelerators 1714 may be used for target workloads (e.g., perception, convolutional neural networks ("CNN"), recursive neural networks ("RNN"), etc.), which are stable enough to accommodate acceleration. In at least one embodiment, CNNs may include regional or regional convolutional neural networks ("RCNN") and fast RCNNs (e.g., such as for object detection) or other types of CNNs.

[0169] In at least one embodiment, one or more accelerators 1714 (e.g., a hardware acceleration cluster) may include one or more deep learning accelerators ("DLA"). One or more DLAs may include, but are not limited to, one or more tensor processing units ("TPUs"), which may be configured to provide an additional 10 trillion operations per second for deep learning applications and reasoning. In at least one embodiment, one or more TPUs may be accelerators configured and optimized to perform image processing functions (e.g., for CNN, RCNN, etc.). One or more DLAs may be further optimized for a specific set of neural network types and floating point operations and reasoning. In at least one embodiment, the design of one or more DLAs may provide higher performance per millimeter than a typical general-purpose GPU, and generally greatly exceeds the performance of a CPU. In at least one embodiment, one or more TPUs may perform several functions, including supporting, for example, INT8, INT16, and FP16 data types for single instance convolution functions for features and weights, and post-processor functions. In at least one embodiment, one or more DLAs can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including, for example, but not limited to: a CNN for object recognition and detection using data from a camera sensor; a CNN for distance estimation using data from a camera sensor; a CNN for emergency vehicle detection and identification and detection using data from microphone 1796; a CNN for face recognition and vehicle owner recognition using data from a camera sensor; and / or a CNN for safety and / or security-related events.

[0170] In at least one embodiment, a DLA can perform any function of one or more GPUs 1308, and by using an inference accelerator, for example, a designer can target one or more DLAs or one or more GPUs 1708 for any function. For example, in at least one embodiment, a designer can focus the processing of CNNs and floating point operations on one or more DLAs and leave other functions to one or more GPUs 1708 and / or one or more other accelerators 1714.

[0171] In at least one embodiment, one or more accelerators 1714 (e.g., a hardware acceleration cluster) may include one or more programmable vision accelerators ("PVAs"), which may be referred to herein alternatively as computer vision accelerators. In at least one embodiment, one or more PVAs may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems ("ADAS") 1738, autonomous driving, augmented reality ("AR") applications, and / or virtual reality ("VR") applications. One or more PVAs may strike a balance between performance and flexibility. For example, in at least one embodiment, each of the one or more PVAs may include, for example, but not limited to, any number of reduced instruction set computer ("RISC") cores, direct memory access ("DMA"), and / or any number of vector processors.

[0172] In at least one embodiment, the RISC core can interact with an image sensor (e.g., an image sensor of any camera described herein), an image signal processor, and the like. In at least one embodiment, each RISC core can include any amount of memory. In at least one embodiment, the RISC core can use any of a variety of protocols, depending on the embodiment. In at least one embodiment, the RISC core can execute a real-time operating system ("RTOS"). In at least one embodiment, the RISC core can be implemented using one or more integrated circuit devices, application specific integrated circuits ("ASICs"), and / or storage devices. For example, in at least one embodiment, the RISC core can include an instruction cache and / or a tightly coupled RAM.

[0173] In at least one embodiment, the DMA may enable components of one or more PVAs to access system memory independently of one or more CPUs 1706. In at least one embodiment, the DMA may support any number of features for providing optimizations to the PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In at least one embodiment, the DMA may support up to six or more dimensions of addressing, which may include, but are not limited to, block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0174] In at least one embodiment, the vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In at least one embodiment, the PVA can include a PVA core and two vector processing subsystem partitions. In at least one embodiment, the PVA core can include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. In at least one embodiment, the vector processing subsystem can be used as the main processing engine of the PVA and can include a vector processing unit ("VPU"), an instruction cache, and / or a vector memory (e.g., "VMEM"). In at least one embodiment, the VPU can include a digital signal processor, such as a single instruction multiple data ("SIMD"), a very long instruction word ("VLIW") digital signal processor. In at least one embodiment, the combination of SIMD and VLIW can increase throughput and speed.

[0175] In at least one embodiment, each vector processor may include an instruction cache and may be coupled to a dedicated memory. As a result, in at least one embodiment, each vector processor may be configured to execute independently of the other vector processors. In at least one embodiment, the vector processors included in a particular PVA may be configured to exploit data parallelism. For example, in at least one embodiment, multiple vector processors included in a single PVA may execute the same computer vision algorithm, except on different regions of an image. In at least one embodiment, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on sequence images or portions of images. In at least one embodiment, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA, among other things. In at least one embodiment, the PVA may include additional error correction code ("ECC") memory to enhance overall system security.

[0176] In at least one embodiment, one or more accelerators 1714 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network and static random access memory ("SRAM") to provide high bandwidth, low latency SRAM for one or more accelerators 1714. In at least one embodiment, the on-chip memory may include at least 4MB of SRAM, including, for example, but not limited to, eight field-configurable memory blocks that can be accessed by both the PVA and the DLA. In at least one embodiment, each pair of memory blocks may include an advanced peripheral bus ("APB") interface, configuration circuits, controllers, and multiplexers. In at least one embodiment, any type of memory may be used. In at least one embodiment, the PVA and DLA may access the memory via a backbone network that provides high-speed access to the memory for the PVA and the DLA. In at least one embodiment, the backbone network may include an on-chip computer vision network that interconnects the PVA and the DLA to the memory (e.g., using the APB).

[0177] In at least one embodiment, the on-chip computer vision network may include an interface that determines that both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. In at least one embodiment, the interface may provide separate phases and separate channels for sending control signals / addresses / data, as well as burst-type communications for continuous data transmission. In at least one embodiment, the interface may comply with the International Organization for Standardization ("ISO") 26262 or International Electrotechnical Commission ("IEC") 61508 standards, although other standards and protocols may be used.

[0178] In at least one embodiment, one or more SoCs 1704 may include a real-time ray tracing hardware accelerator. In at least one embodiment, the real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the position and range of objects (e.g., within a world model) to generate real-time visual simulations for use in RADAR signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR systems, for general wave propagation simulations, for comparison with LIDAR data for positioning and / or other functions, and / or for other purposes.

[0179] In at least one embodiment, one or more accelerators 1714 (e.g., a hardware accelerator cluster) have a wide range of uses for autonomous driving. In at least one embodiment, the PVA can be a programmable vision accelerator that can be used in key processing stages in ADAS and autonomous vehicles. In at least one embodiment, the capabilities of the PVA are well matched to algorithmic domains that require predictable processing at low power consumption and low latency. In other words, the PVA excels in semi-intensive or intensive conventional computations, even on small data sets, which require predictable runtimes with low latency and low power consumption. In at least one embodiment, such as an autonomous vehicle (vehicle 1700), the PVA is designed to run classic computer vision algorithms because they are efficient at object detection and integer math operations.

[0180] For example, according to at least one embodiment of the technology, PVA is used to perform computer stereo vision. In at least one embodiment, a semi-global matching based algorithm can be used in some examples, although this is not meant to be limiting. In at least one embodiment, applications for Level 3-5 autonomous driving use dynamic estimation / stereo matching on the fly (e.g., structure recovery from motion, pedestrian recognition, lane detection, etc.). In at least one embodiment, PVA can perform computer stereo vision functions on input from two monocular cameras.

[0181] In at least one embodiment, the PVA can be used to perform dense optical flow. For example, in at least one embodiment, the PVA can process raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR data. In at least one embodiment, the PVA is used for time-of-flight depth processing, for example, by processing raw time-of-flight data to provide processed time-of-flight data.

[0182] In at least one embodiment, the DLA can be used to run any type of network to enhance control and driving safety, including, for example, but not limited to, a neural network, which outputs a confidence score for each object detection. In at least one embodiment, the confidence score can be expressed or interpreted as a probability, or as providing a relative "weight" of each detection relative to other detections. In at least one embodiment, the confidence score enables the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, in at least one embodiment, the system can set a threshold for the confidence score, and only detects that exceed the threshold are considered true positive detections. In an embodiment using an automatic emergency braking ("AEB") system, a false positive detection will cause the vehicle to automatically perform emergency braking, which is obviously undesirable. In at least one embodiment, a highly confident detection can be considered a trigger for AEB. In at least one embodiment, the DLA can run a neural network for regressing confidence values. In at least one embodiment, the neural network may take as its input at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), outputs of one or more IMU sensors 1766 associated with vehicle 1700 orientation, range, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., one or more LIDAR sensors 1764 or one or more RADAR sensors 1760), and the like.

[0183] In at least one embodiment, one or more SoCs 1704 may include one or more data stores 1716 (e.g., memory). In at least one embodiment, one or more data stores 1716 may be on-chip memory of one or more SoCs 1704 that may store neural networks to be executed on one or more GPUs 1708 and / or DLAs. In at least one embodiment, one or more data stores 1716 may have a capacity large enough to store multiple instances of a neural network for redundancy and safety. In at least one embodiment, one or more data stores 1716 may include an L2 or L3 cache.

[0184] In at least one embodiment, one or more SoCs 1704 may include any number of processors 1710 (e.g., embedded processors). In at least one embodiment, one or more processors 1710 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle boot power and management functions and related security implementations. In at least one embodiment, the boot and power management processor may be part of the boot sequence of one or more SoCs 1704 and may provide runtime power management services. In at least one embodiment, the boot power and management processor may provide clock and voltage programming, assist system low power state transitions, one or more SoCs 1704 thermal and temperature sensor management, and / or one or more SoCs 1704 power state management. In at least one embodiment, each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and one or more SoCs 1704 may use the ring oscillator to detect the temperature of one or more CPUs 1706, one or more GPUs 1708, and / or one or more accelerators 1714. In at least one embodiment, if the temperature is determined to exceed a threshold, the boot and power management processor can enter a temperature fault routine and place one or more SoCs 1704 into a lower power consumption state and / or place the vehicle 1700 into a driver's safe parking mode (e.g., bringing the vehicle 1700 to a safe stop).

[0185] In at least one embodiment, one or more processors 1710 may further include a set of embedded processors that can be used as an audio processing engine. In at least one embodiment, the audio processing engine can be an audio subsystem that can provide full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In at least one embodiment, the audio processing engine is a dedicated processor core that has a digital signal processor with dedicated RAM.

[0186] In at least one embodiment, one or more processors 1710 may further include an always-on processor engine that may provide the necessary hardware features to support low-power sensor management and wake-up use cases. In at least one embodiment, the processors on the always-on processor engine may include, but are not limited to, a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0187] In at least one embodiment, one or more processors 1710 may further include a safety cluster engine, which includes but is not limited to a dedicated processor subsystem for handling safety management of automotive applications. In at least one embodiment, the safety cluster engine may include but is not limited to two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.) and / or routing logic. In safe mode, in at least one embodiment, two or more cores may operate in lockstep mode and may be used as a single core with comparison logic to detect any differences between their operations. In at least one embodiment, one or more processors 1710 may further include a real-time camera engine, which may include but is not limited to a dedicated processor subsystem for handling real-time camera management. In at least one embodiment, one or more processors 1710 may further include a high dynamic range signal processor, which may include but is not limited to an image signal processor, which is a hardware engine as part of the camera processing pipeline.

[0188] In at least one embodiment, one or more processors 1710 may include a video image synthesizer, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate a final video to generate a final image for a player window. In at least one embodiment, the video image synthesizer may perform lens distortion correction on one or more wide-angle cameras 1770, one or more surround cameras 1774, and / or one or more in-cabin surveillance camera sensors. In at least one embodiment, preferably, the in-cabin surveillance camera sensors are monitored by a neural network running on another instance of SoC 1704, which is configured to identify cabin events and respond accordingly. In at least one embodiment, the in-cabin system may perform, but is not limited to, lip reading to activate cellular service and make calls, dictate emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. In at least one embodiment, certain functions are available to the driver when the vehicle is operating in autonomous mode, otherwise they are disabled.

[0189] In at least one embodiment, the video image synthesizer may include enhanced temporal noise reduction for simultaneous spatial and temporal noise reduction. For example, in at least one embodiment, where motion occurs in the video, the noise reduction appropriately weights the spatial information, thereby reducing the weight of information provided by adjacent frames. In at least one embodiment, where an image or portion of an image does not include motion, the temporal noise reduction performed by the video image synthesizer may use information from previous images to reduce noise in the current image.

[0190] In at least one embodiment, the video image compositor may also be configured to perform stereoscopic correction on the input stereoscopic lens frames. In at least one embodiment, the video image compositor may also be used for user interface composition when the operating system desktop is used, and does not require one or more GPUs 1708 to continuously render new surfaces. In at least one embodiment, when one or more GPUs 1708 are powered and actively performing 3D rendering, the video image compositor may be used to offload one or more GPUs 1708 to improve performance and responsiveness.

[0191] In at least one embodiment, one or more SoCs 1704 may further include a Mobile Industry Processor Interface ("MIPI") camera serial interface for receiving video and input from a camera, a high-speed interface, and / or a video input block that may be used for a camera and associated pixel input functionality. In at least one embodiment, one or more SoCs 1704 may further include an input / output controller that may be controlled by software and may be used to receive I / O signals that are not committed to a specific role.

[0192] In at least one embodiment, the one or more SoCs 1704 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio encoders / decoders ("codecs"), power management, and / or other devices. The one or more SoCs 1704 may be used to process data from cameras (e.g., connected via Gigabit multimedia serial links and Ethernet), sensors (e.g., one or more LIDAR sensors 1764, one or more RADAR sensors 1760, etc., which may be connected via Ethernet), data from the bus 1702 (e.g., speed of the vehicle 1700, steering wheel position, etc.), data from one or more GNSS sensors 1758 (e.g., connected via Ethernet or CAN bus), etc. In at least one embodiment, the one or more SoCs 1704 may further include dedicated high-performance mass storage controllers, which may include their own DMA engines and may be used to free the one or more CPUs 1706 from routine data management tasks.

[0193] In at least one embodiment, one or more SoC 1704 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that utilizes and effectively uses computer vision and ADAS technology to achieve diversity and redundancy, which provides a platform that can provide a flexible and reliable driving software stack and deep learning tools. In at least one embodiment, one or more SoC 1704 can be faster and more reliable than conventional systems, and even more energy efficient and space efficient. For example, in at least one embodiment, one or more accelerators 1714, when combined with one or more CPUs 1706, one or more GPUs 1708, and one or more data stores 1716, can provide a fast and effective platform for level 3-5 autonomous vehicles.

[0194] In at least one embodiment, computer vision algorithms can be executed on a CPU, which can be configured using a high-level programming language (e.g., the C programming language) to execute a variety of processing algorithms on a variety of visual data. However, in at least one embodiment, CPUs generally cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In at least one embodiment, many CPUs are unable to execute complex object detection algorithms in real time, which are used in in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0195] The embodiments described herein allow for multiple neural networks to be executed simultaneously and / or sequentially, and for the results to be combined together to achieve Level 3-5 autonomous driving functions. For example, in at least one embodiment, a CNN executed on a DLA or a discrete GPU (e.g., one or more GPUs 1720) may include text and single-word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. In at least one embodiment, the DLA may also include a neural network that is capable of recognizing, interpreting, and providing semantic understanding of symbols and passing that semantic understanding to a path planning module running on the CPU Complex.

[0196] In at least one embodiment, for level 3, 4, or 5 driving, multiple neural networks may be run simultaneously. For example, in at least one embodiment, a warning sign consisting of "Caution: flashing lights indicate icy conditions" coupled with electric lights may be interpreted by multiple neural networks independently or collectively. In at least one embodiment, the sign itself may be recognized as a traffic sign by a first deployed neural network (e.g., an already trained neural network), and the text "flashing lights indicate icy conditions" may be interpreted by a second deployed neural network, which notifies the vehicle's path planning software (preferably executing on a CPU Complex) that icy conditions exist when flashing lights are detected. In at least one embodiment, flashing lights may be recognized by operating a third deployed neural network over multiple frames, notifying the vehicle's path planning software of the presence (or absence) of flashing lights. In at least one embodiment, all three neural networks may be run simultaneously, such as within a DLA and / or on one or more GPUs 1708.

[0197] In at least one embodiment, a CNN for facial recognition and vehicle owner identification can use data from the camera sensor to identify the presence of an authorized driver and / or owner of the vehicle 1700. In at least one embodiment, the always-on sensor processor engine can be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and, in a security mode, can be used to disable the vehicle when the owner leaves the vehicle. In this way, one or more SoCs 1704 provide protection against theft and / or carjacking.

[0198] In at least one embodiment, a CNN for emergency vehicle detection and identification can use data from microphone 1796 to detect and identify emergency vehicle sirens. In at least one embodiment, one or more SoCs 1704 use CNNs to classify environmental and urban sounds, as well as classify visual data. In at least one embodiment, the CNN running on the DLA is trained to identify the relative approach speed of the emergency vehicle (e.g., by using the Doppler effect). In at least one embodiment, the CNN can also be trained to identify emergency vehicles for the area where the vehicle is operating, as identified by one or more GNSS sensors 1758. In at least one embodiment, when operating in Europe, the CNN will seek to detect European sirens, while when in the United States, the CNN will seek to identify only North American sirens. In at least one embodiment, once an emergency vehicle is detected, a control program can be used with the assistance of one or more ultrasonic sensors 1762 to execute emergency vehicle safety routines, slow down the vehicle, drive the vehicle to the side of the road, stop the vehicle, and / or idle the vehicle until one or more emergency vehicles pass.

[0199] In at least one embodiment, the vehicle 1700 may include one or more CPUs 1718 (e.g., one or more discrete CPUs or one or more dCPUs) that may be coupled to one or more SoCs 1704 via a high-speed interconnect (e.g., PCIe). In at least one embodiment, the one or more CPUs 1718 may include an X86 processor, such as one or more CPUs 1718 that may be used to perform any of a variety of functions, including, for example, arbitrating potential inconsistent results between ADAS sensors and one or more SoCs 1704, and / or monitoring the status and health of one or more controllers 1736 and / or an on-chip information system ("information SoC") 1730.

[0200] In at least one embodiment, the vehicle 1700 may include one or more GPUs 1720 (e.g., one or more discrete GPUs or one or more dGPUs) that may be coupled to one or more SoCs 1704 via a high-speed interconnect (e.g., NVIDIA's NVLINK). In at least one embodiment, the one or more GPUs 1720 may provide additional artificial intelligence functionality, such as by executing redundant and / or different neural networks, and may be used to train and / or update the neural network based at least in part on input from sensors of the vehicle 1700 (e.g., sensor data).

[0201] In at least one embodiment, the vehicle 1700 may further include a network interface 1724, which may include, but is not limited to, one or more wireless antennas 1726 (e.g., one or more wireless antennas 1726 for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). In at least one embodiment, the network interface 1724 may be used to enable wireless connection with the cloud (e.g., servers and / or other network devices), other vehicles, and / or computing devices (e.g., passenger's client devices) via the Internet. In at least one embodiment, in order to communicate with other vehicles, a direct link may be established between the vehicle 170 and the other vehicle and / or an indirect link may be established (e.g., via a network and the Internet). In at least one embodiment, a vehicle-to-vehicle communication link may be used to provide a direct link. The vehicle-to-vehicle communication link may provide the vehicle 1700 with information about vehicles near the vehicle 1700 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 1700). In at least one embodiment, the aforementioned functionality may be part of the cooperative adaptive cruise control functionality of the vehicle 1700.

[0202] In at least one embodiment, the network interface 1724 may include a SoC that provides modulation and demodulation functions and enables one or more controllers 1736 to communicate via a wireless network. In at least one embodiment, the network interface 1724 may include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. In at least one embodiment, frequency conversion may be performed in any technically feasible manner. For example, frequency conversion may be performed by a known process and / or using a superheterodyne process. In at least one embodiment, the radio frequency front end function may be provided by a separate chip. In at least one embodiment, the network interface may include a wireless function for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0203] In at least one embodiment, the vehicle 1700 may further include one or more data stores 1728, which may include, but are not limited to, off-chip (e.g., one or more off-SoC 1704) storage. In at least one embodiment, the one or more data stores 1728 may include, but are not limited to, one or more storage elements including RAM, SRAM, dynamic random access memory ("DRAM"), video random access memory ("VRAM"), flash memory, hard disk, and / or other components and / or devices that can store at least one bit of data.

[0204] In at least one embodiment, the vehicle 1700 may further include one or more GNSS sensors 1758 (e.g., GPS and / or assisted GPS sensors) to assist in mapping, perception, occupancy grid generation, and / or path planning functions. In at least one embodiment, any number of GNSS sensors 1758 may be used, including, for example but not limited to, GPS connected to a serial interface (e.g., RS-232) bridge using a USB connector with Ethernet.

[0205] In at least one embodiment, the vehicle 1700 may further include one or more RADAR sensors 1760. One or more RADAR sensors 1760 may be used by the vehicle 1700 for remote vehicle detection, even in darkness and / or inclement weather conditions. In at least one embodiment, the RADAR functional safety level may be ASIL B. One or more RADAR sensors 1760 may use CAN and / or bus 1702 (e.g., to transmit data generated by one or more RADAR sensors 1760) for control and access to object tracking data, and in some examples may access Ethernet to access raw data. In at least one embodiment, a variety of RADAR sensor types may be used. For example, but not limited to, one or more RADAR sensors 1760 may be suitable for front, rear, and side RADAR use. In at least one embodiment, one or more RADAR sensors 1760 are one or more pulse Doppler RADAR sensors.

[0206] In at least one embodiment, one or more RADAR sensors 1760 may include different configurations, such as long-range with narrow field of view, short-range with wide field of view, short-range side coverage, etc. In at least one embodiment, the long-range RADAR can be used for adaptive cruise control functions. In at least one embodiment, the long-range RADAR system can provide a wide field of view achieved by two or more independent scans (e.g., within 250m). In at least one embodiment, one or more RADAR sensors 1760 can help distinguish between static objects and moving objects, and can be used by the ADAS system 1738 for emergency braking assistance and forward collision warning. One or more sensors 1760 included in the long-range RADAR system may include, but are not limited to, a single-base multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In at least one embodiment, having six antennas, the central four antennas can create a focused beam pattern designed to record the surroundings of the vehicle 1700 at a higher speed with minimal traffic interference in adjacent lanes. In at least one embodiment, the other two antennas can provide an expanded field of view, allowing for quick detection of vehicles entering or leaving the lane of vehicle 1700 .

[0207] In at least one embodiment, as an example, the medium-range RADAR system may include, for example, a range of up to 160m (front) or 80m (rear), and a field of view of up to 42 degrees (front) or 150 degrees (rear). In at least one embodiment, the short-range RADAR system may include, but is not limited to, any number of RADAR sensors 1760 designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, in at least one embodiment, the RADAR sensor system can generate two light beams that constantly monitor the rear of the vehicle and nearby blind spots. In at least one embodiment, the short-range RADAR system can be used in the ADAS system 1738 for blind spot detection and / or lane change assistance.

[0208] In at least one embodiment, the vehicle 1700 may further include one or more ultrasonic sensors 1762. One or more ultrasonic sensors 1762 that may be positioned at the front, rear, and / or sides of the vehicle 1700 may be used for parking assistance and / or creating and updating occupancy barriers. In at least one embodiment, a variety of ultrasonic sensors 1762 may be used, and different ultrasonic sensors 1762 may be used for different detection ranges (e.g., 2.5m, 4m). In at least one embodiment, the ultrasonic sensor 1762 may operate at a functional safety level of ASIL B.

[0209] In at least one embodiment, the vehicle 1700 may include one or more LIDAR sensors 1764. The one or more LIDAR sensors 1764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. In at least one embodiment, the one or more LIDAR sensors 1764 may be functional safety level ASIL B. In at least one embodiment, the vehicle 1700 may include multiple (e.g., two, four, six, etc.) LIDAR sensors 1764 that may use Ethernet (e.g., provide data to a Gigabit Ethernet switch).

[0210] In at least one embodiment, one or more LIDAR sensors 1764 may be able to provide a list of objects and their distances for a 360-degree field of view. In at least one embodiment, one or more commercially available LIDAR sensors 1764 may, for example, have an advertised range of approximately 100m, have an accuracy of 2cm-3cm, and support 100Mbps Ethernet connections. In at least one embodiment, one or more non-obtrusive LIDAR sensors 1764 may be used. In such an embodiment, one or more LIDAR sensors 1764 may be implemented as small devices that can be embedded in the front, rear, sides, and / or corners of the vehicle 1700. In at least one embodiment, one or more LIDAR sensors 1764, in such an embodiment, may provide up to 120 degrees of horizontal field of view and 35 degrees of vertical field of view, even for low reflectivity objects, and have a range of 200m. In at least one embodiment, the forward one or more LIDAR sensors 1764 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0211] In at least one embodiment, LIDAR technology (such as 3D flash LIDAR) may also be used. 3D flash LIDAR uses laser flashes as a transmission source to illuminate approximately 200m around the vehicle 1700. In at least one embodiment, the flash LIDAR unit includes, but is not limited to, a receiver that records the laser pulse propagation time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle 1700 to the object. In at least one embodiment, the flash LIDAR can allow each laser flash to be used to generate a highly accurate and distortion-free image of the surrounding environment. In at least one embodiment, four flash LIDAR sensors can be deployed, one on each side of the vehicle 1700. In at least one embodiment, the 3D flash LIDAR system includes, but is not limited to, a solid-state 3D line-of-sight array LIDAR camera with no moving parts other than a fan (e.g., a non-scanning LIDAR device). In at least one embodiment, the flash LIDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture the reflected laser light in the form of a 3D range point cloud and co-registered intensity data.

[0212] In at least one embodiment, the vehicle may also include one or more IMU sensors 1766. In at least one embodiment, one or more IMU sensors 1766 may be located at the rear axle center of the vehicle 1700, in at least one embodiment. In at least one embodiment, one or more IMU sensors 1766 may include, for example but not limited to, one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In at least one embodiment, for example in a six-axis application, one or more IMU sensors 1766 may include but are not limited to accelerometers and gyroscopes. In at least one embodiment, for example in a nine-axis application, one or more IMU sensors 1766 may include but are not limited to accelerometers, gyroscopes, and magnetometers.

[0213] In at least one embodiment, one or more IMU sensors 1766 may be implemented as a miniature high-performance GPS-aided inertial navigation system ("GPS / INS") that combines micro-electromechanical systems ("MEMS") inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and attitude; in at least one embodiment, one or more IMU sensors 1766 may enable vehicle 1700 to estimate heading without input from a magnetic sensor by directly observing and correlating velocity changes from GPS to one or more IMU sensors 1766. In at least one embodiment, one or more IMU sensors 1766 and one or more GNSS sensors 1758 may be combined in a single integrated unit.

[0214] In at least one embodiment, the vehicle 1700 can include one or more microphones 1796 positioned within and / or around the vehicle 1700. In at least one embodiment, among other things, the one or more microphones 1796 can be used for emergency vehicle detection and identification.

[0215] In at least one embodiment, the vehicle 1700 may further include any number of camera types, including one or more stereo cameras 1768, one or more wide-angle cameras 1770, one or more infrared cameras 1772, one or more surround cameras 1774, one or more long-range cameras 1798, one or more mid-range cameras 1776, and / or other camera types. In at least one embodiment, the cameras may be used to capture image data around the entire periphery of the vehicle 1700. In at least one embodiment, the type of camera used depends on the vehicle 1700. In at least one embodiment, any combination of camera types may be used to provide the necessary coverage around the vehicle 1700. In at least one embodiment, the number of cameras may vary depending on the embodiment. For example, in at least one embodiment, the vehicle 1700 may include six cameras, seven cameras, ten cameras, twelve cameras, or another number of cameras. The cameras may, by way of example but not limitation, support Gigabit Multimedia Serial Link ("GMSL") and / or Gigabit Ethernet. In at least one embodiment, the present invention previously described with reference to FIG. Fig.17A and Figure 1 Each camera is described in more detail.

[0216] In at least one embodiment, the vehicle 1700 may further include one or more vibration sensors 1742. In at least one embodiment, one or more vibration sensors 1742 may measure vibrations of a component (e.g., an axle) of the vehicle 1700. For example, in at least one embodiment, a change in vibration may indicate a change in the road surface. In at least one embodiment, when two or more vibration sensors 1742 are used, the difference between the vibrations may be used to determine friction or slippage of the road surface (e.g., when there is a vibration difference between a powered drive shaft and a freely rotating shaft).

[0217] In at least one embodiment, the vehicle 1700 may include an ADAS system 1738. In some examples, the ADAS system 1738 may include, but is not limited to, a SoC. In at least one embodiment, the ADAS system 1738 may include, but is not limited to, any number of autonomous / adaptive / automatic cruise control ("ACC") systems, cooperative adaptive cruise control ("CACC") systems, forward collision warning ("FCW") systems, automatic emergency braking ("AEB") systems, lane departure warning ("LDW") systems, lane keeping assist ("LKA") systems, blind spot warning ("BSW") systems, rear cross traffic warning ("RCTW") systems, collision warning ("CW") systems, lane centering ("LC") systems, and / or other systems, features, and / or functions, and combinations thereof.

[0218] In at least one embodiment, the ACC system may use one or more RADAR sensors 1760, one or more LIDAR sensors 1764, and / or any number of cameras. In at least one embodiment, the ACC system may include a longitudinal ACC system and / or a lateral ACC system. In at least one embodiment, the longitudinal ACC system monitors and controls the distance to vehicles immediately adjacent to the vehicle 1700 and automatically adjusts the speed of the vehicle 1700 to maintain a safe distance from the vehicle ahead. In at least one embodiment, the lateral ACC system performs distance keeping and suggests that the vehicle 1700 change lanes when necessary. In at least one embodiment, the lateral ACC is related to other ADAS applications, such as LC and CW.

[0219] In at least one embodiment, the CACC system uses information from other vehicles, which can be received from other vehicles via a wireless link or indirectly via a network connection (e.g., via the Internet) via a network interface 1724 and / or one or more wireless antennas 1726. In at least one embodiment, a direct link can be provided by a vehicle-to-vehicle ("V2V") communication link, while an indirect link can be provided by an infrastructure-to-vehicle ("I2V") communication link. Typically, the V2V communication concept provides information about the vehicle immediately ahead (e.g., the vehicle immediately ahead of vehicle 1700 and in the same lane as it), while the I2V communication concept provides information about traffic further ahead. In at least one embodiment, the CACC system can include one or both of the I2V and V2V information sources. In at least one embodiment, given the information of the vehicle ahead of vehicle 1700, the CACC system can be more reliable and has the potential to improve the smoothness of traffic flow and reduce road congestion.

[0220] In at least one embodiment, the FCW system is designed to warn the driver of the hazard so that the driver can take corrective action. In at least one embodiment, the FCW system uses a forward-facing camera and / or one or more RADAR sensors 1760, which are coupled to a dedicated processor, DSP, FPGA and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker and / or vibration component. In at least one embodiment, the FCW system can provide warnings, such as in the form of sounds, visual warnings, vibrations and / or rapid brake pulses.

[0221] In at least one embodiment, the AEB system detects an impending forward collision with another vehicle or other object and may automatically apply brakes if the driver does not take corrective action within specified time or distance parameters. In at least one embodiment, the AEB system may use one or more forward-facing cameras and / or one or more RADAR sensors 1760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. In at least one embodiment, when the AEB system detects a hazard, the AEB system typically first warns the driver to take corrective action to avoid the collision, and, if the driver does not take corrective action, the AEB system may automatically apply brakes to attempt to prevent or at least mitigate the effects of the predicted collision. In at least one embodiment, the AEB system may include technologies such as dynamic brake support and / or braking for impending collisions.

[0222] In at least one embodiment, the LDW system provides visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to warn the driver when the vehicle 1700 crosses a lane marking. In at least one embodiment, the LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. In at least one embodiment, the LDW system may use a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to components such as a display, speaker, and / or vibration. In at least one embodiment, the LKA system is a variation of the LDW system. If the vehicle 1700 begins to leave the lane, the LKA system provides steering input or braking to correct the vehicle 1700.

[0223] In at least one embodiment, the BSW system detects that a car is in a blind spot and warns the vehicle driver. In at least one embodiment, the BSW system may provide visual, audible, and / or tactile alerts to indicate that it is unsafe to merge or change lanes. In at least one embodiment, the BSW system may provide additional warnings when the driver uses a turn signal. In at least one embodiment, the BSW system may use one or more rear-facing cameras and / or one or more RADAR sensors 1760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0224] In at least one embodiment, the RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside of the rear camera range while the vehicle 1700 is in reverse. In at least one embodiment, the RCTW system includes an AEB system to ensure that the vehicle brakes are applied to avoid a collision. In at least one embodiment, the RCTW system can use one or more rear-facing RADAR sensors 1760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0225] In at least one embodiment, conventional ADAS systems may be prone to false positive results, which may be annoying and distracting to the driver, but are generally not catastrophic because conventional ADAS systems warn the driver and allow the driver to decide whether the safety condition actually exists and take appropriate action. In at least one embodiment, in the event of conflicting results, the vehicle 1700 itself decides whether to listen to the results of the main computer or the auxiliary computer (e.g., the first controller 1736 or the second controller 1736). For example, in at least one embodiment, the ADAS system 1738 can be a backup and / or auxiliary computer for providing perception information to a backup computer rationality module. In at least one embodiment, the backup computer rationality monitor can run redundant various software on hardware components to detect faults in perception and dynamic driving tasks. In at least one embodiment, the output from the ADAS system 1738 can be provided to the monitoring MCU. In at least one embodiment, if the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU decides how to coordinate the conflict to ensure safe operation.

[0226] In at least one embodiment, the primary computer can be configured to provide a confidence score to the supervisory MCU to indicate the primary computer's confidence in the selected result. In at least one embodiment, if the confidence score exceeds a threshold, the supervisory MCU can follow the instructions of the primary computer regardless of whether the secondary computer provides conflicting or inconsistent results. In at least one embodiment, in the event that the confidence score does not meet the threshold, and in the event that the primary computer and the secondary computer indicate different results (e.g., conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0227] In at least one embodiment, the supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the auxiliary computer provides a false alarm based at least in part on outputs from the primary computer and the auxiliary computer. In at least one embodiment, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot be trusted. For example, in at least one embodiment, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system identifies metal objects that are not actually dangerous, such as drainage grates or manhole covers that would trigger an alarm. In at least one embodiment, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to override the LDW when there are cyclists or pedestrians and lane departure is actually the safest action. In at least one embodiment, the supervisory MCU may include at least one of a DLA or a GPU suitable for running a neural network with associated memory. In at least one embodiment, the supervisory MCU may include and / or be included as a component of one or more SoCs 1704.

[0228] In at least one embodiment, the ADAS system 1738 may include an auxiliary computer that performs ADAS functions using traditional computer vision rules. In at least one embodiment, the auxiliary computer may use classical computer vision rules (if-then), and the presence of a neural network in the supervisory MCU may improve reliability, safety, and performance. For example, in at least one embodiment, the diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functions. For example, in at least one embodiment, if there is a software vulnerability or error in the software running on the main computer, and the different software code running on the auxiliary computer provides the same overall result, the supervisory MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the main computer will not cause a major error.

[0229] In at least one embodiment, the output of the ADAS system 1738 can be input into the perception module of the main computer and / or the dynamic driving task module of the main computer. For example, in at least one embodiment, if the ADAS system 1738 indicates a forward collision warning due to an object directly ahead, the perception block can use this information in identifying the object. In at least one embodiment, as described herein, the auxiliary computer can have its own neural network that is trained to reduce the risk of false positives.

[0230] In at least one embodiment, the vehicle 1700 may further include an infotainment SoC 1730 (e.g., an in-vehicle infotainment system (IVI)). Although shown and described as a SoC, in at least one embodiment, the infotainment system 1730 may not be a SoC and may include, but is not limited to, two or more discrete components. In at least one embodiment, the infotainment SoC 1730 may include, but is not limited to, a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open / closed, air filter information, etc.) to the vehicle 1700. For example, the infotainment SoC 1730 may include a radio, a disk player, a navigation system, a video player, USB and Bluetooth connections, a car, an in-vehicle entertainment system, WiFi, steering wheel audio controls, hands-free voice control, a head-up display ("HUD"), an HMI display 1734, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. In at least one embodiment, the infotainment SoC 1730 may be further used to provide information (e.g., visual and / or auditory) to a user of the vehicle, such as information from an ADAS system 1738, autonomous driving information (such as planned vehicle maneuvers), trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0231] In at least one embodiment, the infotainment SoC 1730 may include any number and type of GPU functionality. In at least one embodiment, the infotainment SoC 1730 may communicate with other devices, systems, and / or components of the vehicle 1700 via a bus 1702 (e.g., CAN bus, Ethernet, etc.). In at least one embodiment, the infotainment SoC 1730 may be coupled to a supervisory MCU so that the GPU of the infotainment system may perform some autonomous driving functions in the event of a failure of a main controller 1736 (e.g., a main computer and / or backup computer of the vehicle 1700). In at least one embodiment, the infotainment SoC 1730 may cause the vehicle 1700 to enter a driver-to-safety stop mode, as described herein.

[0232] In at least one embodiment, the vehicle 1700 may further include a dashboard 1732 (e.g., a digital dashboard, an electronic dashboard, a digital instrument panel, etc.). In at least one embodiment, the dashboard 1732 may include, but is not limited to, a controller and / or a supercomputer (e.g., a discrete controller or a supercomputer). In at least one embodiment, the dashboard 1732 may include, but is not limited to, any number and combination of a set of instruments, such as a speedometer, a fuel level, an oil pressure, a tachometer, an odometer, a turn indicator, a shift position indicator, one or more seat belt warning lights, one or more parking brake warning lights, one or more engine fault lights, supplementary restraint system (e.g., airbag) information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 1730 and the dashboard 1732. In at least one embodiment, the dashboard 1732 may be included as part of the infotainment SoC 1730, and vice versa.

[0233] The reasoning and / or training logic 1115 is used to perform reasoning and / or training operations associated with one or more embodiments. Fig.11A and / or Fig. 11B Provides details about the reasoning and / or training logic 1115. In at least one embodiment, the reasoning and / or training logic 1115 may be in the system Fig. 17B to perform inference or prediction operations based at least in part on weight parameters computed using neural network training operations, neural network functions and / or architectures, or neural network use cases described herein.

[0234] In at least one embodiment, such a component can be used to perform classification or segmentation of one or more objects represented in the image data. This can include using a shape stream in parallel with the main image analysis stream to force the main stream to consider shape information of the object.

[0235] Fig. 17CAccording to at least one embodiment, a cloud-based server and Fig.17A 1700. In at least one embodiment, system 1776 may include, but is not limited to, one or more servers 1778, one or more networks 1790, and any number and type of vehicles, including vehicle 1700. In at least one embodiment, one or more servers 1778 may include, but are not limited to, multiple GPUs 1784(A)-1784(H) (collectively referred to herein as GPUs 1784), PCIe switches 1782(A)-1782(D) (collectively referred to herein as PCIe switches 1782), and / or CPUs 1780(A)-1780(B) (collectively referred to herein as CPUs 1780), GPUs 1784, CPUs 1780, and PCIe switches 1782 may be interconnected with high-speed connections, such as, but not limited to, NVLink interface 1788 and / or PCIe connections 1786 developed by NVIDIA. In at least one embodiment, the GPU 1784 is connected via NVLink and / or NVSwitch SoC, and the GPU 1784 and PCIe switch 1782 are connected via a PCIe interconnect. In at least one embodiment, although eight GPUs 1784, two CPUs 1780, and four PCIe switches 1782 are shown, this is not intended to be limiting. In at least one embodiment, each of the one or more servers 1778 may include, but is not limited to, any combination of any number of GPUs 1784, CPUs 1780, and / or PCIe switches 1782. For example, in at least one embodiment, one or more servers 1778 may each include eight, sixteen, thirty-two, and / or more GPUs 1784.

[0236] In at least one embodiment, one or more servers 1778 may receive image data representing an image from a vehicle via one or more networks 1790, the image showing an unexpected or changed road condition, such as a recently started road project. In at least one embodiment, one or more servers 1778 may transmit a neural network 1792, an updated neural network 1792, and / or map information 1794, including but not limited to information about traffic and road conditions, via one or more networks 1790 and to the vehicle. In at least one embodiment, updates to the map information 1794 may include but are not limited to updates to the HD map 1722, such as information about construction sites, potholes, access roads, flooding, and / or other obstacles. In at least one embodiment, the neural network 1792, the updated neural network 1792, and / or the map information 1794 may be generated by new training and / or experience represented in data received from any number of vehicles in the environment, and / or based at least on training performed at a data center (e.g., using one or more servers 1778 and / or other servers).

[0237] In at least one embodiment, one or more servers 1778 can be used to train a machine learning model (e.g., a neural network) based at least in part on the training data. In at least one embodiment. The training data can be generated by the vehicle, and / or can be generated in simulation (e.g., using a game engine). In at least one embodiment, any amount of the training data is labeled (e.g., where the associated neural network benefits from supervised learning) and / or undergoes other preprocessing. In at least one embodiment, no amount of the training data is labeled and / or preprocessed (e.g., where the associated neural network does not require supervised learning). In at least one embodiment, once the machine learning model is trained, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via one or more networks 1790, and / or the machine learning model can be used by one or more servers 1778 to remotely monitor the vehicle.

[0238] In at least one embodiment, one or more servers 1778 may receive data from the vehicle and apply the data to the latest real-time neural network for real-time intelligent reasoning. In at least one embodiment, one or more servers 1778 may include a deep learning supercomputer and / or a dedicated AI computer powered by one or more GPUs 1784, such as the DGX and DGX Station machines developed by NVIDIA. However, in at least one embodiment, one or more servers 1778 may include a deep learning infrastructure of a data center powered by a CPU.

[0239] In at least one embodiment, the deep learning infrastructure of one or more servers 1778 may be capable of fast, real-time inference, and may use this capability to assess and validate the health of the processor, software, and / or associated hardware in the vehicle 1700. For example, in at least one embodiment, the deep learning infrastructure may receive periodic updates from the vehicle 1700, such as an image sequence and / or objects that the vehicle 1700 has located in that image sequence (e.g., via computer vision and / or other machine learning object classification techniques). In at least one embodiment, the deep learning infrastructure may run its own neural network to identify objects and compare them to those identified by the vehicle 1700, and, if the results do not match and the deep learning infrastructure concludes that the AI ​​in the vehicle 1700 is malfunctioning, the one or more servers 1778 may send a signal to the vehicle 1700 to instruct the fail-safe computer of the vehicle 1700 to take control, notify passengers, and complete a safe parking maneuver.

[0240] In at least one embodiment, one or more servers 1778 may include one or more GPUs 1784 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT 3). In at least one embodiment, the combination of GPU-driven servers and inference acceleration can enable real-time response. In at least one embodiment, for example, in situations where performance is not critical, servers driven by CPUs, FPGAs, and other processors can be used for inference. In at least one embodiment, inference and / or training logic 1115 is used to execute one or more embodiments. Fig.11A and / or Fig. 11B Provide details about the inference and / or training logic 1115.

[0241] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described in detail above. However, it should be understood that there is no intention to limit the disclosure to one or more specific forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure as defined by the appended claims.

[0242] Unless otherwise noted, unless otherwise noted or clearly contradicted by context, the use of the terms "a", "an" and "the" and similar references in the context of describing the disclosed embodiments (particularly in the context of the appended claims) should be interpreted as covering the singular and the plural, rather than as a definition of a term. The terms "comprising", "having", "including" and "containing" should be interpreted as open-ended terms (meaning "including but not limited to"). The term "connected" when unmodified refers to a physical connection and should be understood as part or all of being contained in, connected to or connected together, even if there are any intervening. Unless otherwise noted herein, the reference to numerical ranges herein is intended only to be used as a shorthand method of referring to each individual value falling within the range individually, and each individual value is incorporated into the specification as if it were recited individually herein. Unless otherwise indicated or contradicted by context, the use of the term "set" (e.g., "a set of items") or "subset" should be interpreted as a non-empty set including one or more members. In addition, unless otherwise indicated or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but a subset and a corresponding set may be equal.

[0243] Unless otherwise expressly indicated or clearly contradicted by context, conjunction language such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C" is understood in context to be generally used to refer to items, and the term etc. can be A or B or C, or any non-empty subset of the set A and B and C. For example, in a set having three members, the conjunction phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such conjunction language is not generally intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. In addition, unless otherwise indicated or contradicted by context, the term "plurality" refers to a plural state (e.g., "plurality of items" means a plurality of items). The number of items in the plural is at least two, but can be plural when expressly or by context indicates. Further, the phrase "based on" means "based at least in part on" rather than "based solely on" unless stated otherwise or clear from the context.

[0244] Unless otherwise noted herein or clearly contradictory to the environment, the operations of the processes described herein may be performed in any suitable sequence. In at least one embodiment, processes such as those described herein (or their variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions, and are implemented as codes (e.g., executable instructions, one or more computer programs, or one or more applications) that are jointly executed on one or more processors by hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program, which includes a plurality of instructions that can be executed by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that does not include a transient signal (e.g., a transient electrical or electromagnetic transmission of a propagation), but includes a non-transitory data storage circuit (e.g., a buffer, a cache, and a queue) in a transceiver of a transient signal. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, and when executed (i.e., due to being executed) by one or more processors of a computer system, the computer system performs the operations described herein. In at least one embodiment, a set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media and one or more single non-transitory storage media of the plurality of non-transitory computer-readable storage media lacking all codes, while the plurality of non-transitory computer-readable storage media collectively store all codes. In at least one embodiment, the executable instructions are executed so that different instructions are executed by different processors, for example, a non-transitory computer-readable storage medium stores instructions, and a main central processing unit ("CPU") executes some instructions, while a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and different processors execute different subsets of instructions.

[0245] Thus, in at least one embodiment, a computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and / or software capable of implementing the operations. In addition, a computer system that implements at least one embodiment of the present disclosure is a single device, and in another embodiment, is a distributed computer system that includes multiple devices that operate in different ways, so that the distributed computer system performs the operations described herein, and so that a single device does not perform all operations.

[0246] The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate embodiments of the present disclosure and does not limit the scope of the disclosure unless otherwise stated. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0247] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference.

[0248] In the description and claims, the terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms may not be intended as synonyms for each other. On the contrary, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0249] Unless otherwise specifically stated, it is understood that throughout the specification, terms such as "processing", "calculating", "computing", "determining", etc. refer to the actions and / or processes of a computer or computing system or similar electronic computing device to process and / or convert data represented as physical quantities (e.g., electronic) in registers and / or memories of the computing system into other data similarly represented as physical quantities in the memory, registers or other such information storage, transmission or display devices of the computing system.

[0250] In a similar manner, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Moreover, each process may refer to multiple processes to execute instructions continuously or intermittently, sequentially or in parallel. Because a system may embody one or more methods and a method may be considered a system, the terms "system" and "method" are used interchangeably herein.

[0251] In this document, reference may be made to obtaining, acquiring, receiving or inputting analog or digital data into a subsystem, a computer system or a computer-implemented machine. The process of obtaining, acquiring, receiving or inputting analog and digital data may be accomplished in a variety of ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In some embodiments, the process of obtaining, acquiring, receiving or inputting analog or digital data may be accomplished by transmitting data via a serial or parallel interface. In another embodiment, the process of obtaining, acquiring, receiving or inputting analog or digital data may be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. Reference may also be made to providing, outputting, transmitting, sending or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending or presenting analog or digital data may be accomplished by transmitting data as an input or output parameter of a function call, an application programming interface or an inter-process communication mechanism.

[0252] Although the above discussion sets forth example implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of the present disclosure. In addition, although specific responsibility allocations are defined above for discussion purposes, various functions and responsibilities may be allocated and divided in different ways depending on the circumstances.

[0253] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.

Claims

1. A computer-implemented method comprising: inferring boundary data of one or more objects represented in the image; inferring semantic data of the image at least partially in parallel with inferring the boundary data; as well as The boundary data is fused with the semantic data to infer a segmentation and an object type corresponding to at least one of the one or more objects.

2. The computer-implemented method of claim 1 , further comprising: A multi-stream convolutional neural network (CNN) is used to infer the boundary data in parallel with the mainstream inference of the semantic data.

3. The computer-implemented method of claim 2, further comprising: The boundary data is fused with the semantic data at one or more layers of the CNN.

4. The computer-implemented method of claim 2, wherein: The intermediate layers of the CNN are connected using multiple gated convolutional layers.

5. The computer-implemented method of claim 1 , further comprising: The boundary data is fused with the semantic data using layers of a fusion network.

6. The computer-implemented method of claim 2, further comprising: Boundary data is inferred by successive convolutional layers of a shape stream by identifying and removing portions of the image that do not correspond to the boundary data.

7. The computer-implemented method of claim 1 , further comprising: The segmentation is inferred based on a confidence value associated with the neural network and at least one of the one or more objects to execute at least one voice-activated input command.

8. The computer-implemented method of claim 1 , further comprising: The segmentation is inferred using a deep learning accelerator DLA for advanced driver assistance systems ADAS, autonomous driving systems, augmented reality AR applications, and virtual reality VR applications.

9. The computer-implemented method of claim 1 , further comprising: The semantic data of the image is inferred based on a deep neural network trained using a subset of the training data from the simulation.

10. The computer-implemented method of claim 1, further comprising: At least one action is determined based on the boundary data, the action associated with at least one of an autonomous vehicle, a robotic arm, and a safety alarm.

11. A processor, comprising: One or more circuits for: receiving image data of a captured environment; inferring boundary data for one or more objects represented in the image data; inferring semantic data for the image data at least partially in parallel with inferring the boundary data; as well as The boundary data is fused with the semantic data to infer a segmentation and an object type corresponding to at least one of the one or more objects.

12. The processor of claim 11, wherein: The one or more circuits are also configured to: A multi-stream convolutional neural network (CNN) is used to infer the boundary data in parallel with the mainstream inference of the semantic data.

13. The processor of claim 11, wherein: The one or more circuits are also configured to: The segmentation is inferred based on a confidence value associated with the neural network and at least one of the one or more objects for executing at least one voice activated input command.

14. The processor of claim 11, wherein: The one or more circuits are also configured to: The segmentation is inferred using a deep learning accelerator DLA for advanced driver assistance systems ADAS, autonomous driving systems, augmented reality AR applications, and virtual reality VR applications.

15. The processor of claim 11, wherein: The one or more circuits are also configured to: The semantic data of the image is inferred based on a deep neural network trained using a subset of the training data from the simulation.

16. The processor of claim 11, wherein: The one or more circuits are also configured to: At least one action is determined based on the boundary data, the action associated with at least one of an autonomous vehicle, a robotic arm, and a safety alarm.

17. A control system for an automation device, comprising: camera; Control mechanisms; at least one processor; as well as a memory comprising instructions that, when executed by the at least one processor, cause the control system to: capturing, by the camera, at least one image of an environment associated with the automation device; inferring boundary data for one or more objects represented in the at least one image; inferring semantic data for the at least one image at least partially in parallel with inferring the boundary data; as well as The boundary data is fused with the semantic data to infer a segmentation and an object type corresponding to at least one of the one or more objects.

18. The control system of claim 17, wherein: When the instructions are executed, the system also causes: A convolutional neural network (CNN) is used to infer the boundary data in parallel with the mainstream inference of the semantic data.

19. The control system of claim 17, wherein: When the instructions are executed, the system also causes: The segmentation is inferred based on a confidence value associated with the neural network and at least one of the one or more objects for executing at least one voice activated input command.

20. The control system of claim 17, wherein: When the instructions are executed, the system also causes: Based on the inferred segmentation, a navigation path associated with the current position and the current orientation of the automation device is adjusted.

Images