Low-cost models for optical flow estimation and stereo matching based on softmax with norm folding

Abstract

An apparatus includes a memory configured to store input data, and further includes one or more processors configured to process the input data using a machine learning model that incorporates a softmax with a norm folding mechanism.

Description

[0001] I. Cross-references to related applications

[0002] This application claims the benefit of priority to jointly owned U.S. non-provisional patent application No. 18 / 583,185, filed February 21, 2024, the entire contents of which are expressly incorporated herein by reference.

[0003] II. Technical Field

[0004] This disclosure relates in general to low-cost models applicable to optical flow estimation and stereo matching.

[0005] III. Relevant Technical Descriptions

[0006] Technological advancements have led to smaller and more powerful computing devices. For example, a wide variety of portable personal computing devices exist today, including small, lightweight, and easily portable cordless phones (such as mobile and smartphones, tablets, and laptops). These devices can transmit voice and data packets over wireless networks. Furthermore, many of these devices incorporate additional functionality, such as digital still cameras, digital camcorders, digital recorders, and audio file players. Moreover, such devices can process executable instructions, including software applications such as web browser applications that can be used to access the internet. Therefore, these devices can include significant computing power.

[0007] Such computing devices typically incorporate functionality for performing multi-image processing techniques that involve comparisons between pairs of images, such as using optical flow techniques to track object motion between consecutive video frames or estimating depth information based on the disparity between corresponding points in a pair of stereo images (referred to in this paper as stereo depth (DFS)). However, conventional methods for OF and DFS estimation often have large memory requirements or "uses," which can make such techniques impractical for use in resource-constrained environments.

[0008] As an example, methods for OF and DFS estimation conventionally rely on the use of cost volumes, which are large data structures used to store values ​​associated with cost calculations for individual paths, displacements, or disparities. As a simplified example, DFS estimation uses stereo matching for the left and right images and includes performing feature extraction to generate values ​​with height (H) and width (W). L The feature map for the left image has dimensions of H and W. R The feature map for the right image is generated in three dimensions. A volume is generated that stores the three dimensions (H, W). R W LInteractive data (such as correlation values) are then processed to determine the disparity (the pixel offset between the left and right images) associated with each row of the image. For OF estimation, data with dimension H is used to store... t W t The feature map of the first image "t" has dimension H t+1 W t+1 The cost of the interaction data between the feature maps of the next image "t+1" is even greater and requires storing four-dimensional (H) data. t W t H t+1 W t+1 Interactive data. Both DFS and OF require additional memory for intermediate computations.

[0009] In a concrete example, the evaluation of the softmax operation in a conventional model involves performing multiple passes on the input data. For instance, since the softmax operation involves generating a series of normalized values ​​based on the input tensor (such as a matrix of input data), the first pass is used to generate the series of values ​​based on the input data, and the second pass is used to normalize the values ​​based on the sum of the intermediate values. This multi-pass evaluation requires buffering the input tensor to sequentially traverse the tensor entries and also creates "breakpoints" in the computation pipeline that prevent further processing until the multi-pass evaluation is complete. When using softmax in attention mechanisms (such as in transformers), the impact of multi-pass softmax evaluation increases because the input tensor involves quadratic complexity. For illustration, when the evaluation is associated with two images and each has... N order (e.g.) O ( N When dealing with two sets of input data of different sizes, the complexity associated with processing the input tensor is... O ( N 2 ).

[0010] Some devices, such as graphics processing units (GPUs) that are specialized for massive computations, may include large enough on-chip memory for the cost bulk and intermediate computations required for DFS and OF. However, such devices are expensive and power-intensive, and therefore impractical for use in resource-constrained environments such as mobile phones or extended reality (XR) headsets.

[0011] IV. Summary of the Invention

[0012] According to a specific implementation of the technology disclosed herein, a device includes a memory configured to store input data. The device also includes one or more processors configured to process the input data using a machine learning (ML) model incorporating a softmax mechanism with norm folding.

[0013] According to a specific implementation of the technology disclosed herein, a method includes: obtaining input data at a device. The method further includes: processing the input data at the device using a machine learning (ML) model, including performing a softmax operation with norm folding.

[0014] According to a specific implementation of the technology disclosed herein, a non-transitory computer-readable medium storage instruction, when executed by one or more processors, causes the one or more processors to: obtain input data and process the input data using a machine learning (ML) model that incorporates a softmax with a norm folding mechanism.

[0015] Other specific embodiments, advantages, and features of this disclosure will become apparent upon examination of the entire application, which includes the following parts: description of drawings, detailed description, and claims.

[0016] V. Attached Figure Description

[0017] Figure 1 This is a block diagram illustrating specific implementations of a system capable of performing input data processing using a low-cost model, based on some examples of this disclosure.

[0018] Figure 2 These are examples illustrating some aspects of this disclosure. Figure 1 A block diagram illustrating examples of components and operations implemented in the system.

[0019] Figure 3 These are examples illustrating some aspects of this disclosure. Figure 1 A block diagram illustrating examples of components and operations implemented in the system.

[0020] Figure 4 Examples based on this disclosure are available. Figure 1 A block diagram of an example low-cost model with a stereo depth or optical flow architecture implemented in the system.

[0021] Figure 5 This is a block diagram illustrating a specific implementation of an integrated circuit capable of performing input data processing using a low-cost model, based on some examples of this disclosure.

[0022] Figure 6These are illustrations of specific implementations of portable electronic devices capable of performing input data processing using a low-cost model, based on some examples of this disclosure.

[0023] Figure 7 This is an illustration of a camera that, according to some examples of this disclosure, is capable of performing input data processing using a low-cost model.

[0024] Figure 8 These are illustrations of wearable electronic devices that are capable of performing input data processing using a low-cost model, based on some examples of this disclosure.

[0025] Figure 9 These are illustrations of extended reality devices (such as extended reality glasses) that are capable of performing input data processing using low-cost models, based on some examples of this disclosure.

[0026] Figure 10 The diagram illustrates, according to some examples of this disclosure, a head-mounted device (such as a virtual reality, mixed reality, or extended reality head-mounted device) capable of performing input data processing using a low-cost model.

[0027] Figure 11 This is a diagram illustrating a voice-controlled speaker system, based on some examples of this disclosure, capable of performing input data processing using a low-cost model.

[0028] Figure 12 This is an illustration of a first example of a vehicle capable of performing input data processing using a low-cost model, based on some examples of this disclosure.

[0029] Figure 13 This is a diagram illustrating a second example of a vehicle capable of performing input data processing using a low-cost model, based on some examples of this disclosure.

[0030] Figure 14 This is a diagram illustrating a specific implementation of a method for performing input data processing using a low-cost model, based on some examples of this disclosure.

[0031] Figure 15 This is a block diagram of a specific exemplary example of a device that is operable to perform input data processing using a low-cost model, based on some examples of this disclosure.

[0032] VI. Detailed Implementation

[0033] Systems and methods for performing input data processing using low-cost models are disclosed. Conventional methods for performing techniques such as OF and DFS estimation often have large memory consumption, making such techniques impractical for resource-constrained environments. For example, methods for OF and DFS estimation conventionally rely on the use of cost volumes, which are large data structures used to store values ​​associated with cost calculations for individual paths, displacements, or disparities. Some such conventional methods may include evaluation of a softmax operation, which involves performing multiple iterations on the input data (which may require significant buffering) and creating breakpoints in the computation pipeline that prevent further processing until the multiple iterations of evaluation are complete.

[0034] The disclosed techniques enable input data processing (such as for performing OF and DFS estimation) with significantly reduced memory usage compared to conventional methods. The relatively low-usage model using the techniques according to this disclosure enables efficient processing for applications such as OF and DFS estimation in resource-constrained environments.

[0035] According to the technology disclosed herein, instead of using a cost body to perform conventional OF or DFS processing, a machine learning model including an attention mechanism generates probabilistic geometric measures without generating a cost body data structure. For illustration, according to the first aspect, regression is performed on geometric coordinates using the probabilistic measures instead of a cost body. Eliminating the cost body reduces memory usage associated with OF or DFS processing.

[0036] According to the second aspect, the disclosed technique improves the computation of the softmax computation path in the attention mechanism. For example, a norm folding mechanism is incorporated into the softmax computation path, which applies a normalization factor at a later point in the computation path compared to conventional multi-pass techniques. Therefore, the data flow in the attention operation remains "streamlined" (i.e., even for large input data sizes, buffering or execution breakpoints are not required), and depth-first computation is achieved without the need for a softmax memory buffer.

[0037] According to the third aspect, the disclosed techniques include using instantaneous regression for OF or DFS geometric coordinates. For example, each matrix in the attention mechanism has linear space complexity with respect to the token size, such as... O ( N ). and requires secondary memory buffers (e.g., O ( N 2Compared to conventional attention mechanisms, matrix multiplication is performed by using appropriate matrix partitioning of the rows of one matrix and the columns of another, eliminating the need for secondary buffering in memory. Depth-first search can also be applied to reduce buffering in the computational pipeline.

[0038] Combining the above aspects into a single, low-usage model for processing input data (such as for OF or DFS estimation) can significantly enhance the operation of the device implementing the model. Furthermore, operation can be enhanced in devices implementing fewer than all of the above aspects. Specifically, each of the above aspects independently improves the performance of OF or DFS estimation by reducing memory usage, which reduces read / write accesses that could otherwise result from overflow memory accesses when memory usage exceeds the available on-chip memory capacity. Reducing overflow memory accesses directly leads to reduced latency and power consumption, and thus improves the performance of devices implementing one or more of the disclosed techniques.

[0039] While the description below primarily relates to examples of the techniques of this disclosure in the context of using OF and DFS estimation, the techniques of this disclosure are not limited to OF and DFS applications. For example, the techniques of this disclosure can also be used to reduce memory usage and improve the performance of applications that may incorporate attention and / or softmax mechanisms, such as large language models, transformer models (such as for computer vision), diffusion models (such as for text-to-image generative models), etc. For illustrative purposes, in some embodiments, the techniques of this disclosure are used to improve the performance of large language models (LLM), large vision models (LVM), large multimodal models (LMM), or combinations thereof.

[0040] Specific aspects of this disclosure are described below with reference to the accompanying drawings. In this description, common features are designated by common reference numerals. As used herein, various terms are used only for the purpose of describing particular embodiments and are not intended to limit the scope of the embodiments. For example, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context clearly indicates otherwise. Furthermore, some features described herein are singular in some embodiments and plural in others. For illustrative purposes, Figure 1 It describes a system that includes one or more processors ( Figure 1 The device 102 (with the “processor” 116) indicates that in some embodiments, device 102 includes a single processor 116, and in other embodiments, device 102 includes multiple processors 116. For ease of reference herein, such features are generally introduced as “one or more” features and are subsequently referred to in the singular or optional plural form (as indicated by “(multiple)” in the feature name) unless describing an aspect relating to multiple features in a feature.

[0041] In some accompanying figures, multiple instances of a particular type of feature are used. Although these features are physically and / or logically different, the same reference numerals are used for each feature, and these different instances are distinguished by adding letters to the reference numerals. When features are referred to herein as a group or a type (e.g., when a specific feature among these features is not mentioned), reference numerals are used without distinguishing letters. However, when a specific feature of multiple features of the same type is referred to herein, reference numerals are used with distinguishing letters. For example, reference... Figure 4 The diagram illustrates multiple convolutional neural networks (CNNs), which are associated with reference numerals 402A and 402B. When referring to a specific CNN (such as CNN 402A), the distinguishing letter "A" is used. However, when referring to any single CNN or to these CNNs as a group, the indistinguishable reference numeral 402 is used.

[0042] As used herein, the term "comprising" may be used interchangeably with "including". Additionally, it should be understood that the term "wherein" may be used interchangeably with "where". As used herein, "exemplary" may indicate an example, specific implementation, and / or aspect, and should not be construed as restrictive or indicating a preference or preferred implementation. As used herein, ordinal terms used to modify elements (such as structures, components, operations, etc.) (e.g., "first", "second", "third", etc.) do not themselves indicate any priority or order of that element relative to another element, but merely distinguish that element from another element with the same name (but using ordinal terms). As used herein, the term "set" refers to one or more specific elements among specific elements, while the term "multiple" refers to multiple (e.g., two or more) specific elements.

[0043] As used herein, “coupling” can include “communicationally coupled,” “electrically coupled,” or “physically coupled,” and may also (or alternatively) include any combination thereof. Two devices (or components) may be coupled directly or indirectly (e.g., communicationally coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). As an illustrative, non-limiting example, two electrically coupled devices (or components) may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling. In some specific implementations, two communicationally coupled (such as electrical communication) devices (or components) may transmit and receive signals (e.g., digital or analog signals) directly or indirectly via one or more wires, buses, networks, etc. As used herein, “direct coupling” can include two devices coupled without intermediate components (e.g., communicationally coupled, electrically coupled, or physically coupled).

[0044] In this disclosure, terms such as “obtain,” “determine,” “calculate,” “estimate,” “transfer,” and “adjust” are used to describe how one or more operations are performed. It should be noted that such terms should not be construed as restrictive, and similar operations can be performed using other techniques. Additionally, as mentioned herein, “obtain,” “generate,” “calculate,” “estimate,” “use,” “select,” “access,” and “determine” are used interchangeably. For example, “obtain,” “generate,” “calculate,” “estimate,” or “determine” a parameter (or signal) can refer to actively generating, estimating, calculating, or determining the parameter (or signal), or it can refer to using, selecting, retrieving, receiving, or accessing a parameter (or signal) such as one already generated by another component or device.

[0045] As used herein, the term “machine learning” should be understood to have any of its usual and conventional meanings within the fields of computer science and data science. Such meanings include, for example, processes or techniques by which one or more computers can learn to perform certain operations or functions without being explicitly programmed to do so. As a typical example, machine learning can be used to enable one or more computers to analyze data to identify patterns in the data and generate results based on that analysis. For some types of machine learning, the generated results include data that indicates the underlying structure or patterns of the data itself. For example, such techniques include so-called “clustering” techniques, which identify clusters (e.g., groupings of data elements).

[0046] For some types of machine learning, the resulting output includes a data model (also known as a "machine learning model" or simply a "model"). Typically, a model is generated using a first dataset to facilitate analysis on a second dataset. For example, the first portion of a large dataset can be used to generate a model that can then be used to analyze the remaining portion of the large dataset. As another example, historical datasets can be used to generate models that can be used to analyze future data.

[0047] Because a model can be used to evaluate datasets different from those used to generate the model, it can be viewed as a type of software (e.g., instructions, parameters, or both) automatically generated by a computer during the machine learning process. Therefore, the model can be transferable (e.g., it can be generated at a first computer and subsequently moved to a second computer for further training, use, or both). Additionally, the model can be combined with one or more other models to perform a desired analysis. For example, first data can be provided as input to a first model to generate first model output data, and the first model output data (alone, with the first data, or with other data) can be provided as input to a second model to generate second model output data indicative of the results of the desired analysis. Depending on the analysis and data involved, different combinations of models can be used to generate such results. In some examples, multiple models can provide model outputs that are input to a single model. In some examples, a single model provides model outputs as input to multiple models.

[0048] Examples of machine learning models include, but are not limited to, perceptrons, neural networks, support vector machines, regression models, decision trees, Bayesian models, Boltzmann machines, adaptive neurofuzzy inference systems, and combinations, sets, and variations of these and other types of models. Variations of neural networks include, for example, but not limited to, prototype networks, autoencoders, transformers, self-focused networks, convolutional neural networks, deep neural networks, deep belief networks, etc. Variations of decision trees include, for example, but not limited to, random forests, boosted decision trees, etc.

[0049] Since machine learning models are generated by computers based on input data, they can be discussed within at least two distinct time windows: the creation / training phase and the runtime phase. During the creation / training phase, a model is created, trained, adapted, validated, or otherwise configured by a computer based on input data (often referred to as "training data" during the creation / training phase). It's important to note that a trained model corresponds to software that has been generated and / or refined during the creation / training phase to perform a specific operation (such as classification, prediction, encoding, or other data analysis or data synthesis operations). During the runtime phase (or "inference" phase), the model is used to analyze the input data to generate model outputs. The content of the model outputs depends on the type of model. For example, as a non-limiting example, a model can be trained to perform a classification task or a regression task. In some implementations, the model may be updated continuously, periodically, or occasionally, in which case training time and runtime may be interleaved, or one form of the model may be used for inference while a copy is updated, and subsequently, the updated copy may be deployed for inference.

[0050] In some implementations, machine learning techniques are used to train (or retrain) a previously generated model. In this context, "training" refers to adapting a model or its parameters to a specific dataset. Unless otherwise clearly understood from the specific context, the term "training" as used herein includes "retraining" or refining a model for a specific dataset. For example, training may include so-called "transfer learning." In transfer learning, a base model is trained using a general or typical dataset, and subsequently refined (e.g., retrained or further trained) the base model using a more specific dataset.

[0051] The dataset used during training is called the "training dataset" or simply "training data." The dataset can be labeled or unlabeled. "Labeled data" refers to data that has been assigned classification labels indicating the groups or categories associated with the data, and "unlabeled data" refers to unlabeled data. Typically, "supervised machine learning processes" use labeled data to train machine learning models, while "unsupervised machine learning processes" use unlabeled data; however, it should be understood that the labels associated with the data are simply another data element that can be used in any appropriate machine learning process. For example, many clustering operations can be performed using unlabeled data; however, such clustering operations can use labeled data by ignoring the labels assigned to the data or by treating the labels in the same way as other data elements.

[0052] Training a model on a training dataset typically involves modifying the model's parameters with the goal of making the model's output possess specific characteristics based on the data input to the model. To distinguish it from model generation operations, model training may be referred to as optimization or optimization training in this paper. In this context, "optimization" refers to improving a metric, not necessarily finding an ideal value for that metric (e.g., a global maximum or minimum). Examples of optimization trainers include, but are not limited to, backpropagation trainers, derivative-free optimizers (DFO), and extreme learning machines (ELM). As an example of training a model, during supervised training of a neural network, input data samples are associated with labels. When input data samples are fed to the model, the model generates output data, comparing that output data with the labels associated with the input data samples to generate error values. The model's parameters are modified to attempt to reduce (e.g., optimize) the error values. As another example of training a model, during unsupervised training of an autoencoder, data samples are fed as input to the autoencoder, and the autoencoder reduces the dimensionality of the data samples (a lossy operation) and attempts to reconstruct the data samples into output data. In this example, the output data is compared with the input data samples to generate the reconstruction loss, and the parameters of the autoencoder are modified to attempt to reduce (e.g., optimize) the reconstruction loss.

[0053] refer to Figure 1 This describes a specific exemplary aspect of system 100, which includes device 102 configured to perform input data processing using a low-cost model. For example, device 102 is configured to process input data 122 using a machine learning (ML) model 140, which includes a softmax 142 with a norm folding mechanism. Using a softmax 142 with a norm folding mechanism reduces the buffering requirements of a conventional softmax mechanism and can further reduce the cost of ML model 140 by implementing depth-first computation, as further described below.

[0054] Optionally, device 102 includes or is coupled to one or more image sensors 104. Image sensors 104 are configured to generate image data 105, which in some embodiments corresponds to input data 122. In a particular embodiment, image sensor 104 corresponds to or is incorporated into a camera (such as a still image camera, video camera, stereo camera, thermal imaging camera, one or more other types of cameras, or combinations thereof). According to one aspect, image data 105 includes data (e.g., pixel values) of individual images, video data, or combinations thereof.

[0055] Device 102 includes a memory 110 coupled to one or more processors 116 and configured to store instructions 112 and input data 122, such as individual images or data corresponding to images included in video data (e.g., video frames). For example, memory 110 may include a first image 124 and a second image 126 as part of the input data 122 to be processed at ML model 140, as described in further detail below. Memory 110 may also store data (e.g., parameters, such as weights and biases) associated with one or more ML models (such as ML model 140), which may be implemented at one or more processors 116. In a particular implementation, memory 110 corresponds to dynamic random access memory (DRAM) of a double data rate (DDR) memory subsystem.

[0056] One or more processors 116 are configured to execute instructions 112 to perform operations associated with the ML model 140. In various embodiments, some or all of the functionalities associated with the ML model 140 are executed via the execution of instructions 112 by one or more processors 116, by the processing circuitry of one or more processors 116 in a hardware embodiment, or by a combination thereof.

[0057] One or more processors 116 may include an input data source 120 coupled to the ML model 140 and configured to provide input data 122 to the ML model 140. For example, the input data source 120 may correspond to a portion of an image sensor 104, one or more media files (e.g., media files including input images 124, 126 retrieved from memory 110), one or more other sources of image information (such as from a remote media server), or a combination thereof.

[0058] One or more processors 116 are configured to process input data 122 using an ML model 140, which incorporates a softmax 142 with a norm folding mechanism to generate output data 152. For illustration, in a particular embodiment, input data 122 includes a first image 124 and a second image 126, and the ML model 140 corresponds to an OF or DFS architecture (such as further references) that generates either a flow graph 154 or a disparity map 156, respectively. Figure 4 (As described). According to one aspect, instead of using cost bodies to perform conventional OF or DFS, ML model 140 is used to generate probabilistic geometric measures without generating cost body data structures.

[0059] As illustrated, a softmax 142 with a norm folding mechanism is included in the streamable attention mechanism 144 of the ML model 140. Generally, the attention function can be calculated as follows:

[0060]

[0061] Where Q can correspond to the feature matrix (e.g., feature map) of one image, and K can correspond to the feature matrix of another image. It is a constant normalization factor, and V can correspond to a matrix of displacement values ​​(e.g., the values ​​of Δx for DFS or Δx, Δy for OF). In this example, P represents a matrix of probability values ​​(in the range [0,1]) that, when multiplied by the displacement values ​​in V, yield the expected value. Q, K, and V can each have linear complexity (e.g., O(N)) and are used to compute... The naive approach has quadratic complexity (e.g., O(N)). 2 This naive approach is similar to the conventional cost body approach.

[0062] Conventionally, the softmax function,

[0063] For i = 1, ..., K and ,

[0064] It is calculated using a multi-pass technique, in which the molecular value is calculated. Calculate the sum of the denominator as the numerator values, and calculate the sum by dividing each numerator value by the denominator. The value. Such multi-pass methods require buffering the values ​​generated in each pass. Therefore, the stream of input values ​​passed to the softmax calculation block is interrupted due to buffering, creating a pipeline breakpoint that prevents further processing until the multi-pass evaluation is complete. In some examples, the latency caused by buffering the incoming values ​​used for the softmax calculation may be prohibitive. In some instances, the amount of data to be buffered may exceed the capacity of the on-chip memory of processor 116, and data transfers to and from higher-capacity storage devices (e.g., memory 110) of device 102 and buffered data from those higher-capacity storage devices result in additional increases in latency and power consumption.

[0065] The disclosed techniques improve the computation of the softmax computation path, including by incorporating a norm folding mechanism that keeps the data flow in the streamable attention mechanism 144 "streamable" (i.e., even for large Q and K sizes, buffering or execution breakpoints are not required), and enables depth-first computation. Examples of computational components (e.g., network layers) of the streamable attention mechanism 144 (including a softmax 142 with a norm folding mechanism) are shown in... Figure 2 and Figure 3 Described in, and with Figure 2 The examples of conventional attention mechanisms described in the text are compared.

[0066] According to one aspect, ML model 140 is configured to perform regression on OF or DFS geometric coordinates using on-the-fly computation. On-the-fly computation allows one or more processors 116 to compute and consume computations as late as possible to avoid having to store computation results for later use. Generally, there are basic operand units required at any given time during the processing of input data 122 at ML model 140, and once computation has been completed for that range of operands, all intermediate computations are no longer needed and can be discarded, and the next batch of operands is loaded into memory. For example, in row-Q based... i and column K j Determine P ij During the softmax calculation, device 102 will Q i and K j Load them into memory (either as a whole or as a series of partitioned parts, as described below), and evaluate Q. i and K j The dot product to determine P ij Except for Q i Any row other than K or except K j Reading any column other than the one required into memory would be premature and unnecessarily increase memory usage beyond what is needed for computation.

[0067] For illustration, in the naive calculation method, Q is... i and K j All entries are loaded into memory, and the data for Q is stored. i and K j All calculations in the intermediate calculations. If Q i and K j Each has a size "S", then store Q. i and K j The required memory is 2S, and the memory required to store all the intermediate calculations is S. 2 However, by using Q iand K j Dividing the data into basic operand units that are loaded into memory as needed and performing on-the-fly computations can significantly reduce the amount of memory required to process the input data 122 at ML model 140. For example, the basic operand units could be 64 entries from each of Qi and Kj, and could be configured for Q. i and K j The first group of 64 entries performs the summation calculation of the numerator terms used for the softmax calculation, followed by Q. i and K j The next group of 64 entries, and so on, until Q has been processed. i and K j All entries and the cumulative total of all numerators has been generated.

[0068] According to one aspect, one or more processors 116 are configured to perform depth-first computation, in which intermediate computation is computed and consumed for an operand unit before processing the next operand unit, thus avoiding the need to store intermediate computation for later use and providing enhanced efficiency.

[0069] Output data 152 may include information generated as a result of processing input data 122 at ML model 140. For example, in an implementation where ML model 140 has an OF architecture, output data 152 may include flow graph 154. Generally, optical flow can indicate apparent motion patterns of objects, surfaces, and edges in a visual scene caused by relative motion between an observer (e.g., image sensor 104, robotic device, etc.) and the scene. Flow graph 154 may include optical flow information that can be used to measure motion of device 102 relative to objects or the scene. Optical flow information can also be used to measure visual motion or relative motion between device 102 and one or more other objects located near device 102. Additionally or alternatively, device 102 may use optical flow information for motion detection, object segmentation, stereo parallax measurement, etc.

[0070] In an implementation of ML model 140 with a DFS architecture, output data 152 may include a disparity map 156. The disparity map 156 can be used to generate depth information (such as a depth map), which can be employed in several different implementations (e.g., for navigation in an environment to avoid obstacles or plan a journey). For illustration, device 102 may be implemented in or correspond to a fully autonomous, semi-autonomous, or fully user-controlled drone or other motorized vehicle. In some implementations, device 102 (e.g., image sensor 104) may capture images of one or more objects or scenes and generate depth maps from those images. Such depth maps can be used to determine distances between objects in a scene, object heights, etc.

[0071] In some implementations, one or more processors 116 may generate or update a 3D representation 160 of an object or scene based on image data 105 and output data 152. For example, one or more processors 116 may process the output data 152 to determine 3D characteristics (such as height, depth, etc.) of an object in the input data 122, thereby generating the 3D representation 160. One or more processors 116 may update or refine the 3D representation 160 based on updated information in the input data 122 and / or output data 152 (such as in response to image sensor 104 capturing an additional image of an object that differs from a previous image due to relative motion between device 102 and the object). As an illustrative and non-limiting example, one or more processors 116 may be configured to generate one or more 2D images (such as novel viewpoints from the 3D representation) representing the 3D representation 160 of an object or scene for display to a user of device 102.

[0072] While in some embodiments the ML model 140 may be trained at device 102, in other embodiments the ML model 140 is not trained at device 102. For illustration, the ML model 140 may be trained at a remote device (such as remote device 198), and the trained ML model 140 may be sent to device 102 and stored in memory 110.

[0073] Device 102 may optionally include or be coupled to display device 106. Display device 106 is configured to display output image data 107 corresponding to output data 152 for viewing by a user of device 102. For example, in some embodiments where one or more processors 116 generate a 3D representation 160 of an object or scene based on image data 105, one or more processors 116 may generate output image data 107 based on 3D representation 160 (such as in extended reality applications).

[0074] Device 102 optionally includes a modem 118 coupled to one or more processors 116 and configured to enable communication with one or more other devices (such as via one or more wireless networks). According to some aspects, modem 118 is configured to receive image data 105, input data 122, or both (such as image data streamed from a remote device (such as remote device 198 (e.g., a remote server)) via wireless transmission 194) from a second device for processing at device 102. According to some aspects, modem 118 is configured to transmit data corresponding to output data 152, 3D representation 160, or both to the second device (such as image data streamed via wireless transmission 194 to remote device 198 (e.g., a remote server or user equipment) for storage or playback.

[0075] The advantage of using ML model 140 is that, compared to conventional techniques, ML model 140 is a low-usage model that operates with reduced memory usage. This low-usage model reduces read / write accesses caused by overflow memory accesses that could otherwise result from memory usage exceeding the available on-chip memory capacity. Reducing overflow memory accesses directly leads to reduced latency and power consumption, which improves the overall performance of device 102.

[0076] According to some aspects, one or more processors 116 are integrated in an integrated circuit (such as...) Figure 5 (As illustrated). According to some aspects, one or more processors 116 are integrated in a mobile phone or tablet computer device (such as... Figure 6 As illustrated), camera equipment (such as) Figure 7 (as illustrated) or wearable electronic devices (such as Figure 8 At least one of the following (as illustrated). According to some aspects, one or more processors 116 are integrated in a head-mounted device that includes a display and is configured to display an output image (such as...) based on the output of the ML model 140 when worn by a user. Figure 9 and Figure 10 (As illustrated). According to some aspects, one or more processors 116 are integrated into the voice-controlled speaker system (such as...). Figure 11 (As illustrated). According to some aspects, one or more processors 116 are integrated into a vehicle, which also includes one or more cameras configured to capture image data (such as...) corresponding to input data 122. Figure 12 and Figure 13 exemplified).

[0077] It should be understood that, for clarity of explanation, one or more aspects of device 102 may have been omitted from the above description. For example, while in some embodiments input data 122 may match image data 105, in other embodiments, input data 122 may be the result of additional processing performed on image data 105. As illustrative and non-limiting examples, such additional processing may include cropping, enlarging, tone mapping, color enhancement, magnification, or reduction.

[0078] Although OF and DFS are described as example architectures of ML model 140, it should be noted that the techniques disclosed herein are not limited to OF and DFS applications. In specific examples, the less-cost OF and DFS processing eliminates the secondary memory requirements associated with on-device attention and transformers used to determine OF, DFS, and stereo matching / depth estimates (such as depth estimation for spatial matching and OF estimation for temporal matching), enabling improved performance for applications such as mobile cameras, autonomous vehicles, extended reality head-mounted devices (e.g., glasses), etc. In another example, streamable softmax and attention processing (e.g., streamable attention mechanism 144 and softmax 142 with norm folding mechanism) helps remove execution pipelined and memory bottlenecks associated with operating transformer models on mobile devices (such as computer vision and language models where the softmax function can bear more than half of the model's overall latency). In another example, eliminating secondary memory requirements by using streamable softmax processing improves the operation of device 102 in an implementation where the ML model 140 is a diffusion model (such as a text-to-image generative model on a mobile device). In yet another example, streamlining via the streamable attention of this disclosure and eliminating secondary memory requirements associated with on-device attention and transformers improves performance associated with running large language models (LLMs) (such as LLMs with very large token sizes) and LLMs on mobile devices with relatively limited on-device memory. According to some implementations, the techniques of this disclosure are used to improve the performance of language models (such as LLMs), visual models (such as LVMs), and / or multimodal models (such as LMMs) (which can be viewed as an extension of LLMs or LVMs that operate on multimodal input data). For example, the techniques disclosed herein can be implemented in an LMM that processes input data corresponding to multiple modalities of an input / sensor source (such as audio, voice, text, image, video, etc.) and incorporates neural network processing including a streamable attention mechanism, which provides improved performance to handle large attention token sizes in a manner similar to LLM or LVM.

[0079] Figure 2Examples of works available according to this disclosure are depicted. Figure 1 Example 200 illustrates components and operations implemented in device 102. Specifically, example 200 illustrates components and operations that can be implemented in a streamable attention mechanism 144.

[0080] The streamable attention mechanism 144 is configured to receive first data (Q) 202 and second data (K) 204. Each of the first data (Q) 202 and the second data (K) 204 may correspond to a feature value matrix (e.g., a feature map) of a corresponding input image. According to one aspect, the second data 204 represents the transpose of the feature value matrix (e.g., K). T In a particular embodiment, first data 202 corresponds to features associated with the first image 124, and second data 204 corresponds to features associated with the second image 126. The streamable attention mechanism 144 also receives third data (V) 206, which corresponds to a displacement matrix of coordinates, as further described below.

[0081] The streamable attention mechanism 144 is configured to generate a softmax input stream 224 based on a first matrix multiplication operation 210 of a specific row of the first data 202 and the corresponding row of the second data 204. For illustration, the stream of values ​​212 output from the first matrix multiplication operation 210 (e.g., ...) ) by scaling factor 222 (e.g., ) Scale at multiplication operation 220 to generate softmax input stream 224 (e.g., It corresponds to the tensor (elements).

[0082] The exponentiation operation 230 of softmax 142 with norm folding mechanism is applied to softmax input stream 224 to generate a stream of softmax numerator values ​​232 (e.g., The exponent of each element in the softmax input stream of 224 is... z Executed in one iteration, because each entry It is available independently in the softmax input stream 224. The exponent of each element in the softmax input stream 224 does not depend on any other element in the softmax input stream 224. Therefore, the exponentiation operation 230 is streamable because no buffering is required.

[0083] In the first processing path, the stream of softmax numerator value 232 is input as the first input to the second matrix multiplication operation 240. In the second processing path parallel to the first processing path, the accumulated sum 235 of the softmax numerator value 232 is generated at the summation operation 234 by a single pass along the same Z. The accumulated sum 235 may include parallel evaluations of M attention functions (M is a positive integer) and corresponds to the norm vector. ,in, and This is for the attention function. In terms of the appropriate range.

[0084] The streamable attention mechanism 144 is configured to perform norm operation 250 to apply the sum 235 to third data (V) 206, thereby generating a second input 239 to the second matrix multiplication operation 240. For example, after all the softmax numerator values ​​in the softmax numerator values ​​232 have been accumulated into the sum 235, norm operation 250 includes applying inverse operation 236 to generate the reciprocal of the sum 235, which is represented as the inverse sum 237 (e.g., a norm vector). At multiplication operation 238, the inverse sum 237 is multiplied by the elements of the third data (V) 206 to generate the second input 239 to the second matrix multiplication operation 240.

[0085] The second matrix multiplication operation 240 multiplies the softmax numerator value 232 with the second input 239 (e.g., the product of the inverse sum 237 and the third data (V) 206) to generate the attention output value 260 (e.g., ).

[0086] Applying the inverse sum 237 at a later stage of the computation flow (e.g., not immediately applying the inverse sum 237 to the softmax numerator value 232 to generate a normalized softmax value) is called "norm folding" and can be used with or without depth-first computation, depending on the design. Using norm folding allows the processing of the stream of softmax numerator values ​​232 to continue without interruption due to buffering. An implementation where the point at which the inverse sum 237 is applied in the computation flow is selected based on one or more factors (such as the size of the input image) can be implemented as dynamic norm folding. For example, the inverse sum 237 may be applied earlier in the computation flow for smaller images (e.g., Video Graphics Array (VGA)) (e.g., applied to V 206 before being input to the second matrix multiplication operation 240) and later for larger images (e.g., 4K) (e.g., applied after the second matrix multiplication operation 240, such as...). Figure 3(As depicted).

[0087] For comparison, Example 280 depicts a conventional attention mechanism. The softmax input stream 224 is generated in a manner substantially similar to that described for Example 200 and is provided to the conventional softmax computation block 290. At the second matrix multiplication operation 240, the output 292 of the conventional softmax computation block 290 is multiplied by the third data (V) 206 to generate the attention output value 260. Since the conventional softmax computation involves multiplying the numerator value (e.g., ...)... The denominator is calculated by summing all the numerator values ​​in the formula (e.g., ...). The process involves one iteration of the input and another iteration of dividing each numerator value by the denominator, thus interrupting the stream because the input values ​​must be buffered until the denominator is calculated. For very large inputs (e.g., each row of Q and each column of K may have millions of entries), the latency introduced by buffering input values ​​while calculating a block 290 of regular softmax can be prohibitive. Furthermore, to add to the latency, the amount of data to be buffered may exceed the capacity of the on-chip memory, necessitating transfers to / from slower storage devices (e.g., direct memory access (DMA) writes / reads). These transfers are typically performed as block-by-block data transfers and do not support random access, resulting in additional latency and power consumption.

[0088] Compared to Example 280, the softmax 142 with norm folding mechanism in Example 200 enables the data flow in attention computation to remain "streamable" (that is, even for large Q and K sizes, no buffering or breakpoints are required) and enables depth-first computation.

[0089] For illustration, in example 200, the value of the softmax input stream 224 is evaluated at exponentiation operation 230. Each of the elements is processed to generate the corresponding softmax molecular value. The resulting softmax molecule value 232 is streamed as a single element into the second matrix multiplication block. Due to norm folding in Example 200, the "norm" operation is not performed directly on the molecule value (i.e., divided by the sum α) as in the regular softmax processing of Example 280. Instead, the norm operation is "folded" into another operation, that is, the norm is indirectly applied to the molecule value during the second matrix multiplication operation 240 after element-wise multiplication with V 206. Therefore, without buffering or execution breakpoints, the streamability of the attention computation is interrupted.

[0090] Figure 3 Examples of works available according to this disclosure are depicted. Figure 1Example 300 illustrates components and operations implemented in device 102. Specifically, example 300 illustrates components and operations that can be implemented in a streamable attention mechanism 144, another example of norm folding.

[0091] In Example 300, the softmax input stream 224 is generated in a manner substantially similar to that described for Example 200 and is provided to the exponentiation operation 230. The stream of the softmax numerator value 232 input in the first processing path serves as the first input to the second matrix multiplication operation 240. Also in the first processing path, the third data (V) 206 is input to the second matrix multiplication operation 240, which performs the operation to generate the output 342.

[0092] The streamable attention mechanism 144 is configured to perform a norm operation 350 to apply the sum 235 to the output 342 of the second matrix multiplication operation 240, such as by dividing the output 342 by the sum 235 (or equivalently, multiplying the output 342 by the inverse sum 237). For example, in a second processing path parallel to the first processing path, after all the softmax numerator values ​​in the softmax numerator value 232 have been accumulated into the sum 235, the norm operation 350 includes applying an inverse operation 236 to generate the inverse sum 237. At the multiplication operation 338, the inverse sum 237 is multiplied by the output 342 of the second matrix multiplication operation 240 to generate the attention output value 260.

[0093] Therefore, in Figure 3 The example illustrates norm folding, where the norm is applied to the output 342 of the second matrix multiplication operation 240. In general, norm folding can be applied even later in the processing to ensure that norm computation does not interrupt the streamability of attention computation. Depending on the aspect, attention mechanisms can employ dynamic norm folding, where the location of the norm folding is dynamically selected (e.g., based on the sizes of Q and K).

[0094] although Figure 2 and Figure 3 The examples are illustrations and descriptions of the operations performed by the function block, but it should be understood that in some implementations, each operation in the function block typically corresponds to a function performed by one or more corresponding layers of the neural network.

[0095] Figure 4 Examples of what can be done according to this disclosure are depicted. Figure 1 Example 400 of ML model 140 with low usage model having OF or DFS architecture implemented in system 100.

[0096] like Figure 4As shown, the ML model 140 includes a convolutional neural network (CNN) 402A for performing feature extraction on a first image 124 to generate features 404A (e.g., feature maps) and a CNN 402B for performing feature extraction on a second image 126 to generate features 404B (e.g., feature maps).

[0097] Feature enhancement transformer 410 is configured to process features 404A and 404B to generate enhanced features 412A and 412B, respectively. For example, feature enhancement transformer 410 may be configured to perform self-attention processing, cross-attention processing, or a combination thereof on features 404A and 404B.

[0098] Feature matching is performed on augmented features 412A and 412B at feature matching block 420 to generate output 422. In the example, feature matching block 420 may include or correspond to a softmax matching layer. Output 422 includes streaming or disparity information based on whether ML model 140 is configured to perform OF or DFS processing.

[0099] Propagation is performed on output 422 and enhancement features 412A at propagation block 430 (such as a self-attention layer). This propagation improves occluded pixels that extend beyond the boundary to generate output 432 of ML model 140. Output 432 includes streaming or disparity information based on whether ML model 140 is configured to perform OF or DFS processing.

[0100] According to one aspect, the feature enhancement transformer 410 includes a streamable attention mechanism 144A, the feature matching block 420 includes a streamable attention mechanism 144B, and the propagation block 430 includes a streamable attention mechanism 144C. Each of the streamable attention mechanisms 144A to 144C enables its respective processing block 410, 420, or 430 to perform attention processing without the secondary memory requirements of conventional attention mechanisms that include conventional multi-pass softmax operations, and helps to remove the execution pipeline and memory bottlenecks associated with operating the transformer model on mobile devices.

[0101] As illustrated, ML model 140 has an overall architecture similar to the GMFlow model, where the conventional attention mechanism has been replaced by a streamable attention mechanism 144. Although GMFlow is a transformer-based model that does not explicitly compute volumetric values, the various softmax operations performed in GMFlow (e.g., in the feature enhancement transformer 410, in the feature matching block 420, and in the propagation block 430) may require a memory amount substantially similar to that of conventional volumetric methods.

[0102] Table 1 presents illustrative, non-limiting examples comparing the ML model 140, the CNN-based RAFT (Recursive Full Endpoint-Pair Field Transformation for Optical Flow) model, and the transformer-based GMFlow model. Data for the RAFT model is based on “RAFT: Recursive Full Endpoint-Pair Field Transformation for Optical Flow, ECCV 2020”, and data for the GMFlow model is based on “GMFlow: Unified Flow, Stereo, and Depth Estimation”, TPAMI IEEE 2023.

[0103]

[0104] Table 1

[0105] In the examples depicted in Table 1, the cost-body equivalent memory size (e.g., the amount of memory associated with cost-body data structures and / or cost-body-related computations) on a neural signal processor (NSP) using VGA input is 30.6 megabytes (MB) for the RAFT model and 23.0 MB for the GMFlow model. Figure 4 The ML model 140 is 1.2MB. Therefore, compared to the GMFlow model, the associated memory usage of the ML model 140 is reduced by approximately 95%, while maintaining substantially similar performance metrics (such as endpoint error (EPE)) while maintaining mathematical equivalence of the models. Thus, the associated memory usage of the ML model 140 can be kept within the local memory capacity of the NSP (such as 8MB of vector tightly coupled memory (VTCM) available in mobile devices).

[0106] although Figure 4 The specific example of the depicted ML model 140 has an overall architecture similar to that of GMFlow, but in other implementations, the techniques disclosed herein can be used in combination with a variety of different architectures. For example, in other implementations, the feature extraction process may omit CNN 402, feature enhancement transformer 410, or both (e.g., another type of feature extractor may be used instead), the propagation block 430 may be omitted and / or replaced by another mechanism to improve occluded pixels that extend beyond the boundary, or a combination thereof.

[0107] Figure 5 This is a block diagram illustrating a specific implementation 500 of a device 502 for performing input data processing using an integrated circuit 502 with a low-cost model. The integrated circuit 502 includes one or more processors 116, which include an ML model 140 (e.g., a softmax 142 with a norm folding mechanism in a streamable attention mechanism 144). For example, the ML model 140 may include... Figure 2 Example 200 Figure 3Example 300 Figure 4 The integrated circuit 502 includes one or more components of Example 400 or any combination thereof. The integrated circuit 502 also includes a signal input 504 (such as a bus interface) to enable receiving input data 505 (such as image data 105 or input data 122). The integrated circuit 502 includes a signal output 506 (such as a bus interface) to enable outputting output data 507 (such as output data 152), outputting image data 107, or a 3D representation 160. Optionally, the integrated circuit 502 also includes a memory 110, an image sensor 104, an input data source 120, a modem 118, a display engine, etc. The integrated circuit 502 enables input data processing (e.g., optical flow or stereo depth processing) using low-volume models as components in a system performing image processing. Figure 1 (As depicted).

[0108] As an illustrative and non-restrictive example, Figure 6 A specific implementation 600 of which device 102 includes mobile device 602 (such as a telephone or tablet) is depicted. Mobile device 602 includes display screen 604 and camera 612 (e.g., image sensor 104). ML model 140 is integrated into mobile device 602 (e.g., integrated into integrated circuit 502), and the mobile device is illustrated using dashed lines to indicate internal components of mobile device 602 that are not typically visible to the user. In a specific example, ML model 140 operates to perform input data processing (e.g., optical flow or stereo depth processing). For example, mobile device 602 may generate image data 105 from camera 612, process image data 105 using ML model 140, and display the resulting output image data 107 at display screen 604, and / or send the resulting output image data 107, output data 152, and / or 3D representation 160 to another device (such as remote device 198).

[0109] Figure 7A specific implementation 700 is depicted in which device 102 includes a portable electronic device corresponding to camera device 702. Camera device 702 includes an image sensor 712, such as image sensor 104. ML model 140 is integrated into camera device 702 (such as being integrated into integrated circuit 502). In a specific example, ML model 140 operates to perform input data processing (e.g., optical flow or stereo depth processing). For example, camera device 702 may generate image data 105 from image sensor 712, process image data 105 using ML model 140, and display the resulting output image data 107 at a display screen of camera device 702, store the resulting output image data 107, output data 152, and / or 3D representation 160 in the memory of camera device 702, and / or transmit the resulting output image data 107, output data 152, and / or 3D representation 160 to another device (such as remote device 198).

[0110] Figure 8 A specific implementation 800 of a wearable electronic device 802 (illustrated as a "smartwatch") is depicted. In a particular aspect, the wearable electronic device 802 includes a device 102. The wearable electronic device 802 includes a display screen 804 and a camera 812 (e.g., an image sensor 104). An ML model 140 is integrated into the wearable electronic device 802 (e.g., integrated into an integrated circuit 502). In a specific example, the wearable electronic device 802 includes a haptic device that provides haptic notifications (e.g., vibrations) associated with the display of image or video data based on image or video data captured by the camera 812 and processed by the ML model 140 (e.g., output image data 107), which can be displayed via the display screen 804. For example, the haptic notification may allow a user to view the wearable electronic device 802 to watch video playback.

[0111] Figure 9A specific implementation 900 of the device 102 is depicted, comprising a portable electronic device corresponding to an extended reality device, such as augmented reality or mixed reality glasses 902. Glasses 902 include a holographic projection unit 904 configured to project visual data onto the surface of a lens 906, or to reflect the visual data from the surface of the lens 906 onto the wearer's retina. Glasses 902 includes a camera 912, such as an image sensor 104. An ML model 140 is integrated into glasses 902 (e.g., integrated into an integrated circuit 502). In a specific example, the ML model 140 operates to perform input data processing (e.g., optical flow or stereo depth processing). For example, image data 105 may be received from the camera 912, processed using the ML model 140, and the resulting output image data 107 (e.g., an output image based on the output of the ML model 140) may be displayed via projection onto the surface of the lens 906, enabling the display of images and / or videos associated with an extended reality, mixed reality, or virtual reality scene to the user while glasses 902 are worn.

[0112] Figure 10 A specific implementation 1000 is depicted in which device 102 includes a portable electronic device corresponding to a virtual reality, augmented reality, or mixed reality head-mounted device 1002. Head-mounted device 1002 includes a camera 1012 (such as an image sensor 104) and a visual display device 1004. An ML model 140 is integrated into head-mounted device 1002 (such as being integrated into integrated circuit 502). In a specific example, ML model 140 operates to perform input data processing (e.g., optical flow or stereo depth processing). For example, image data 105 may be received from camera 1012, processed using ML model 140, and the resulting output image data 107 (e.g., an output image based on the output of ML model 140) may be displayed at visual display device 1004, enabling the display of images and / or videos associated with augmented reality, mixed reality, or virtual reality scenes to a user while head-mounted device 1002 is worn.

[0113] Figure 11 This is a specific implementation 1100 of the wireless speaker and voice activation device 1102. In a particular aspect, the wireless speaker and voice activation device 1102 includes device 102. The wireless speaker and voice activation device 1102 may have wireless network connectivity and is configured to perform auxiliary operations. One or more processors 116 are included in the wireless speaker and voice activation device 1102, and an ML model 140 is included.

[0114] The wireless speaker and voice activation device 1102 includes a camera 1112 (such as an image sensor 104) and a display device 1120. In a specific example, the ML model 140 operates to perform input data processing (e.g., optical flow or stereo depth processing). For example, image data 105 may be received from the camera 1112 and processed using the ML model 140, and the resulting output image data 107 (e.g., an output image based on the output of the ML model 140) may be displayed at the display device 1120 and / or transmitted to a remote device for playback at the remote device.

[0115] In certain aspects, the wireless speaker and voice activation device 1102 includes one or more microphones 1110 and one or more speakers 1104. During operation, in response to receiving a verbal command via one or more microphones 1110, the wireless speaker and voice activation device 1102 can perform auxiliary operations, such as by executing a voice activation system (e.g., integrating an assistance application). Auxiliary operations may include adjusting the temperature, activating the camera 1112 to capture video or image content, and displaying the output image or video data at a display device 1120 based on the captured video content (e.g., output image data 107). For example, an auxiliary operation may be performed in response to receiving a command following a keyword or key phrase (e.g., “hello assistant”).

[0116] Figure 12 The illustration depicts a specific implementation 1200 in which device 102 corresponds to vehicle 1202 (exemplified as a manned or unmanned aerial device, such as a package delivery drone) or is integrated within the vehicle. ML model 140 is integrated into vehicle 1202 (e.g., integrated into integrated circuit 502). Vehicle 1202 may also include display device 1204 configured to display output (such as output image data 107) based on processing of input data at ML model 140.

[0117] In some implementations, vehicle 1202 is manned (e.g., carrying a pilot, one or more passengers, or both), display device 1204 is located inside the cabin of vehicle 1202, and input data processing (e.g., optical flow or stereo depth processing) is performed using images and / or video captures via one or more cameras 1212, which can be used to generate navigation information (such as based on depth or flow information and / or based on 3D representation (e.g., 3D representation 160)) corresponding to one or more objects or scenes adjacent to vehicle 1202, for purposes such as playback to the pilot or passengers of vehicle 1202 and / or for semi-autonomous or autonomous operation of vehicle 1202. In another specific implementation, the vehicle 1202 is driverless, and input data processing (e.g., optical flow or stereo depth processing) is performed using images and / or video captures via one or more cameras 1212 to generate navigation information corresponding to one or more objects or scenes adjacent to the vehicle 1202, which can be displayed to a remote operator of the vehicle 1202 and / or used for semi-autonomous or autonomous operation of the vehicle 1202.

[0118] In some implementations, the display device 1204 and camera 1212 are mounted to the outer surface of the vehicle 1202, and the input data processing at the ML model 140 is performed during video playback to one or more viewers outside the vehicle 1202. For example, the vehicle 1202 may move while playing back video or images based on video or image data captured via camera 1212 (e.g., circling around an outdoor audience during a concert).

[0119] Figure 13 The illustration depicts a device 102 corresponding to a vehicle 1302 (exemplified as an automobile) or a specific implementation 1300 integrated within such a vehicle. An ML model 140 is integrated into the vehicle 1302 (e.g., integrated into integrated circuit 502). In a specific example, the ML model 140 operates to perform input data processing based on image data received from one or more cameras 1312. The input data processing (e.g., optical flow or stereo depth processing) can be used to generate navigation messages (e.g., based on depth or flow information and / or based on 3D representations (e.g., 3D representation 160)) corresponding to one or more objects or scenes adjacent to the vehicle 1302, for purposes such as playback to the operator of the vehicle 1302 via a display screen 1320 or a speaker 1310 and / or for semi-autonomous or autonomous operation of the vehicle 1302.

[0120] For example, in a particular embodiment, vehicle 1302 may generate image data 105 from one or more cameras 1312, process the image data 105 at ML model 140, and display the resulting output image data 107 at display screen 1320 of vehicle 1302, store the resulting output image data 107, output data 152, and / or 3D representation 160 in the memory of vehicle 1302, and / or transmit the resulting output image data 107, output data 152, and / or 3D representation 160 to another device (such as remote device 198). In a particular embodiment, one or more cameras in camera 1312 may be mounted to capture interior scenes including one or more other passengers of vehicle 1302 (such as to monitor children in the back seat of vehicle 1302). Additionally or alternatively, one or more cameras in camera 1312 correspond to forward-facing and / or rearward-facing cameras that, in conjunction with autonomous or driver-assisted operation of vehicle 1302, capture the field of view outside vehicle 1302.

[0121] Figure 14 An example of input data processing method 1400 is illustrated. As an illustrative and non-limiting example, one or more operations of method 1400 may be performed by device 102, one or more processors 116, or... Figure 1 At least one of the systems 100 shall be executed.

[0122] At box 1402, method 1400 includes obtaining input data at a device. For example, input data 122 may be obtained from input data source 120, via image data 105 from image sensor 104, or from remote device 198.

[0123] At box 1404, method 1400 includes processing input data at a device using a machine learning (ML) model, which includes performing a softmax operation with norm folding. For example, device 102 processes input data 122 at an ML model 140 that includes a softmax 142 with a norm folding mechanism. In some embodiments, the softmax with norm folding is included in a streamable attention operation of the ML model (such as according to...). Figure 2 Example 200 or Figure 3 Example 300 is performed by a streamable attention mechanism 144. According to one aspect, the input data includes a first image and a second image (such as first image 124 and second image 126), and the ML model corresponds to a stereo depth or optical flow architecture (such as a reference). Figure 4 (as described in Example 400).

[0124] In some implementations, the streamable attention computation includes generating a softmax input stream based on a first matrix multiplication operation of a specific row of the first data and the corresponding row of the second data. For example, the softmax input stream 224 is generated based on the first matrix multiplication operation 210 of example 200 or example 300. The streamable attention computation may also include applying an exponentiation operation of softmax with norm folding to the softmax input stream to generate a stream of softmax numerator values ​​(such as a stream of softmax numerator values ​​232 generated by applying exponentiation operation 230 to the softmax input stream 224 of example 200 or example 300). The streamable attention computation may include providing a stream of softmax numerator values ​​as a first input to a second matrix multiplication operation (such as the second matrix multiplication operation 240 of example 200 or example 300) and an accumulated sum that generates the softmax numerator values. For example, the summation operation 234 of example 200 or example 300 generates an accumulated sum 235, which is inversely operated on and applied via norm folding to a later stage of the attention computation path.

[0125] By processing the input data using an ML model that includes performing a softmax operation with norm folding, the device avoids the conventional buffering of the softmax numerator value stream (e.g., buffering the softmax numerator value until the normalized value has been calculated and applied to the buffered softmax numerator value), thereby reducing the latency associated with the softmax operation and also avoiding the additional latency and power consumption associated with data transfers to / from another storage device of device 102 that could otherwise be combined with buffering. Furthermore, avoiding the buffering of the softmax numerator value stream enables the use of just-in-time computation, depth-first computation, or both (as associated with the processing of the softmax numerator value), thereby reducing the total amount of memory used for processing the input data at the ML model.

[0126] Figure 15 Method 1400 can be implemented by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a digital signal processor (DSP), a controller, another hardware device, a firmware device, or any combination thereof. As an example, Figure 14 Method 1400 can be executed by a processor that executes instructions, such as references Figure 15 As described.

[0127] refer to Figure 15 This diagram depicts a specific, exemplary embodiment of the device, and generally designates the device as 1500. In various embodiments, device 1500 may have the same... Figure 15The illustrated components may be more or fewer than the number of components. In the illustrative embodiment, device 1500 may correspond to... Figure 1 Device 102. In an exemplary implementation, device 1500 may execute the reference. Figures 1 to 14 One or more operations as described.

[0128] In a particular embodiment, device 1500 includes a processor 1506 (e.g., a CPU). Device 1500 may include one or more additional processors 1510 (e.g., one or more DSPs). In a particular embodiment, Figure 1 One or more processors 116 correspond to processor 1506, processor 1510, or combinations thereof. For example, processor 1510 may include ML model 140. For example, ML model 140 may include... Figure 2 Example 200 Figure 3 Example 300 Figure 4 The processor 1510 may include one or more components of Example 400 or combinations thereof. The processor 1510 may also include a speech and music decoder-decoder (CODEC) 1508. The speech and music CODEC 1508 may include a voice decoder (“vocoder”) encoder 1536, a vocoder decoder 1538, or combinations thereof.

[0129] In this context, the term "processor" refers to an integrated circuit composed of logic units, interconnects, input / output blocks, clock management components, memory, and optional other dedicated hardware components, designed to execute instructions and perform various computational tasks. Examples of processors include, but are not limited to, CPUs, digital signal processors (DSPs), neural processing units (NPUs), graphics processing units (GPUs), FPGAs, microcontrollers, quantum processors, coprocessors, vector processors, other similar circuits, and their variations and combinations. In some cases, a processor may be integrated with other components, such as communication components, input / output components, etc., to form a system-on-a-chip (SoC) device or a packaged electronic device.

[0130] Starting with the CPU, a CPU typically includes one or more processor cores. Each of these cores comprises a complex network of interconnected transistors and other circuitry defining logic gates, memory elements, and so on. The cores are responsible for executing instructions to perform, for example, arithmetic and logical operations. Typically, a CPU includes an arithmetic logic unit (ALU) that handles mathematical operations and a control unit that generates signals to coordinate the operations of other CPU components, such as managing fetch-decode-execute loops.

[0131] CPUs and / or individual processor cores typically include local memory circuitry, such as registers and caches, to temporarily store data during operation. Registers consist of high-speed, small-size memory cells tightly connected to logic units within the CPU. Typically, registers include transistors arranged as groups of flip-flops configured to store binary data. Caches consist of fast on-chip memory circuitry for storing frequently accessed data. For example, a cache can be implemented using static random access memory (SRAM) circuitry.

[0132] CPU operations (e.g., arithmetic, logical, and flow control operations) are guided by software and firmware. At the lowest level, the CPU includes an Instruction Set Architecture (ISA), which specifies how hardware resources (e.g., registers, arithmetic units, etc.) are used to perform individual operations. Higher-level software and firmware are then translated into various combinations of ISA operations to enable the CPU to perform specific higher-level operations. For example, an ISA typically specifies how the CPU's hardware components move and modify data to perform operations such as addition, multiplication, and subtraction, and higher-level software is translated into sets of such operations to accomplish larger tasks, such as adding two columns in a spreadsheet. Generally, the CPU operates on various levels of software, including the kernel, operating system, applications, etc., where each higher-level software is typically more abstract than the ISA and is generally easier for human users to understand.

[0133] GPUs, NPUs, DSPs, microcontrollers, coprocessors, FPGAs, ASICs, and vector processors include components similar to those described above for CPUs. The differences between these various types of processors are often related to the use of dedicated interconnect schemes and ISAs to enhance the processor's ability to perform specific types of operations. For example, the logic gates, local memory circuitry, and interconnects between them of a GPU are specifically designed to improve parallel processing, data sharing between processor cores, and vector operations, and the GPU's ISA can define operations that utilize these structures. As another example, an ASIC is a highly specialized processor that includes similar circuitry arranged and interconnected for specific tasks, such as encryption or signal processing. As yet another example, an FPGA is a programmable device that includes an array of configurable logic blocks (e.g., an interconnect set of transistors and memory elements) that can be configured (typically in flight) to perform customizable logic functions.

[0134] Device 1500 may include memory 1586 and CODEC 1534. Memory 1586 may include instructions 1556 that can be executed by one or more additional processors 1510 (or processor 1506) to implement the functionality described in reference processor 116. In a specific example, memory 1586 corresponds to Figure 1The memory 110 and instruction 1556 correspond to instruction 112. The device 1500 may include a modem 118 coupled to an antenna 1552 via a transceiver 1550. The device 1500 may also include one or more cameras 1594, one or more of which may correspond to an image sensor 104.

[0135] Device 1500 may include a display 1528, such as display device 106, coupled to display controller 1526. One or more speakers 1592, one or more microphones 1590, or a combination thereof may be coupled to CODEC 1534. CODEC 1534 may include digital-to-analog converter (DAC) 1502 and analog-to-digital converter (ADC) 1504. In a particular embodiment, CODEC 1534 may receive analog signals from microphone 1590, use ADC 1504 to convert the analog signals to digital signals, and transmit the digital signals to voice and music codec 1508. In a particular embodiment, voice and music codec 1508 may provide digital signals to CODEC 1534. CODEC 1534 may use ADC 1502 to convert digital signals to analog signals and may provide analog signals to speaker 1592.

[0136] In a particular embodiment, device 1500 may be included in a system-in-package (SoC) or system-on-a-chip (SoC) 1522. In a particular embodiment, memory 1586, processor 1506, processor 1510, display controller 1526, CODEC 1534, and modem 118 are included in the SoC or SoC 1522. In a particular embodiment, input device 1530 (e.g., keyboard, touchscreen, or pointing device) and power supply 1544 are coupled to the SoC or SoC 1522. Furthermore, in a particular embodiment, such as... Figure 15 As illustrated, camera 1594, display 1528, input device 1530, speaker 1592, microphone 1590, antenna 1552, and power supply 1544 are located external to the system-in-package or system-on-chip device 1522. In a particular implementation, each of camera 1594, display 1528, input device 1530, speaker 1592, microphone 1590, antenna 1552, and power supply 1544 may be coupled to components (such as interfaces or controllers) of the system-in-package or system-on-chip device 1522.

[0137] Device 1500 may include smart speakers, soundbars, mobile communication devices, smartphones, cellular phones, laptops, computers, tablets, personal digital assistants, display devices, televisions, game consoles, music players, radios, digital video players, digital video disc (DVD) players, tuners, cameras, navigation devices, vehicles, head-mounted devices, augmented reality head-mounted devices, mixed reality head-mounted devices, virtual reality head-mounted devices, air vehicles, home automation systems, voice-activated devices, wireless speakers and voice-activated devices, portable electronic devices, automobiles, vehicles, computing devices, communication devices, Internet of Things (IoT) devices, virtual reality (VR) devices, base stations, mobile devices, or any combination thereof.

[0138] In conjunction with the described technology, an apparatus includes components for acquiring input data. In an example, the components for acquiring input data may include an input data source 120, an image sensor 104, a modem 118, one or more processors 116, a device 102, a system 100, one or more other circuits or devices for acquiring input data, or combinations thereof.

[0139] The apparatus also includes components for processing the input data using a machine learning model, which incorporates components for performing a softmax operation with norm folding. In the example, the components for processing the input data using a machine learning model, incorporating components for performing a softmax operation with norm folding, may include one or more processors 116, an ML model 140 executed by one or more processors 116, a device 102, a system 100, one or more other circuits or devices, or combinations thereof, incorporating components for performing a softmax operation with norm folding. In the example, the components for performing a softmax operation with norm folding may include one or more processors 116, a softmax mechanism 142 with norm folding, a streamable attention mechanism 144, a device 102, a system 100, one or more other circuits or devices, or combinations thereof, for performing a softmax operation with norm folding.

[0140] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as memory 110) includes instructions (e.g., instruction 112) that, when executed by one or more processors (e.g., one or more processors 116), cause one or more processors to perform operations related to reference. Figures 1 to 13 Any technology described Figure 14The instructions correspond to at least a portion of the methods or any combination thereof. In the example, when executed by one or more processors, these instructions cause one or more processors to receive input data (e.g., input data 122). When executed by one or more processors, these instructions also cause one or more processors to process the input data using a machine learning model (e.g., ML model 140) that incorporates a softmax with a norm folding mechanism (e.g., softmax 142 with a norm folding mechanism).

[0141] Specific aspects of this disclosure are described below in the following set of related embodiments:

[0142] According to Embodiment 1, an apparatus includes: a memory configured to store input data; and one or more processors configured to process the input data using a machine learning (ML) model, the machine learning (ML) model incorporating a softmax with a norm folding mechanism.

[0143] Example 2 includes the device described in Example 1, wherein the input data includes a first image and a second image, and wherein the ML model corresponds to a stereo depth or optical flow architecture.

[0144] Example 3 includes the device described in Example 1 or Example 2, wherein the softmax having a norm folding mechanism is included in a streamable attention mechanism.

[0145] Example 4 includes the device described in Example 3, wherein the streamable attention mechanism is configured to: generate a softmax input stream based on a first matrix multiplication operation of a specific row of first data and a corresponding row of second data; apply an exponential operation of the softmax with a norm folding mechanism to the softmax input stream to generate a stream of softmax numerator values; use the stream of softmax numerator values ​​as a first input to a second matrix multiplication operation; and generate an accumulated sum of the softmax numerator values.

[0146] Example 5 includes the device described in Example 4, wherein the streamable attention mechanism is configured to perform norm operations to apply the sum to third data to generate a second input to the second matrix multiplication operation.

[0147] Example 6 includes the device described in Example 4, wherein the streamable attention mechanism is configured to perform norm operations to apply the sum to the output of the second matrix multiplication operation.

[0148] Example 7 includes the device described in any one of Examples 4 to 6, wherein: the first data corresponds to a feature associated with a first image; the second data corresponds to a feature associated with a second image; and the second input to the second matrix multiplication operation corresponds to a displacement matrix of coordinates.

[0149] Example 8 includes the device described in any one of Examples 1 to 7, wherein the ML model generates probabilistic geometric measures without generating a volumetric data structure.

[0150] Example 9 includes the device described in any one of Examples 1 to 8, wherein the ML model is configured to perform regression for stereo depth or optical flow geometry using real-time computation.

[0151] Example 10 includes the device described in any one of Examples 1 to 9 and further includes an image sensor configured to generate image data corresponding to the input data.

[0152] Example 11 includes the device of any one of Examples 1 to 10 and further includes a modem coupled to the one or more processors, the modem being configured to receive image data corresponding to the input data from the second device.

[0153] Example 12 includes the device of any one of Examples 1 to 11, wherein one or more processors are integrated in a head-mounted device including a display, and wherein the head-mounted device is configured, when worn by a user, to display an output image based on the output of the ML model.

[0154] Example 13 includes the device described in any one of Examples 1 to 11, wherein one or more processors are integrated in at least one of a mobile phone, a tablet computer device, a wearable electronic device, or a camera device.

[0155] Example 14 includes the device described in any one of Examples 1 to 11, wherein the one or more processors are integrated in a vehicle, and the vehicle further includes one or more cameras configured to capture image data corresponding to the input data.

[0156] Example 15 includes the device described in any one of Examples 1 to 14, wherein one or more processors are included in an integrated circuit.

[0157] Example 16 includes the device described in any one of Examples 1 to 15, wherein the ML model includes a language model, a visual model, or a multimodal model.

[0158] According to Embodiment 17, a method includes obtaining input data at a device; and processing the input data at the device using a machine learning (ML) model, including performing a softmax operation with norm folding.

[0159] Example 18 includes the method described in Example 17, wherein the input data includes a first image and a second image, and wherein the ML model corresponds to a stereo depth or optical flow architecture.

[0160] Example 19 includes the method described in Example 17 or Example 18, wherein the softmax with norm folding operation is included in the streamable attention operation of the ML model.

[0161] Example 20 includes the method described in Example 19, wherein the streamable attention operation includes: generating a softmax input stream based on a first matrix multiplication operation of a specific row of first data and a corresponding row of second data; applying an exponential operation of the softmax with norm folding to the softmax input stream to generate a stream of softmax numerator values; providing the stream of softmax numerator values ​​as a first input to a second matrix multiplication operation; and generating an accumulated sum of the softmax numerator values.

[0162] Example 21 includes the method described in Example 20, wherein the streamable attention operation includes: performing a norm operation to apply the sum to third data to generate a second input to the second matrix multiplication operation.

[0163] Example 22 includes the method described in Example 20, wherein the streamable attention operation includes: performing a norm operation to apply the sum to the output of the second matrix multiplication operation.

[0164] Example 23 includes the method of any one of Examples 20 to 22, wherein: the first data corresponds to a feature associated with a first image; the second data corresponds to a feature associated with a second image; and the second input to the second matrix multiplication operation corresponds to a displacement matrix of coordinates.

[0165] Example 24 includes the method of any one of Examples 17 to 23, wherein the ML model generates probabilistic geometric measures without generating a volumetric data structure.

[0166] Example 25 includes the method of any one of Examples 17 to 24, wherein the ML model uses real-time computation to perform regression against stereo depth or optical flow geometry coordinates.

[0167] Example 26 includes the method of any one of Examples 17 to 25 and further includes: receiving image data corresponding to the input data.

[0168] Example 27 includes the method of any one of Examples 17 to 26 and further includes: receiving image data corresponding to the input data from a second device.

[0169] Example 28 includes the method of any one of Examples 17 to 27 and further includes: displaying an output image based on the output of the ML model.

[0170] Example 29 includes the method of any one of Examples 17 to 28, wherein the ML model includes a language model, a visual model, or a multimodal model.

[0171] According to Embodiment 30, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to: obtain input data; and process the input data using a machine learning (ML) model, the machine learning (ML) model incorporating a softmax with a norm folding mechanism.

[0172] Example 31 includes the non-transitory computer-readable medium of Example 30, wherein the softmax having a norm folding mechanism is included in a streamable attention mechanism.

[0173] Example 32 includes the non-transitory computer-readable medium of Example 30 or Example 31, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: generate a softmax input stream at the streamable attention mechanism based on a first matrix multiplication operation of a specific row of first data and a corresponding row of second data; apply an exponential operation of the softmax with a norm folding mechanism to the softmax input stream at the streamable attention mechanism to generate a stream of softmax numerator values; input the stream of softmax numerator values ​​at the streamable attention mechanism as a first input to a second matrix multiplication operation; and generate an accumulated sum of the softmax numerator values ​​at the streamable attention mechanism.

[0174] Example 33 includes the non-transitory computer-readable medium of Example 32, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: perform norm operations at the streamable attention mechanism to apply the sum to third data to generate a second input to the second matrix multiplication operation.

[0175] Example 34 includes the non-transitory computer-readable medium of Example 32, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: perform a norm operation at the streamable attention mechanism to apply the sum to the output of the second matrix multiplication operation.

[0176] Example 35 includes a non-transitory computer-readable medium as described in any one of Examples 32 to 34, wherein: the first data corresponds to features associated with a first image; the second data corresponds to features associated with a second image; and the second input to the second matrix multiplication operation corresponds to a displacement matrix of coordinates.

[0177] Example 36 includes a non-transitory computer-readable medium as described in any one of Examples 30 to 35, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: generate probabilistic geometric measures at the ML model without generating a volumetric data structure.

[0178] Example 37 includes a non-transitory computer-readable medium as described in any one of Examples 30 to 36, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: perform regression for stereo depth or optical flow geometric coordinates using on-the-fly computation at the ML model.

[0179] Example 38 includes any one of Examples 30 to 37, which is a non-transitory computer-readable medium, wherein the ML model includes a language model, a visual model, or a multimodal model.

[0180] According to embodiment 39, an apparatus includes: a component for acquiring input data; and a component for processing the input data using a machine learning (ML) model, the machine learning (ML) model incorporating a component for performing a softmax operation with norm folding.

[0181] Example 40 includes the apparatus described in Example 39, wherein the input data includes a first image and a second image, and wherein the ML model corresponds to a stereo depth or optical flow architecture.

[0182] Example 41 includes the apparatus described in Example 39 or Example 40, wherein the component for performing a softmax operation with norm folding is included in the component for performing a streamable attention operation.

[0183] Example 42 includes the apparatus of Example 41, wherein the components for performing a streamable attention operation include: components for generating a softmax input stream based on a first matrix multiplication operation of a specific row of first data and a corresponding row of second data; components for applying an exponential operation of the softmax with norm folding to the softmax input stream to generate a stream of softmax numerator values; components for providing the stream of softmax numerator values ​​as a first input to a second matrix multiplication operation; and components for generating an accumulated sum of the softmax numerator values.

[0184] Example 43 includes the apparatus described in Example 42, wherein the component for performing a streamable attention operation includes: a component for performing a norm operation to apply the sum to third data to generate a second input to the second matrix multiplication operation.

[0185] Example 44 includes the apparatus described in Example 42, wherein the component for performing the streamable attention operation includes: a component for performing a norm operation to apply the sum to the output of the second matrix multiplication operation.

[0186] Example 45 includes the apparatus of any one of Examples 42 to 44, wherein: the first data corresponds to a feature associated with a first image; the second data corresponds to a feature associated with a second image; and the second input to the second matrix multiplication operation corresponds to a displacement matrix of coordinates.

[0187] Example 46 includes the apparatus of any one of Examples 39 to 45, wherein the component for processing the input data using an ML model generates a probabilistic geometric measure without generating an intrinsic data structure.

[0188] Example 47 includes the apparatus of any one of Examples 39 to 46, wherein the component for processing the input data using an ML model performs regression for stereo depth or optical flow geometric coordinates using real-time computation.

[0189] Example 48 includes the apparatus of any one of Examples 39 to 47 and further includes: a component for receiving image data corresponding to the input data.

[0190] Example 49 includes the apparatus of any one of Examples 39 to 48 and further includes: a component for receiving image data corresponding to the input data from a second device.

[0191] Example 50 includes the apparatus of any one of Examples 39 to 49 and further includes: a component for displaying an output image based on the output of the ML model.

[0192] Example 51 includes the apparatus of any one of Examples 39 to 50, wherein the ML model includes a language model, a visual model, or a multimodal model.

[0193] Those skilled in the art will also recognize that the various exemplary logic blocks, configurations, circuits, and algorithm steps described in connection with the specific embodiments disclosed herein can be implemented as electronic hardware, computer software executed by a processing device (such as a hardware processor), or a combination of both. The various exemplary components, blocks, configurations, circuits, and steps have been generally described above regarding their functionality. Whether this functionality is implemented as hardware or executable software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

[0194] The steps of the methods or algorithms described in conjunction with the specific embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in a memory device such as random access memory (RAM), magnetoresistive random access memory (MRAM), spin-torque-conversion MRAM (STT-MRAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, compressed optical disc read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art. An exemplary memory device is coupled to a processor such that the processor can read information from and write information to the memory device. Alternatively, the memory device may be integrated with the processor. The processor and storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. Alternatively, the processor and storage medium may reside as discrete components in a computing device or a user terminal.

[0195] A prior description of the disclosed specific embodiments is provided to enable those skilled in the art to implement or use the disclosed specific embodiments. Various modifications to these specific embodiments will be apparent to those skilled in the art, and the principles defined herein may be applied to other specific embodiments without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the specific embodiments shown herein, but should be given the broadest scope that may be consistent with the principles and novel features as defined by the appended claims.

Claims

1. An apparatus, the apparatus comprising: A memory configured to store input data; and One or more processors are configured to process the input data using a machine learning (ML) model that incorporates a softmax algorithm with a norm folding mechanism.

2. The device of claim 1, wherein the input data includes a first image and a second image, and wherein the ML model corresponds to a stereo depth or optical flow architecture.

3. The device according to claim 1, wherein the ML model includes a language model, a visual model, or a multimodal model.

4. The device of claim 1, wherein the softmax having a norm folding mechanism is included in a streamable attention mechanism.

5. The device of claim 4, wherein the streamable attention mechanism is configured as follows: The softmax input stream is generated by multiplying a specific row of the first data with the corresponding row of the second data using a first matrix multiplication operation. The exponential operation of the softmax with norm folding mechanism is applied to the softmax input stream to generate a stream of softmax numerator values; The stream of input softmax molecule values ​​is used as the first input to the second matrix multiplication operation; as well as Generate the cumulative sum of the softmax molecule values.

6. The device of claim 5, wherein the streamable attention mechanism is configured to perform norm operations to apply the sum to the third data to generate a second input to the second matrix multiplication operation.

7. The device of claim 5, wherein the streamable attention mechanism is configured to perform norm operations to apply the sum to the output of the second matrix multiplication operation.

8. The device according to claim 5, wherein: The first data corresponds to features associated with the first image; The second data corresponds to features associated with the second image; and The second input to the second matrix multiplication operation corresponds to the displacement matrix of the coordinates.

9. The device of claim 1, wherein the ML model generates probabilistic geometric measures without generating a volumetric data structure.

10. The device of claim 1, wherein the ML model is configured to perform regression for stereo depth or optical flow geometry using instantaneous computation.

11. The apparatus of claim 1, further comprising an image sensor configured to generate image data corresponding to the input data.

12. The device of claim 1, further comprising a modem coupled to the one or more processors, the modem being configured to receive image data corresponding to the input data from the second device.

13. The device of claim 1, wherein the one or more processors are integrated in a head-mounted device including a display, and wherein the head-mounted device is configured, when worn by a user, to display an output image based on the output of the ML model.

14. The device of claim 1, wherein the one or more processors are integrated in at least one of a mobile phone, a tablet computer device, a wearable electronic device, or a camera device.

15. The device of claim 1, wherein the one or more processors are integrated in a vehicle, the vehicle further comprising one or more cameras configured to capture image data corresponding to the input data.

16. The device of claim 1, wherein the one or more processors are included in an integrated circuit.

17. A method, the method comprising: Obtain input data from the device; as well as The input data is processed at the device using a machine learning (ML) model, including performing a softmax operation with norm folding.

18. The method of claim 17, wherein the softmax having norm folding operation is included in the streamable attention operation of the ML model.

19. The method of claim 18, wherein the streamable attention computation comprises: The softmax input stream is generated by multiplying a specific row of the first data with the corresponding row of the second data using a first matrix multiplication operation. The exponential operation of the softmax with norm folding is applied to the softmax input stream to generate a stream of softmax numerator values; The stream providing the softmax molecule value serves as the first input to the second matrix multiplication operation; as well as Generate the cumulative sum of the softmax molecule values.

20. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Obtain input data; and The input data is processed using a machine learning (ML) model that incorporates a softmax mechanism with norm folding.

Images