Method and system for accelerated operation of layers used in machine learning models and differentiatable point rendering using proximity attention

Abstract

The invention provides a method and a system for accelerating operation of each layer in a machine learning model. The method comprises the following steps: receiving rows or columns of a weight matrix and an input vector; transforming rows or columns of the weight matrix and the input vector using an embedded function; performing a search to identify transformed rows or columns in the weight matrix that would obtain a high post-activation value; measuring a similarity between rows or columns of the weight matrix returned by the search and the associated input vector using a kernel function to determine a post-activation value; determining an output matrix, including: when a value vector is a column vector of a unit matrix, making the output matrix equal to the post-activation value, otherwise, when a value vector is not a column vector of a unit matrix, combining the post-activation value with an associated value vector; and outputting the output matrix.

Description

TECHNICAL FIELD

[0001] The following relates generally to machine learning models, and more specifically to a method and system for accelerating layer operations used in machine learning models and differentiable point rendering with neighbor attention.

[0002] BACKGROUND

[0003] Attention operation is a commonly used operation in many neural networks. Given a set of keys, queries, and values (represented in the form of vectors), for each query vector, the attention operation combines the value vectors in a weighted manner: the value vector corresponding to a key vector that is more important to the query vector is given a higher weight, while the value vector corresponding to a key vector that is less important to the query vector is given a lower weight. A key vector that is important to a query vector is referred to as a key vector that the query vector pays attention to. The importance of each key vector to a query vector can be determined by its similarity to the query vector. The attention operation usually determines the output in two steps. First, the attention weights of the key vectors and the query vectors are calculated, which are numbers between 0 and 1, indicating the importance of each key vector to each query vector. Then, the value vectors are combined with the attention weights to generate an output vector for each query vector. SUMMARY

[0004] In one aspect, the present disclosure provides a method for accelerating layer operations used in machine learning models, executed by a processor, the method comprising:

[0005] receiving a row vector or a column vector of a weight matrix and an input vector, wherein each row vector or column vector of the weight matrix is associated with a value vector;

[0006] transforming the row vector or the column vector of the weight matrix and the input vector with an embedding function;

[0007] performing a search to identify the row vector or the column vector of the weight matrix that results in a high post-activation value for each transformed input vector;

[0008] measuring the similarity between the row vector or the column vector of the weight matrix returned by the search and the input vector associated therewith with a kernel function to determine the post-activation value;

[0009] outputting the post-activation value for use in the operation of a layer in a machine learning model.

[0010] In a specific instance of the method, the method further includes: determining an output matrix, wherein when the value vector is a column vector of an identity matrix, the output matrix is ​​made equivalent to the post-activation; otherwise, when the value vector is not a column vector of an identity matrix, the post-activation is combined with the value vector associated with it to generate an output matrix; and the output matrix is ​​output for use by a machine learning model.

[0011] In another embodiment of the method, the embedding function is used to transform the row or column vectors of the weight matrix and the input vector, including embedding the row or column vectors of the weight matrix and the input vector into a higher-dimensional space.

[0012] In another embodiment of the method, the search includes: for each transformed query, identifying the row or column of the corresponding weight matrix with higher importance using a k-nearest neighbor search for each transformed input vector.

[0013] In another embodiment of the method, the search includes one of the following:

[0014] Returns all points or subsets located within a specific radius of the query vector, or returns points whose similarity to the query vector exceeds a threshold;

[0015] Identify points that are close to or similar to the query vector;

[0016] Cluster points that are close to or similar to each other, and return the points that are in the clusters closest to the query vector;

[0017] Construct a graph with nodes as nodes, connecting nearby or similar nodes with edges, and then traverse the graph; and

[0018] Estimate the probability density of points and return points whose probability density exceeds a threshold and are close to or similar to the query vector.

[0019] In another embodiment of the method, the method further includes receiving a mask matrix, wherein the search uses only the rows or columns of the unmasked weight matrix in the mask matrix.

[0020] In another embodiment of the method, if the search fails to identify a row or column of the weight matrix, determining the corresponding post-activation may include modifying the post-activation.

[0021] In another instance of the method, for rows or columns of the weight matrix that fail to be identified in the search, the processing method is to treat those rows or columns that depend on the weight matrix and the kernel function of the query as a constant.

[0022] In another embodiment of the method, the value of the constant can be fixed, or the value of the constant can be determined by randomly sampling a subset of rows or columns of the weight matrix. In determining this value, the values ​​of the corresponding kernel functions are first obtained through these sampled values ​​of the weight matrix and a query, and then combined using some form of averaging.

[0023] In another embodiment of the method, determining the output matrix includes treating the value vectors associated with rows or columns of weight matrices that the search failed to identify as constant vectors.

[0024] In another embodiment of the method, the layer includes attention operations, the post-activation includes attention weights, and the rows or columns of the weight matrix contain keys.

[0025] On the other hand, the present invention also provides a system for accelerating layer operations used in machine learning models, the system including a processing unit and a data memory, the data memory containing executable instructions, which, when executed by the processing unit, will cause the system to perform the following operations:

[0026] It receives the row or column vectors of the weight matrix and the input vector, wherein each row or column vector of the weight matrix is ​​associated with a value vector;

[0027] The row vectors or column vectors of the weight matrix and the input vector are transformed using an embedding function;

[0028] For each transformed input vector, a search is performed to identify the row or column vectors in the weight matrix that have obtained high activation values ​​after transformation.

[0029] The similarity between the row or column vectors of the weight matrix returned by the search and their associated input vectors is measured using a kernel function to determine the post-activation value;

[0030] The output is then activated for running machine learning models.

[0031] In a specific case of the system, the executable instructions are further used to determine the output matrix, including: when the value vector is a column vector of the identity matrix, making the output matrix equivalent to the post-activation; otherwise, when the value vector is not a column vector of the identity matrix, combining the post-activation with its associated value vector to generate the output matrix; and outputting the output matrix for use by the machine learning model.

[0032] In another embodiment of the system, the embedding function is used to transform the row or column vectors of the weight matrix and the input vector, including embedding the row or column vectors of the weight matrix and the input vector into a higher-dimensional space.

[0033] In another embodiment of the system, the search comprises: for each transformed query, identifying the row or column of the corresponding weight matrix with higher importance using a k-nearest neighbor search for each transformed input vector.

[0034] In another embodiment of the system, the search includes one of the following:

[0035] Returns all points or subsets located within a specific radius of the query vector, or returns points whose similarity to the query vector exceeds a threshold;

[0036] Identify points that are close to or similar to the query vector;

[0037] Cluster points that are close to or similar to each other, and return the points that are in the clusters closest to the query vector;

[0038] Construct a graph with nodes as nodes, connecting nearby or similar nodes with edges, and then traverse the graph; and

[0039] Estimate the probability density of points and return points whose probability density exceeds a threshold and are close to or similar to the query vector.

[0040] In another scenario of the system, if the search fails to identify a row or column of the weight matrix, determining the corresponding post-activation may include modifying the post-activation.

[0041] In another scenario of the system, for rows or columns of the weight matrix that fail to be identified in the search, the approach is to treat these rows or columns, which depend on the weight matrix, and the kernel function of the query as a constant.

[0042] In another embodiment of the system, the value of the constant can be fixed, or it can be determined by randomly sampling a subset of rows or columns of the weight matrix. In this determination process, the values ​​of the corresponding kernel functions are first obtained through these sampled values ​​of the weight matrix and a query, and then combined using some form of averaging.

[0043] In another embodiment of the system, determining the output matrix includes treating the value vectors associated with rows or columns of weight matrices that the search failed to identify as constant vectors.

[0044] On the other hand, the present invention also provides a method executed by a processor for learning a three-dimensional (3D) scene representation using differentiable point rendering with proximity attention, the method comprising:

[0045] Receive an input matrix originating from a set of images;

[0046] Initialize the points in the 3D scene;

[0047] Use the points and the input matrix to extract ray-to-point features;

[0048] The ray-to-point features are aggregated to generate a feature map representing surface-ray intersections;

[0049] Using a convolutional neural network and taking the feature map as input, an image prediction is generated; and

[0050] The image prediction is output as a representation of the 3D scene.

[0051] This document has conceived and described these and other embodiments. It should be understood that the above abstract only lists representative aspects of the system and method and is intended to assist those skilled in the art in understanding the detailed description below. Attached Figure Description

[0052] The features of the invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, wherein:

[0053] Figure 1 is a schematic diagram of a system for accelerating computation of layers used in a machine learning model and rendering differentiable points using proximity attention, according to various embodiments.

[0054] Figure 2 is a flowchart of a method for accelerating computation of layers used in a machine learning model according to an embodiment;

[0055] Figure 3 shows an example implementation of the previously performed attention operation method and compares it with the attention operation method of Figure 2;

[0056] Figures 4A to 4C show example diagrams of an end-to-end learnable point-based rendering pipeline, where Figure 4A shows ray and point features, Figure 4B shows the attention layer, and Figure 4C shows the generator.

[0057] Figure 5 is a flowchart illustrating a differentiable point rendering method for learning a 3D scene representation using proximity attention, according to one embodiment.

[0058] Figures 6A and 6B show examples of learned point coordinates and features, where Figure 6A shows a reference scene and Figure 6B shows the learned point cloud of that scene;

[0059] Figure 7 is an example diagram of a neural network architecture used to perform at least part of the method shown in Figure 5;

[0060] Figure 8 shows a qualitative comparison of the method in Figure 5 with the baseline and other methods on the NeRF synthetic dataset;

[0061] Figure 9 is a graph illustrating the impact of ablation studies on the number of points on the quality of rendered images. The results show that increasing the number of points can improve the quality of the rendered images. Figure 5 The performance of the method shown;

[0062] Figure 10 is a graph showing the ablation study of ray-dependent point embedding using the method shown in Figure 5. Detailed Implementation

[0063] The embodiments will now be described with reference to the accompanying drawings. For the sake of simplicity and clarity, reference numerals may be repeated in the drawings where appropriate to indicate corresponding or similar elements. Furthermore, numerous specific details are set forth herein to enable the reader to thoroughly understand the embodiments described herein. However, those skilled in the art will understand that the embodiments described herein can be practiced even without these specific details. In other instances, well-known methods, steps, and components have not been described in detail to avoid obscuring the embodiments described herein. Moreover, the description herein should not be construed as limiting the scope of the embodiments described herein.

[0064] Unless the context otherwise requires, the various terms used in this specification shall be understood and interpreted as follows: the word "or" is inclusive, as if written as "and / or"; singular articles and pronouns are inclusive of their plural forms, and vice versa; similarly, gender pronouns are inclusive of their corresponding gender pronouns, and therefore nothing described herein should be limited to use, implementation, performance, etc., of a single gender; "exemplary" should be understood as "illustrative" or "exemplary" and does not necessarily imply superiority over other embodiments. Further definitions of terms may be provided herein; these definitions may apply to both prior and subsequent occurrences of these terms, as should be understood from reading this specification.

[0065] Any module, unit, component, server, computer, terminal, engine, or device that executes instructions as described herein may include any otherwise readable medium accessible to the computer, such as storage media, computer storage media, or data storage devices (removable and / or non-removable) and such as disks, optical discs, or magnetic tapes. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing computer-readable instructions, data structures, program modules, or other data information. Examples of computer storage media include RAM, ROM, EEPROM, flash memory, and other storage technologies; CD-ROM, digital versatile optical disc (DVD), and other optical storage media; magnetic tape, disk storage, and other magnetic storage devices, and any other media that can be used to store desired information and is accessible to applications or modules. Any such computer storage media may be part of a device, whether accessed or connected to it. Furthermore, unless the context explicitly states otherwise, any processor or controller described herein may be a single processor or multiple processors. Multiple processors may be arrayed or distributed, and any processing function mentioned herein may be executed by one or more processors, even if only a single processor is used in the examples. Any method, application, or module described herein may be implemented using computer-readable / executable instructions that may be stored in a computer-readable medium or otherwise stored and executed by one or more processors.

[0066] The following content is generally related to machine learning models, and more specifically, to a method and system for accelerating computation of layers used in machine learning models and for rendering differentiable points using proximity attention.

[0067] In machine learning models, the mathematical principle of attention operations is: inputting three matrices, Represent the key, query, and value respectively, and output a matrix. Each key is associated with a value. The key-value pair is called a token. In some cases, the attention operation can also accept a mask. As input, its elements are either 0 or 1. Matrix , , and The i-th row is denoted as , , and , representing the i-th key, i-th query, i-th value, and i-th output, respectively. Mask matrix The element in the i-th row and j-th column is denoted as... This indicates whether the i-th query is allowed to follow the j-th key; if it is 1, following is allowed; if it is 0, following is not allowed. (When the mask...) When not specified, it is typically assumed that all its elements are 1. The key and the query have the same dimensions. Each key is associated with a value, therefore the number of keys and values ​​is the same, denoted as . .

[0068] In most cases, attention operations can be performed by first determining an attention weight matrix. The element in the i-th row and j-th column of this matrix is ​​usually denoted as... It can be calculated using the following formula:

[0069] (1)

[0070] in A kernel function is a function that returns a score that measures the similarity between input parameters. For example, a kernel function can be a dot product, i.e. or negative Euclidean distance .function It can be viewed as a vector-valued function For different The different components. In other cases, it can be replaced by another vector-valued function whose components are all non-negative and whose sum is a fixed constant; for example, the softmax function, the sparsemax function, or any other function that will... 3D vector space mapping to A simplex function of dimension 1.

[0071] Attention operations can combine values ​​with attention weights in the following ways:

[0072] (2)

[0073] To distinguish the mask The cases where attention is specified and unspecified are sometimes referred to as masked attention, while the latter can be called attention or ordinary attention. A special case of masked attention is causal attention, where the mask is 1 for all keys located before the query and 0 for the rest; that is, if... ,but ;if ,but .

[0074] Within ordinary attention mechanisms and masked attention mechanisms, there is a special case called self-attention, characterized by always having the same number of keys and queries. In self-attention mechanisms, the i-th key and the j-th query are typically functions of the same vector. Ordinary attention mechanisms, ordinary self-attention mechanisms, and causal self-attention mechanisms form the basis of a class of neural network architectures called Transformers.

[0075] In many neural network models, the number of bonds and the number of queries These can all be quite large. For example, in language models, to generate a reasonable continuation of a given text prompt, each key represents a word in the prompt or a previously generated word, while each query represents a word currently being generated. To generate coherent text, such models need to capture long-range dependencies, which typically requires each query to pay attention to a large amount of context related to the prompt and previously generated words. This is usually computationally infeasible due to the computation and storage of attention weights. and The time and space required for their gradients are respectively and This is related to the quadratic length of the context. Consequently, when… and When the length is large, the computational cost of attention becomes very high in both time and space. Therefore, the computational power of modern hardware often limits the context length, thus preventing the full modeling of long-range dependencies.

[0076] In Transformer-based architectures, various methods have attempted to address this problem. These methods typically rely on one of two strategies: sparse approximation or low-rank approximation. Sparsity-based methods usually focus only on a subset of key values, while low-rank methods typically approximate the attention weight matrix using the product of a high-order and a low-order matrix. Most of these methods require modifications to the architecture to constrain the model's computation or increase the learnable parameters, thus necessitating retraining. As a result, these methods cannot be readily applied to pre-trained, unmodified models.

[0077] In many cases, retraining a model is neither practical nor economical because training modern models on large datasets requires significant computational resources. Instead, methods that do not require retraining are more advantageous. Currently, all such methods are based on low rank; for example, multiplying the key and query matrices with a random projection matrix, or truncating the minimum significant singular value of a predefined submatrix. These methods tend to lead to high approximation errors because the constraint rank introduces dependencies between different keys and different queries that might not otherwise exist. On the other hand, sparsity-based methods generally do not suffer from these problems.

[0078] However, sparsity-based methods that do not require retraining face several challenges. On one hand, sparse approximation requires identifying the most important keys for each query, and these keys often differ across queries. On the other hand, since learnable parameters cannot be introduced, the method for identifying these keys must be consistent with the data. To avoid missing important keys, the system could iterate through all keys and calculate the importance of each key to the query. However, this method has a quadratic computational complexity, which cannot handle a large number of keys and queries; therefore, it may be necessary to find a method that can identify the importance of keys without actually calculating their importance.

[0079] As mentioned earlier, the machine learning field has invested heavily in improving the computational efficiency of Transformers by reducing the quadratic time or space complexity of self-attention mechanisms. These methods are collectively referred to as efficient Transformers. They can be categorized along two dimensions: method type and compatibility with pre-trained models. The first dimension includes sparsity-based methods and low-rank methods. The second dimension includes both methods that cannot be directly applied to unmodified pre-trained Transformer models (requiring retraining) and other methods that can be directly applied to these models.

[0080] Sparsity-based methods employ sparse attention mechanisms to capture global information and combine it with local attention results. Notable examples include the Sparse Transformer, Longformer, and Extended Transformer Construction (ETC). A major limitation of these models is that the terms of attention are predefined and invariant, making them unsuitable for different input sequences. Since the original attention operation allows attention to any term, these models must predefine the constraints on the terms of attention during training. Reformer can group terms into blocks using Locality Sensitive Hashing (LSH) and then only pay attention to terms located in the same block as each query and its neighboring blocks, thus allowing for different sets of terms to be paid attention to for different input sequences. However, Reformer imposes two constraints not present in the original attention operation: the key must be normalized, and the query and key must be identical. Therefore, Reformer must embed these constraints during training. Consequently, these methods cannot be directly applied to pre-trained, unmodified models. That is, to use these methods, the model must first be retrained with the required constraints.

[0081] Low-rank methods approximate the attention weight matrix using a low-rank matrix, thereby reducing quadratic time and space complexity. Examples of low-rank Transformers include the Linformer and Performer, which decompose the attention weight matrix into the product of a high matrix and a wide matrix consisting of learned linear or random features of the key and query. However, these Transformers often introduce significant approximation errors because the attention weight matrix produced by the original attention operation (especially with long input sequences) typically has a high rank. Models using these methods must be trained with a built-in low-rank approximation to learn robustness to the relevant approximation errors. Therefore, these methods cannot be directly applied to pre-trained, unmodified models; in other words, the ability to use these methods requires retraining the model with the desired approximation method.

[0082] Other methods offer more general approaches that directly utilize pre-trained weights from standard Transformers without requiring retraining. These Transformers accelerate the execution of standard attention operations without altering the underlying architecture. For example, Nyströmformer and LARA replace the softmax structure in self-attention mechanisms with the product of the query and key matrices, which are activated separately. Nyströmformer employs the Nyström method, while LARA combines stochastic attention (RA) and stochastic feature attention (RFA) to reconstruct the attention weight matrix. Another example is H-Transformer-1D, which recursively partitions the attention weight matrix into blocks and truncates small singular values ​​in each off-diagonal block. All these methods leverage low-rank approximations rather than sparsity.

[0083] The inventors have determined the attention weight matrix. Typically, the attention weight matrix is ​​sparse, meaning that in each row of the matrix, only a few attention weights have significant (large) values, while most of the remaining values ​​are close to zero. This sparsity of the attention weight matrix is ​​consistent with the function of the attention operation: only a few words in the input sequence are highly relevant to the given context, and the model selectively focuses on these words when processing the sequence.

[0084] Table 1 presents example evaluations of the sparsity of the attention matrix learned on five different tasks in the LRA benchmark. We experimented with three different settings: retaining only the three words with the highest weights in each row of the attention weight matrix (top 5, top 10, and top 20), and setting all other attention weights to zero. The results show that even with very long input sequences (over 1000 words), focusing on only 1% of the words with the highest weights can, on average, cover words that account for 89% of the total attention weights. This phenomenon can be used to improve computational efficiency, especially with long sequences, because it can reduce the number of words that need to be merged through the attention mechanism without significantly reducing model performance.

[0085] The embodiments of this disclosure effectively utilize this sparsity, thereby accelerating Transformer inference while reducing its memory overhead. In this way, the embodiments of this disclosure provide a sparsity-based method that does not require retraining.

[0086] Table 1

[0087] Quantitative assessment of sparsity of attention matrix

[0088]

[0089] Figure 1A schematic diagram of a system 100 for accelerating computations of layers used in a machine learning model and rendering differentiable points using proximity attention, according to various embodiments, is shown. As shown, system 100 has multiple physical and logical components, including a processing unit (“PU”) 160, local data storage 164, input / output (I / O) interface 168, network interface 176, and a local bus 184 enabling PU 160 to communicate with other components. PU 160 may include one or more central processing units, one or more graphics processing units, a microprocessor, dedicated hardware, or other integrated processing circuitry. Local data storage 264 provides relatively fast-response storage for PU 160. I / O interface 168 allows for direct or indirect input and output; for example, input may be sent directly through a user input device or indirectly through an external device, and output may be sent directly through a user display or indirectly through network interface 176. Network interface 176 allows communication with other systems or computing devices, such as via a local area network or the Internet. Local data storage 164 can store operating systems and programs, including computer-executable instructions for implementing the methods described herein, as well as any derived or related data. In some cases, this data may be stored in database 188. In some cases, during system 100 operation, the operating system, programs, and data can be retrieved from database 188 and placed into local storage 164 for easy execution. In other embodiments, any operating system, program, or instruction may be executed in hardware, a dedicated microprocessor, a logic array, etc. Although... Figure 1 The system is shown as implemented on a single computing device, but it should be understood that the processing or any function of system 100 can be distributed across multiple computing devices, such as in a cloud computing or distributed computing environment.

[0090] In one embodiment, PU160 can be configured to execute multiple concept modules 101, such as input module 102, embedding module 104, search module 106, value module 108, output module 110, initialization module 112, extraction module 114, attention module 116, and generator module 118. Furthermore, the functions of the above modules can be combined or executed on other modules. In some cases, the functions of the above modules can also be executed on remote computing devices, such as centralized servers and cloud computing resources communicating via network interface 176.

[0091] Figure 2 illustrates a method 200 for accelerating layer operations in a machine learning model according to one embodiment. Specifically, method 200 is used to accelerate attention operations in a machine learning model. This method may be simply referred to as "DCI-Attention".

[0092] In block 202, input module 102 receives rows or columns of a weight matrix, an input vector, and columns of an identity matrix, wherein each row or column of the weight matrix is ​​associated with a value vector. In the specific case of the attention mechanism, input module 102 receives a key matrix, a query matrix, and a value matrix, wherein each key is associated with a value.

[0093] In the 204th module, embedding module 104 transforms the rows or columns of the weight matrix and the input vector using an embedding function. In the specific case of the attention mechanism, embedding module 104 transforms the keys in the key matrix and the queries in the query matrix using the embedding function. In this way, the keys and queries are embedded into a higher-dimensional space.

[0094] Here are some (not all) examples of available embeddings:

[0095] make and These represent the key embedding function and the query embedding function, respectively.

[0096] First example of an embedded function:

[0097] set up and The definition is as follows:

[0098]

[0099]

[0100] in It is the largest norm among all data points.

[0101] Second example of an embedded function:

[0102] set up and The definition is as follows:

[0103]

[0104]

[0105] in, It is the largest norm across all data points and queries.

[0106] The third example of an embedded function:

[0107] set up and The definition is as follows:

[0108]

[0109]

[0110] in, It is the largest norm among all data points.

[0111] Fourth example of an embedded function:

[0112] set up and The definition is as follows:

[0113]

[0114]

[0115] in, It is the largest norm among all data points.

[0116] The above embedding function is most suitable for the case where the kernel function is a dot product, that is, These embedding functions can be used when the kernel function is different; moreover, different embedding functions can be used when the kernel function is different. In the latter case, a new embedding function can be derived based on the specific kernel function. For example, each symmetric positive definite kernel function can be coupled to a finite-dimensional or infinite-dimensional (possibly random) feature map. Related, making or The feature map can also be approximated, for example, by using a low-dimensional feature map derived from the original feature map. The original or approximate feature map can then be generated using the aforementioned embedding function or other embedding functions, thus obtaining a new embedding function. Alternatively, the embedding function can be directly derived by designing each component of the embedding such that the distance between the resulting embedding vectors is mathematically related to or equivalent to the kernel function.

[0117] In block 206, search module 106 identifies the transformed rows or columns in the weight matrix that require high post-activation values, based on each transformed input vector. In specific cases of attention operations, search module 106 identifies a subset of keys that may receive high attention weights by searching the transformed keys for each transformed query (e.g., with sublinear query time complexity). For example, for each transformed query, the transformed keys can be efficiently... Nearest neighbor search is used to identify the transformed keys that are important to each transformed query. Nearest neighbor search returns the nearest neighbor to the input query. There are several points. Other suitable search methods and algorithms can also be used. In the example of nearest neighbor search, for each transformed query, Nearest neighbor determination and search return For each transformed query, find the index of the nearest transformed key and its Euclidean distance to the transformed query.

[0118] Other search algorithms can also be used to select these points based on distance or similarity measures to the query or other points. For example: (1) an algorithm that returns all points or subsets within a specific radius of the query or whose similarity to the query exceeds a threshold; (2) an algorithm that finds points that are close to or similar to the query (but may not be the closest points) and returns them; (3) an algorithm that clusters points that are close to or similar to each other and returns the points in the cluster that are close to the query; (4) an algorithm that constructs a graph with points as nodes, connects nearby or similar points with edges and traverses the graph; and (5) an algorithm that estimates the probability density of points and returns points that are close to or similar to the query whose probability density exceeds a threshold. The distance or similarity measures used in these algorithms can be computed in the space contained in the transformed key or query, or in other derived spaces, such as the projection of the original space onto a subspace, or a linear or nonlinear transformation of the original space (whose dimension may be different from the original space).

[0119] In some cases, the input module 102 may also receive a mask matrix as input, and only the unmasked keys may be used in the search.

[0120] In block 208, search module 106 is activated after using the similarity of rows or columns of the weight matrix returned by the search on the associated input vectors, as measured by a kernel function. In specific cases of attention operations, search module 106 uses its identified keys and associated queries to determine attention weights. Attention weights used to accelerate attention operations can be determined by applying the attention operation to a subset of identified keys (rather than all keys). In some cases, attention weights are modified to consider attention weights for searching unidentified keys. If the kernel function differs from the distance or similarity metric returned by the search, search module 106 can either directly compute the kernel function value based on the key and query, or convert the returned distance or similarity score into a kernel function value. For example, the kernel function could be the inner product between the key and query, while the distance returned by the search algorithm could be Euclidean distance. Thus, search module 106 can either directly compute the inner product based on the key and query, or derive the inner product from the search results. For example... Nearest neighbor search may return Euclidean distance instead of inner product, and converting Euclidean distance to inner product may be more efficient than calculating inner product from scratch.

[0121] In block 210, under certain circumstances, value module 108 determines an output matrix in the following ways: when the value vector is a column vector of the identity matrix, the output matrix is ​​made equivalent to the post-activation value; otherwise, when the value vector is not a column vector of the identity matrix, the post-activation value is combined with its corresponding value vector. In specific cases of attention operations, value module 108 determines the output matrix by combining attention weights with values.

[0122] To account for the attention weights of keys not identified in the search, as confirmed in module 208, search module 106 may, in some cases, treat the kernel function determined for each unidentified key and query as a constant, and then determine the attention weight based on this assumption. In some cases, the value of this constant may be fixed. In other cases, the value of this constant can be determined in various ways, such as by randomly sampling a subset of keys independent of the search or based on the search results. In determining this constant, search module 106 may obtain the values ​​of the kernel functions corresponding to these sampled keys and queries, and then determine the constant through some combination (e.g., averaging). In some cases, search module 106 may also treat these unidentified keys differently; for example, treating the kernel functions corresponding to different unidentified keys and queries as different constants. The value of this constant can be based on the obtained value of the kernel function, or it can use other functions that depend on the key and query.

[0123] Similarly, for the output matrix of module 210, value module 108 may, in some cases, treat the value vectors associated with the searched unidentified keys as a constant vector and determine the output vector based on this assumption. In some cases, the value of this constant vector may be fixed. In other cases, the value of the constant vector can be determined in various ways, such as by randomly sampling a subset of value vectors independent of the search or based on the search results. In still other cases, the determination method may include sampling a subset of value vectors based on the subset of keys sampled in module 208 and combining each value vector in some way (e.g., by averaging). Value module 108 may also treat the value vectors associated with different unidentified keys differently, for example, treating each value vector as a different constant vector. The value of this constant vector may be based on a function applied to the associated key, value, or query.

[0124] In some cases, such as when the values ​​are column vectors of the identity matrix, the performance of block 210 can be omitted because the output matrix consists of attention weights.

[0125] In block 212, output module 110 outputs the post-activation value (attention weight in the case of attention operation), output matrix (representing the result of attention operation in the case of attention operation), or both to local data storage 164, database, I / O interface 168, and network interface 176.

[0126] In another embodiment, blocks 208 and 210 can be executed in parallel with block 206, wherein attention weights and output vectors are determined as new keys are identified in block 206.

[0127] Search module 106 performs search examples including: processing one or more items in the set of items to be searched in a certain way, and using the processing results to determine which items to process next or to stop the search. It can process items sequentially or in parallel. It does not necessarily need to process all items in the same way, nor does it necessarily need to process the same set of items in the same way every time.

[0128] Other uses Unlike nearest neighbor search methods for identifying keys, the advantage of this method is that it does not require key normalization; therefore, there is no need to retrain the model using such constraints. As demonstrated in this paper, the inventors have determined that high-dimensional embeddings of keys and queries (also known as transformed keys and transformed queries) are used for... Nearest neighbor search is equivalent to determining The key with the highest inner product in the query.

[0129] Because the attention weight matrix is ​​sparse, System 100 can merge key values ​​associated with high attention weights. The challenge lies in identifying which keys have high attention weights without calculating the attention weights for all keys. The observation that the attention matrix of a self-attention mechanism is sparse provides a clue as to how to identify the keys that receive high attention weights from each query in the attention weight matrix. Note:

[0130] (3)

[0131] (4)

[0132] (5)

[0133] (6)

[0134] (7)

[0135] Therefore, the key with the highest attention weight The key to maximize In neural network architecture, for To put it simply, a specific choice is .

[0136] For all In other words, when :

[0137] (8)

[0138] (9)

[0139] (10)

[0140] (11)

[0141] (12)

[0142] (13)

[0143] (14)

[0144] (15)

[0145] Therefore, after all keys are normalized, the key with the highest attention weight is the key closest to the query. Similarly, having... The keys with the highest attention weights are those closest to the query. Each bond. Therefore, an efficient and possibly approximate... Nearest neighbor search ( The -NNS algorithm can be used to find keys where the key is a data point and the query is a... -NNS query points. For example, this feature can be leveraged by using Locality Sensitive Hash (LSH). - Nearest neighbor search. However, since this property only holds true if all keys are normalized, other methods (such as Reformer) require this constraint to be satisfied. Therefore, Transformer models with unnormalized keys need to be retrained to adapt to this constraint for compatibility with Reformer.

[0146] In contrast, Method 200 can identify keys with high attention weights, thus making it applicable to the general case, rather than just the case where keys are normalized. Below is an example of a key and query embedding function, demonstrating the effectiveness of key and query embedding-based methods. Nearest neighbor search ( -NNS) can recognize The key with the highest inner product in the query.

[0147] set up and The definition is as follows:

[0148] (16)

[0149] (17)

[0150] in It is the largest norm among all data points.

[0151] (18)

[0152] (19)

[0153] (20)

[0154] (twenty one)

[0155] (twenty two)

[0156] (twenty three)

[0157] (twenty four)

[0158] (25)

[0159] (26)

[0160] (27)

[0161] (28)

[0162] The key with the highest attention weight It is its transformed version. With the transformed query The closest bond. Therefore, an efficient and potentially approximate bond can be used. -Nearest Neighbor Search -NNS) to identify keys with high attention weights If the system's runtime is less than 100... If a large number of queries share the same set of keys, then system 100 does not need to traverse all keys for each query to perform attention operations. Since method 200 does not impose any constraints on keys or queries, it can serve as a direct alternative to attention operations in machine learning models without retraining the model.

[0163] Generally speaking, it is used to accelerate attention calculations. - The nearest neighbor search algorithm should ideally (but is not required to) have the following characteristics.

[0164] First, since the sum of the attention weights of all keys for a given query must be a constant (usually 1), there will inevitably be some keys with high attention weights regardless of the query. However, some approximate nearest neighbor search algorithms (such as Locality Sensitive Hashing (LSH)) may not return any data points for some queries; this occurs when no data point falls in the same hash box as the query point. Therefore, regardless of the query, -Nearest neighbor search algorithms should guarantee at least one return value. A vector.

[0165] Secondly, since the key of the next attention layer depends on the output of the previous attention layer, and the output of the previous attention layer changes with different queries, this can lead to different model inputs... - The data points for NNS can also change. Therefore, each time the model is run on new inputs, a new one may need to be constructed. -NNS database. Therefore... Nearest neighbor search algorithms should not require excessive computational overhead for data preprocessing. This effectively excludes algorithms like LSH, which requires a significant amount of time to build a sufficiently large hash table to reduce the probability of failing to return any data points. Furthermore, since queries change each time the model runs on new inputs, the preprocessing overhead cannot be amortized across different model inputs. Therefore, for the same set of keys, the sum of the preprocessing time and query time for all queries should be less than the time required to achieve the same result through brute-force computation.

[0166] Third, in order to The theoretical guarantees of the nearest neighbor search algorithm can be applied to construct theoretical guarantees for the approximate quality of accelerated attention mechanisms. These guarantees typically target the algorithm's return of accurate values. The probability of a set of -nearest neighbors, not the probability of an approximate set of k-nearest neighbors. Algorithms that guarantee the former are called exact algorithms, while those that only guarantee the latter are called approximate algorithms. For example, LSH is an approximate algorithm. The -NNS algorithm is used because it guarantees a high probability of returning a nearest neighbor, meaning the neighbor is farther from the query point than the true nearest neighbor. The number of points is several times higher. However, some real ones are missing. - Nearest neighbors can significantly impact the output of the attention mechanism because even if a nearest neighbor is not much farther from the query point, its value vector can be drastically different from that of its nearest neighbor. Therefore, to better guarantee the approximation quality of the accelerated attention algorithm, The nearest neighbor search algorithm should be accurate; that is, it should guarantee finding the nearest neighbor with a high probability. - An exact set of nearest neighbors.

[0167] Fourth, since autoregressive models (such as language models) typically use attention mechanisms, the key set expands continuously as more words are generated. Because the keys corresponding to previously generated words remain unchanged during generation, it's generally best to avoid reprocessing previous keys every time a new word is generated. Therefore, in this case, The nearest neighbor algorithm should support incremental updates to its database without reprocessing the entire database.

[0168] Finally, the dimensions of keys and queries in common neural network architectures are often high, so it is advisable to choose architectures that can scale to high-dimensional settings whenever possible. - Nearest neighbor search algorithm.

[0169] The inventors have determined that, satisfying most (if not all) of the above properties, - The nearest neighbor search algorithm belongs to the Dynamic Continuous Indexing (DCI) algorithm family. However, it should be understood that this disclosure is not limited to DCI, and any suitable algorithm may be used. -Nearest neighbor search algorithms can all be used.

[0170] Unlike other nearest neighbor search algorithms, DCI is a ranking-based algorithm, not a bucketing-based one. The latter tracks which data points fall into which bucket, and the bucket boundaries are set before the data points are seen. DCI, on the other hand, constructs different ranking methods, sorting the data points according to the probability that each data point is a true nearest neighbor of the query, and then combines the different ranking results to obtain the final overall ranking.

[0171] For the purposes of this disclosure, ranking-based algorithms are generally more suitable than bucket-based algorithms because attention weights are sensitive to relative comparisons between the similarities of different keys and queries, rather than absolute evaluations of the similarity of each key and query relative to a fixed threshold. Bucket-based algorithms effectively have a fixed similarity threshold, so a variable number of data points (including zero) may satisfy that threshold. On the other hand, ranking-based algorithms return a fixed number of data points, which is essentially equivalent to selecting a variable similarity threshold based on the data. The latter is more consistent with the general calculation method of attention weights and should better approximate the attention operation.

[0172] In a special case of method 200, the Prioritized DCI algorithm can be used, which is based on a precise version of the DCI algorithm family. - Nearest Neighbor Search Algorithm. The Prioritized DCI algorithm consists of two phases: a data preprocessing phase, used to preprocess data points and construct the data structure; and a query phase, used to answer the query. - Nearest Neighbor Query. In the data preprocessing stage, Prioritized DCI first projects data points along multiple random directions, then sorts the projected values ​​(scalars) in each direction in ascending order and stores the sorted projected values ​​and projection directions. In the query stage, Prioritized DCI projects the query value along the same projection directions as in the preprocessing stage, and visits data points in order of proximity to the query value in each projection direction. The Prioritized DCI algorithm alternates between different projection directions, but there is usually no fixed order. After visiting a point along a projection direction, the next projection direction processed by Prioritized DCI is the projection direction of the unvisited point closest to the query point in that direction. A data point may be visited multiple times along different projection directions. When a data point has been visited along all projection directions, it is considered a true neighbor. - Neighboring candidate points. When there are enough candidate points, priority DCI will return... The candidate point closest to the query point.

[0173] In some cases, a variant of prioritized DCI can be used instead of standard prioritized DCI. The difference between this variant and standard prioritized DCI is that, during the query phase, it does not access data points one at a time, but rather accesses multiple data points sequentially or in parallel. These data points can be the unvisited data points closest to the query along a given projection direction, or they can be the unvisited data points closest to the query along multiple projection directions. In the former case, it only switches to processing other projection directions after all the data points intended to be accessed at once have been accessed.

[0174] If the random projection direction The quantity is less than the data Given the inherent dimension (assuming it is at least 1), priority DCI can be applied in time. It returns the exact value of the query with a high probability. - Neighbors, among which The log factor was suppressed. The preprocessing used in prioritized DCI is very lightweight, therefore requiring only... Time. Therefore, the total time required to calculate the attention weights is If we compare it with Comparing the computational complexity of attention, we can see that it no longer depends on... The items are such that there is no longer a secondary dependency on the context length. Because... and Typically large, with the middle term usually dominating. This is if the choice is made relative to the inherent dimension. Regarding the larger number of projection directions and a smaller number of neighbor searches Then the intermediate term Much smaller This significantly improves computation speed.

[0175] Method 200, used to accelerate the attention mechanism, can be informally considered as comprising two stages. In the first stage, formulas (16) and (17) are used to apply the key... and query Perform the conversion. Unmasked keys (i.e.) ) is added to the database and executed - Nearest neighbor search (e.g., prioritizing DCI). Determine the query after each transformation. of -Nearest neighbors, search results returned are compared with each transformed query. The closest conversion key Indexes and their transformation to queries The Euclidean distance. Based on the Euclidean distance, the inner product can be calculated using formula (20); that is, .

[0176] In some cases, different queries may mask different sets of keys. In this scenario, typically only the keys that are always masked will not be added to the database; all other keys will be added. Another approach is to divide the keys into subsets and add each subset to a separate database. In both cases, the masking information (regardless of whether a key is masked for the current query) can be stored in the database along with the key and value vectors. When performing a query, the search can proceed only on unmasked keys, and the search can terminate early when there are no more unmasked and promising keys. It's important to note that while both scenarios are proposed in the context of "different queries masking different keys," they can also be used in other scenarios: for example, when all queries mask the same key, when no keys are masked, or when no masking information is provided as input. For instance, when the k-nearest neighbor search algorithm is Hierarchical DCI, the search may terminate early if there are no unmasked keys in the remaining cells to be searched.

[0177] -Nearest neighbors can be used to efficiently determine A simple approach is to construct an approximate matrix. The matrix except - All elements outside the nearest neighbors are zero. Then, formula (2) can be used to approximate the matrix. Sure However, this method is generally slow because directly executing formula (2) takes a significant amount of time. Conversely, in some cases, system 100 may only use... -Neighbors determine all Attention weights The approximate value is obtained by the following method:

[0178] (29)

[0179] in express - The index of a point in the nearest neighbor set. It can be used to linearly combine with the following methods The vector of values ​​associated with the nearest key:

[0180] (30)

[0181] In some cases, System 100 can use an improved prioritized DCI, which can be informally referred to as "Hierarchical DCI." Hierarchical DCI involves constructing a hierarchy of subsets of data points and recursively applying prioritized DCI to each level of the hierarchy. More specifically, the bottom level of the hierarchy contains all data points, some of which are selected to be promoted to the next level. Each data point at a given level is assigned a parent, i.e., a data point in the next higher level, and the parent is the higher-level data point closest to that lower-level data point; this is itself a 1-nearest neighbor search problem that can be solved using hierarchical DCI. A set of data points sharing the same parent is called a cell. During a query, System 100 starts from the top level and uses prioritized DCI to find the distance to the query. The nearest top-level point, and then relative to this For cells associated with a given point (composed of points at the next lower level), DCI (Distributed Core Intelligence) is performed. The cell closest to the query is retrieved from these cells. Each point is identified, and priority DCI is performed on the cells associated with these points, and so on.

[0182] Hierarchical DCI can select data points to be promoted to the next level based on deterministic rules, or randomly select data points from a larger dataset selected according to deterministic rules, or select data points from a larger dataset selected randomly according to deterministic rules. These deterministic rules can be based on information obtained from the data point's value, identifier, order of addition, and the number of other data points added or deleted before the relevant data point was added.

[0183] In other cases, hierarchical DCI can assign each data point to multiple parent levels or multiple levels.

[0184] The hierarchical DCI method constructs a hierarchical structure of subsets of data points and recursively applies a search algorithm (in this case, priority DCI) to each level of the hierarchy. This method can also be used for other search algorithms. The applications of the nearest neighbor search algorithm extend far beyond attention mechanisms. It can also be applied to vector databases, associative memory, collaborative filtering, clustering, vector quantization, classification, retrieval augmentation generation (RAG), near-duplicate detection, interpolation, computational geometry applications, and information retrieval applications (such as search engines and recommender systems).

[0185] Unlike other methods, method 200 does not require shared keys and queries. Other methods typically require this constraint because they use Locality Sensitive Hashing (LSH). - Nearest Neighbor Search (LSH); LSH works by dividing the vector space into cells / hash boxes, tracking which keys and queries fall into each cell / box, and returning the keys that fall into the cell / box containing the query. If the keys and queries are not shared, the cell / box containing the query may not contain any keys. To reduce the probability of this happening, a common solution for LSH is to create multiple hash tables and return all keys in all cells / boxes of all hash tables containing queries. However, this approach is not practical for accelerating the computation of attention mechanisms because building hash tables is time-consuming; since keys and queries may change each time, the hash tables usually need to be rebuilt every time attention is performed. Therefore, other methods typically require shared keys and queries to ensure that at least one key is in the cell / container containing the query (i.e., the query itself).

[0186] Method 200 involves using Dynamic Continuous Indexing (DCI) in certain situations. Nearest neighbor search avoids the aforementioned major problems. Unlike LSH, DCI does not partition the vector space; therefore, for each query, it can always return any number of keys. Furthermore, DCI does not incur heavy preprocessing overhead and can efficiently construct data structures. This allows System 100 to eliminate the constraints of shared keys and queries, thus avoiding the need to retrain the model due to these constraints. Moreover, while other methods may only be applicable to self-attention mechanisms (because shared keys and queries require the same number of keys and queries), Method 200 is applicable to any type of attention operation.

[0187] Method 200 is also applicable to other ways of defining attention weights, such as replacing the softmax layer with sparsemax or other activation functions. Method 200 can also be used to accelerate the computation of softmax, sparsemax, and other activation functions outside the attention mechanism. For these other ways of defining attention weights, Method 200 typically does not apply softmax to the subset of keys identified by the search algorithm, but instead applies sparsemax to those keys. Method 200 is generally suitable for sparsemax because both softmax and sparsemax preserve the relative order of the different components of the input vector. To approximate sparsemax, System 100 can identify the keys with the highest similarity scores, compute the attention weights given by sparsemax on these keys, and set the attention weights of other keys to zero. This significantly improves computational efficiency because it can quickly identify keys from which the search algorithm will obtain high similarity scores.

[0188] Figure 3 shows an example implementation of a previous attention operation method, referred to as "raw attention" (top row), and the attention operation of method 200, referred to as "DCI-attention" (bottom row). This example uses a query, and exist In -NNS. In the two attention matrices shown, the two attention values ​​with the largest weights in each row are represented in dark. The remaining attention weights are displayed in light colors in the original attention weight matrix, but are set to zero (displayed in white) in the attention weight matrix used to accelerate attention operations.

[0189] The inventors conducted example experiments using Method 200 on a long-range domain (LRA) benchmark set. This benchmark set contains five tasks that require learning long-range dependencies. In all five tasks, Method 200 significantly improved inference performance compared to five different baseline models, provided well-trained weights. Notably, Method 200 not only achieved accuracy close to that using a standard Transformer model but also outperformed all baseline methods in efficiency.

[0190] The performance of System 100 was tested on five tasks in the Remote Arena (LRA) benchmark: syntactic parsing, document retrieval, text classification, image classification, and pathfinding. These tasks consist of sequences of 1,000 to 5,000 lexical units, covering a variety of data types, such as text, natural and synthetic images, and mathematical expressions.

[0191] The first task, ListOps, is a summary operation that performs a series of prefix operations on a single-digit integer. The model aims to predict a single-digit integer and classify it into ten subclasses. In the example experiments, the dataset was generated using the LRA benchmark tool, with a maximum sequence length of 4000. The second task is text classification, based on a movie review dataset, aiming for binary classification. This dataset is directly taken from the LRA benchmark, with a maximum sequence length of 4000. The third task is document retrieval, based on relevant article matching from the ACL collection network. This dataset is directly taken from the LRA benchmark, with a maximum sequence length of 4000. The fourth task is image classification, aiming to classify the type of objects in an image presented as a sequence of grayscale values. This dataset is directly taken from the LRA benchmark, with a maximum sequence length of 4000. The fifth task is a pathfinder, aiming to predict whether two points in an image are connected by a path. This dataset is directly taken from the LRA benchmark set, with a maximum sequence length of 4000.

[0192] All tasks used the standard Transformer model as the baseline model. The baseline model was trained for each task. Then, example experiments applied the DCI-attention mechanism of this embodiment to the inference process of the baseline model. The performance of System 100 was compared with the standard self-attention mechanism in terms of prediction accuracy and runtime.

[0193] For each of these tasks, the current implementation has been demonstrated to significantly accelerate the inference and training time of the Transformer and reduce its memory cost.

[0194] For ease of understanding, this embodiment typically refers to the acceleration of attention operations, but it should be understood that the method of this embodiment can also be used to accelerate the operations of any suitable layer in a machine learning model. For example, this method for accelerating attention operations can also be used to accelerate the operations of softmax, sparsemax, or logistic layers in a classifier model, or to accelerate the operations of linear or convolutional layers with modified linear activation functions (ReLU) or softplus activation functions in an artificial neural network, and so on. Thus, the attention weights described herein are functionally equivalent to post-activation functions, the queries described herein are functionally equivalent to input vectors, the keys described herein are functionally equivalent to rows (if the weight matrix is ​​left-multiplied by the input vector) or columns (if the weight matrix is ​​right-multiplied by the input vector) of the weight matrix, and the values ​​described herein are functionally equivalent to column vectors of the identity matrix. If the layer's weight matrix is ​​not directly multiplied by the input vector (e.g., in a convolutional layer), the weight matrix can usually be transformed such that the mathematical operation performed is equivalent to multiplying the transformed weight matrix by the input vector. In this case, the method described herein applies to the transformed weight matrix, not the original weight matrix. For activation functions that map m-dimensional vector space to For layers with a 3D simplex structure (such as softmax, sparsemax, or logistic layers), the post-activation is often referred to as the class probabilities. Since these values ​​are columns of the identity matrix, step 210 may not need to be explicitly performed, as the output matrix is ​​the same as the attention weights.

[0195] The same or similar mechanism used in the accelerated attention operation performed by System 100 can also be used to accelerate queries in a vector database that associates each item stored in the database with a key vector and answers similarity queries by identifying key vectors that are similar to or close to the query vector, regardless of whether the similarity and distance metrics are defined by a kernel function.

[0196] In a specific application instance, the attention operation performed by system 100 can be used to perform differentiable point rendering of a three-dimensional (3D) scene using neighbor attention.

[0197] In most cases, an ideal 3D representation of a scene should satisfy some general properties, such as: (1) sufficient expressiveness to support high rendering quality; (2) high storage and rendering efficiency; (3) ease of manipulation; and (4) easy reconstruction from images captured of the scene using widely available devices, including but not limited to RGB (color) cameras, RGB-D (color and depth) cameras, or infrared (IR) cameras. Most methods focus primarily on the first three properties, paying less attention to the last one. Therefore, such 3D graphics pipelines do not support the easy conversion of captured images into 3D assets, requiring artists to build them manually. However, historically, achieving the last property has often required sacrificing some other properties.

[0198] 3D representations can be broadly categorized into two types: volume representations and surface representations. Volume representations encode scene information in each 3D coordinate, while surface representations encode scene information only in the 3D coordinates of the object's surface. Some methods are based on volume representations and possess properties (1) and (4); for example, they can generate high-quality rendered images and reconstruct scene representations from images taken from multiple perspectives, and minimize the difference between the rendered image and the captured image relative to the representation through gradient-based optimization. The ability to successfully reconstruct scene representations by optimizing for the representation is often referred to as end-to-end learnability. One of the biggest advantages of volume representations is that they can be directly learned end-to-end because the rendered image changes smoothly as the representation changes. However, volume representations are inefficient in terms of storage and / or rendering. In the worst case, volume representations require cubic levels of storage space to accurately represent the scene as the scene diameter increases. Furthermore, rendering volume data requires volume rendering, which requires sampling at multiple points along each ray. Since a typical scene consists of opaque objects and a large amount of empty space, it is only necessary to store the appearance and geometry of the object's surface and only to sample the points where each ray hits the surface. Therefore, volumetric data storage and rendering are less efficient relative to minimum storage and sampling requirements.

[0199] On the other hand, as the scene diameter increases, surface representations typically require only a quadratic increase in storage space. This is far more efficient than volume representations because it avoids storing information in empty spaces and inside objects. Examples of surface representations include polygonal meshes and point clouds located on surfaces, the latter informally referred to as surface point clouds. To fully leverage the efficiency of surface representations, a significant challenge needs to be overcome; primarily because surface representations are more difficult to learn end-to-end than volume representations, since the rendered image does not change continuously with surface normals. While polygonal meshes can accurately model geometry, their expressive power is limited because the topology of the shapes they model cannot be changed without altering the connectivity between polygon vertices. Allowing modification of connectivity is not straightforward, as enforcing the validity of meshes (e.g., planarity, watertightness) is computationally very difficult. Embodiments of this disclosure provide a method for addressing the challenge of end-to-end learning of surface point clouds from color image captures of multiple views.

[0200] End-to-end learning of point clouds typically requires a differentiable point cloud rendering method. However, rendering point clouds presents a unique and significant challenge because each point is infinitesimally small, and rays are almost impossible to intersect with any point. A common solution is to convert points into spheres, disks, or Gaussian kernels. However, choosing an appropriate radius for the sphere / disk or an appropriate standard deviation for the Gaussian kernel is very difficult. Small radii or standard deviations can lead to vanishing gradients for distant rays, while large radii or standard deviations can result in a coarse / blurry rendered image. Therefore, there is an undesirable trade-off between rendering quality and end-to-end learnability. As a workaround, some methods achieve high rendering sharpness by maintaining small point radii and compensate for the vanishing gradient problem with a large number of points (often reaching millions). However, vanishing gradients prevent many points from being moved onto the surface, resulting in a large number of points being far from the surface, thus preventing these methods from achieving the theoretically achievable representational efficiency of surface point clouds. Other methods estimate the intersection points of rays with the surface reconstructed from the point cloud. However, the intersection point of the ray and the surface does not change smoothly with the surface or the surface sampling points. In addition, surface reconstruction itself is an ill-posed problem, because for the same point cloud, there may be multiple consistent surface reconstruction results, so the estimated intersection point may not be accurate.

[0201] Some computer graphics processing workflows extract mesh surfaces from point clouds and find the intersections of rays with the mesh. The mesh extraction operation involves fitting a signed distance function (SDF) from the point cloud by solving the Poisson equation and then performing a traveling cubes algorithm to obtain the mesh. This extraction process is error-prone, and more importantly, it is non-differentiable; therefore, it prevents gradient backpropagation to the point cloud.

[0202] Embodiments of this disclosure provide an end-to-end method for learning point clouds that avoids the drawbacks of the two point cloud rendering methods described above. Instead of making the gradient magnitude of a point depend on its absolute distance to a ray, System 100 makes it depend on the distance of that point relative to other points on the ray. System 100 also does not directly estimate intersections, but instead learns a feature vector for each point and estimates which point features should be combined to generate the desired color. An attention operation is used to identify which points the ray should focus on; this is informally called "proximity attention." In proximity attention, each key is a function of the point features associated with the ray, including the displacement of that point to the nearest point on the ray. Each query is a function of the ray direction. Each value is a function of the point features associated with the ray and the learned feature vector.

[0203] Using the method described above, System 100 can learn high-quality point clouds from scratch, unlike other methods that require initializing point clouds using other 3D reconstruction algorithms such as Structure-of-Motion (SfM) or Multi-View Stereo (MVS) algorithms. Furthermore, System 100 can move the point cloud towards a surface during training, thereby generating surface point clouds. In example experiments, System 100 achieves high-quality rendering even with only a few thousand points; while other point-based renderers require hundreds of thousands of points to render a high-quality image. Moreover, System 100 can automatically adjust the learned point cloud density based on geometric complexity. System 100 can also learn feature vectors for each point, which capture local geometry and material properties.

[0204] This system generates high-quality rendering results with only a small number of points, which facilitates user manipulation and editing of 3D geometry and scenes. Surface continuity is preserved even when non-volume-preserving transformations (such as bending and stretching) are applied. Furthermore, the learned eigenvectors allow users to quickly select all points with the same material and representing similar geometric surfaces. Users can then perform holistic operations on all these points. Users can copy or delete objects or portions of objects by copying or deleting points and their associated eigenvectors. Finally, users can also manipulate the eigenvectors of each point based on the eigenvectors of other points, thereby copying and / or swapping the appearance and / or material properties of different parts of a 3D scene to another part.

[0205] Unlike other differentiable rendering methods that allow users to edit 3D geometry and scenes, System 100 can be trained on a single, unedited 3D scene. Notably, System 100 does not require training with multiple variations of the scene, the differences between which are analogous to the editing operations a user can perform.

[0206] To replicate and / or exchange appearance and / or material properties, principal component analysis (PCA) or related methods, including but not limited to independent component analysis (ICA), factor analysis, neighborhood component analysis (NCA), clustering, and autoencoders, can be applied. These methods can be applied to the set of feature vectors associated with the point where the property will be replicated (informally called the "source point") and the point where the property will be copied (informally called the "target point"). If PCA is used, the first few principal components of the target point... The first few principal components of the source point Replacement is performed, while the coefficients of all principal components can remain unchanged. In some cases, the non-first few principal components of the target point... The coefficients will be set to zero. If another method is used, the process is similar, treating the basis vectors or basis functions as principal components and their coefficients as those of the principal components.

[0207] To model scene variants, scenes with similar objects, or structural and dynamic scenes with moving objects or object parts, point coordinates and feature vectors of different 3D scenes or different variants of the same scene can be shared by training objectives or otherwise correlated. For example, in a scene with moving objects, the feature vectors of the same object at different time points can be shared or tend to be similar through training objectives, and the point coordinates of the same object at different time points can also tend to be close.

[0208] The developed differentiable renderers (DRs) can be used to learn scene representations. Represented by scene and camera posture As input, and output rendered image The learning objective is to minimize the rendered image. Compared with the observed true value Losses between To learn scene representations end-to-end. Differentiability of scene representation It is crucial because it allows the gradient to be computed. Backpropagation to scene representation Differentiable renderer and scene representation It can also be parameterized through neural networks, and these neural network rendering methods can generate photorealistic output renderings.

[0209] Figures 4A to 4C are schematic diagrams showing an example overview of an end-to-end learnable point-based rendering pipeline. Figure 4AFigure 4A shows the ray and point features, Figure 4B shows the attention layer, and Figure 4C shows the generator.

[0210] Figure 5 is a flowchart illustrating a method 500 that utilizes neighbor attention to learn a representation of a 3D scene, thereby enabling differentiable point rendering.

[0211] In module 502, input module 102 receives an input matrix derived from a set of camera extrinsic and intrinsic parameters associated with a set of images. The input can be from a set of... Input matrix of images The combination, in which It is an RGB image. It is a camera rotation matrix. It is the camera translation vector. This refers to camera intrinsics. While this disclosure generally refers to RGB images, it should be understood that any suitable color model and associated image can be used.

[0212] Typically, camera intrinsic parameters refer to camera-specific information that does not depend on the camera's position or orientation; for example, focal length, width, and height per pixel. Camera extrinsic parameters, on the other hand, refer to information that depends on the camera's position and orientation.

[0213] The input image set typically captures the same 3D scene, i.e., the object to be reconstructed, and is usually taken from multiple positions and angles. For example, to capture these images, multiple cameras can be placed at different locations within the 3D scene and their angles adjusted to point at the portion of the 3D scene that needs to be reconstructed. Alternatively, one or several cameras can be used, and they can be moved sequentially to different positions and orientations to capture images.

[0214] Translation vectors typically characterize the position of each camera relative to a point in the scene (usually defined as the origin). For example, translation vectors can be provided if the cameras are mounted on a fixed shooting device and calibrated accordingly. In other cases, translation vectors can be estimated from images taken by all cameras using structure-of-motion (SfM) algorithms and / or sensor measurements (e.g., accelerometer readings). For example, camera positions can be determined by the following steps: (1) taking images using cameras at different positions in the scene; (2) performing a structure-of-motion (SfM) method (e.g., using the COLMAP software package) on the set of captured images, which can estimate the camera position and orientation for each image; and (3) outputting the estimated camera positions for all images. Other suitable steps and methods are also feasible.

[0215] Camera intrinsics can be obtained from camera specifications or from the metadata of the captured images. Camera intrinsics may include information such as focal length, width and height of each pixel. Camera intrinsics can also be estimated using structure-of-motion (SfM) software (e.g., COLMAP). For example, the steps to determine camera intrinsics are as follows: (1) Take images using the camera at different positions in the scene; (2) Perform a structure-of-motion (SfM) method (e.g., using COLMAP software) on the captured image set, which can estimate the camera position and orientation of each image; (3) Output the estimated camera intrinsics for all images.

[0216] A scene can be represented as a set Points The point has the learned position. and neural feature vectors .

[0217] In module 504, initialization module 112 can initialize points; for example, points randomly distributed in a cube. Point initialization is typically necessary because point coordinates are updated iteratively, and the changes in each iteration are usually small and incremental. Any suitable initialization method can be used, whether it's non-informative or informative. Non-informative initialization does not consider any information about scene geometry and camera layout when determining how to initialize, and can include randomly initializing points distributed on a sphere or any other arbitrary geometry. Informative initialization can consider information about scene geometry and camera layout, such as points uniformly distributed in the convex hull of the visible portion of space, points reconstructed by a Structure of Motion (SfM) algorithm or other algorithms, or points acquired by a real-world depth sensor (e.g., LiDAR).

[0218] In another example, in block 504, initialization module 112 can initialize points and neural feature vectors from the associated points and neural feature vectors of a machine learning model previously trained on the current scene or a different scene using method 500 or other similar methods.

[0219] Neural feature vectors Local scene attributes Encoding, for example, local geometry and material properties. During training. and End-to-end optimization was performed on all. Figures 6A and 6B show examples of the learned point coordinates and features, with Figure 6A showing the reference scene. Figure 6B The corresponding point cloud is displayed. Figures 6A and 6B show the point coordinates learned in a scene containing two hot dogs. and characteristics A visualization example. In this example, for ease of visualization, t-SNE is used to represent the feature dimensions. Reduced to 3.

[0220] generally and These are variables that remain constant throughout the training process and can be iteratively updated along with the model weights during training. In some cases, these variables can be randomly initialized to arbitrary values. Then, during training, these two variables can be updated along with the weights of the machine learning model, for example, using gradient-based optimization algorithms. During training, the machine learning model is typically designed to render a 3D scene representation (consisting of learned point coordinates and feature vectors) from a camera associated with the training image, minimizing the difference between the rendered image and the training image. To minimize this difference, system 100 determines the gradient of this difference with respect to the model weights, point coordinates, and feature vectors, and updates the variables using the gradient function. Each update can be an iterative update. In another case, some or all of the point coordinates or neural feature vectors are the output of other input functions, such as the coordinates of other points, the point coordinates of the previous frame in the 3D scene time series, or the parameters of a parameterized model for a shape category. In this case, instead of updating the outputs of these functions, the parameters of these functions are updated.

[0221] In Method 500, the cumulative effect of small iterative updates can generate a set of point coordinates (called a “point cloud”) that corresponds well to the real geometry of the 3D scene, even if the point coordinates are initialized with no information. Therefore, Method 500 can be described as “end-to-end learnable.” In contrast, other methods typically fail to learn the correct point cloud without initialization information and usually require informational initialization to obtain a reasonably configured point cloud; these methods often require prior knowledge of the scene geometry and / or camera layout.

[0222] Before rendering a 3D scene, point coordinates or neural feature vectors can be modified. This results in changes to the geometry and appearance of the rendered scene, reflecting the modified point coordinates or neural feature vectors. The inventors observed that modifying point coordinates leads to changes in geometry, while modifying neural feature vectors leads to changes in appearance. These modifications can be performed by the user or by an algorithm. For example, a user can modify the coordinates of some points, and then automatically calculate the coordinates of other points based on the user-modified coordinates and the original coordinates of all points. Alternatively, method 500 can be used to optimize point coordinates and neural feature vectors in another 3D scene while freezing some or all of the model's weights. Yet another example is the use of algorithms to process point coordinates and generate another set of point coordinates with desired properties (e.g., smoothness or local stiffness). Different modifications can be combined. Modifying point coordinates or neural feature vectors can be used for geometry editing, texture editing, texture transfer, 3D scene interpolation, interpolation editing, 3D motion estimation, motion editing, animation, etc.

[0223] In block 506, extraction module 114 extracts features from rays to points, thereby enabling end-to-end learning of the point cloud from scratch.

[0224] No. One camera Each pixel in the captured image projects a ray. ,in It is from its origin and observation direction Defined as a unit vector in the world coordinate system, this vector Obtained from the defined pinhole camera model. However, any suitable camera model can be used. For ray-point pairs Extraction module 114 found the ray. of projection :

[0225] (31)

[0226] Extraction module 114 Confirm arrive displacement and from arrive Displacement:

[0227] (32)

[0228] in and It is a characteristic of a ray reaching a point.

[0229] In the 508th module, the attention module 116 aggregates the features from the ray to the point and learns the surface ray intersection.

[0230] For each Attention module 116 from the ray that is approached in some way Select A group of points The criterion for selecting a point can be: the minimum orthogonal distance between the point and the ray. The minimum projection distance from the camera image plane to the ray along the ray, the minimum projection distance from another real or virtual camera image plane to the ray, or about and The minimum or maximum value of a certain function. and Two displacements and Transformation can be performed using positional encoding. This is to learn high-frequency functions in low-dimensional problem domains. In another case, or Normalization can be performed before applying the positional encoding transformation. Alternatively, it can be calculated before applying the positional encoding transformation. or The norm of . In another case, the norm of can be calculated before using the position-encoded transformation. or A function of. Value vector descriptor. Key vector descriptor and query vector descriptor Each can consist of one or more of the following (but) , and It can be a combination of different subsets of the following: constants, scalars or vectors, displacements. Displacement Normalized displacement Normalized displacement The norm of displacement The norm of displacement Regarding displacement A function of its norm, with respect to displacement And a function of its norm, and the positional encoding of any of the above quantities. When using the norm In this case, it should be understood that any norm can be used. Value vector descriptor It can also include neural feature vectors Key vector descriptor It can also include point locations. A certain constant zoom point position Normalization point location / A function relating point location and its norm, and the positional encoding of any of the above quantities. Descriptor of the query vector. It can also include the direction of the ray. A certain constant Scaled ray direction The positional encoding is a function relating to the ray direction, and any of the quantities mentioned above. It should be noted that any suitable positional encoding can be used; for example, other basis functions (including but not limited to wavelets, polynomials, spline functions, spherical harmonics, radial basis functions, etc.) can be used to replace or supplement the sine and cosine functions in the positional encoding.

[0231] In specific cases, the attention module 116 can pass descriptors to three multilayer perceptrons (MLPs) or other neural networks and provide the output of the MLPs to the attention layer. This method can be described as follows:

[0232] (33)

[0233] (34)

[0234] (35)

[0235] A single layer of single-head attention layer can be used for polymeric radiation. Feature encoding In one embodiment, the attention weights and output can be determined using the following formula:

[0236] (36)

[0237] (37)

[0238] in It's the dimension of keys and queries. , and It is a learnable projection matrix. It is an attention weight vector, where each element corresponds to a point. , yes The Each element; the greater the attention weight, the more ray... right The more attention it receives, the higher the level of interest. This represents element-wise multiplication. For all elements corresponding to... rays Aggregation The feature map can then be obtained. .

[0239] In other cases, method 200 can be used to accelerate the attention module 116 by leveraging the functionality performed by the embedding module 104, the search module 106, and the value module 108.

[0240] In block 510, generator module 118 can use a convolutional generator to generate RGB predictions from the feature map. Generator module 118 uses a two-dimensional convolutional neural network (CNN) to process the two-dimensional feature map. Convert to RGB prediction For example, a U-Net CNN containing two downsampling layers and two upsampling layers can be used. In some cases, the BatchNorm layers typically included in U-Net can be removed because a batch size of 1 can be used during training. In other cases, any suitable neural network architecture can be used instead of U-Net. In other cases, any suitable function can be used instead of U-Net, including but not limited to lookup tables or database retrieval mechanisms such as nearest neighbor search or interpolators. In other cases, U-Net can be removed entirely. Furthermore, the generator module 118 can also generate depth predictions based on attention weights. To do this, for each ray, it can compute a weighted sum of distances from the camera center to different points, a weighted sum of projected distances from the camera center to different points projected along the camera's optical axis, or other distance concepts, where the weights are the attention weights for a given ray.

[0241] In another case, neural feature vectors It can have different dimensions, specifically designed for reconstructing different modalities, such as RGB, depth, intrinsic image, infrared (IR), etc. In this case, the generator module 118 will generate 2D feature maps. Different channel sets are converted into predictions of different modalities.

[0242] An RGB image is a rendering of a 3D scene representation; given the extrinsic and intrinsic parameters of a camera, an RGB image corresponding to a specific viewpoint of that camera in a 3D scene can be determined. The process of converting a 3D scene representation into an RGB image is called "rendering," and it is done by projecting rays from the camera in different directions, each ray corresponding to a pixel in the RGB image. For each ray, a neighbor attention operation is performed, which outputs attention weights for each point in the 3D representation. These attention weights are then used to aggregate feature vectors associated with these points. Since each ray produces a set of attention weights, each ray corresponds to an aggregated feature vector. The feature vectors from different rays can be arranged into a grid to generate a 2D feature map. Since each ray corresponds to a pixel in the image, the 2D feature map can have the same width and height as the RGB image to be rendered, thus the 2D feature map represents the RGB image.

[0243] In block 512, output module 110 outputs the RGB prediction results as a representation of the 3D scene.

[0244] Given a target RGB image Method 500 is trained by minimizing a loss function, which may include a term representing the target RGB image. Compared with RGB predicted image The loss function is the distance between the target image and the predicted image. This distance can be a variety of distance functions that reach their minimum when the target image and the predicted image are perfectly identical. Furthermore, the loss function can include other terms, such as regularization terms (e.g., regularization terms that encourage point cloud smoothing or local rigidity), or additional loss terms if other forms of supervision information are present (e.g., depth maps, intrinsic images, infrared (IR) images, etc.). In this case, the loss term can be the distance between the target image and the predicted image in each modality (i.e., depth map, intrinsic image, infrared (IR) image, etc.), or it can be a function of the target image and the distance between the predicted image in different modalities, such as shadows, i.e., the ratio of the original image to the intrinsic image in a linear color space.

[0245] In some cases, to learn denser and more accurate point clouds, point pruning and addition strategies can be used. In this case, it can be based on learnable foreground score vectors. Point pruning is performed, and this vector can be jointly optimized with point coordinates and neural network feature vectors during training. For example, when a foreground score vector or confidence vector exists, this vector can be learned as a separate variable along with point coordinates, point features, and neural network weights. After learning the foreground score vector, points with foreground scores below a threshold can be pruned.

[0246] In some cases, point coordinates or neural feature vectors can be updated during the intervals between regular updates in optimization algorithms. For example, point coordinates can be updated to better conform to geometric priors, such as local deformation stiffness relative to another 3D scene.

[0247] Figure 7 An example diagram of the neural network architecture used to execute at least a portion of method 500 is shown.

[0248] The inventors conducted example experiments to verify the advantages of method 500. The example experiments included comparing method 500 with other point-based neural rendering methods, such as Neural Point Graph (NPBG), Differentiable Point Radiation Field (DPBRF), Sculpted Neural Points (SNP), and Point-NeRF. The example experiments also included analyzing the potential broader impact of method 500 by comparing it with the implicit voxel representation method NeRF. For method 500, example experiments... A total of points were used in the scene representation. To ensure fair comparison, if the baseline method initially used fewer points, it was adjusted to the same number of points as method 500. Additionally, parameters for selecting nearest neighbor points were set. and the dimension of the point feature vector The performance of all methods was evaluated in both synthetic and real-world scenarios. For synthetic scenarios, the NeRF synthetic dataset was used; for real-world scenarios, the Tanks & Temples dataset was used. Example experiments used Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and the perceptual quality metric—Learned Perceptual Patch Similarity (LPIPS)—to evaluate the quality of the generated images.

[0249] Table 2

[0250]

[0251] Table 2 compares image quality metrics (PSNR, SSIM, and LPIPS) on the NeRF synthetic dataset and the Tanks & Temples subset. Higher PSNR and SSIM scores are better, while lower LPIPS scores indicate better perceptual quality. As shown in LPIPS, Method 500 outperforms all baseline methods in perceptual quality while also achieving competitive performance in PSNR and SSIM. Furthermore, compared to leading baseline methods, Method 500 achieves these results using significantly fewer points without relying on specific initialization. These findings highlight Method 500's ability to render high perceptual quality images with a small number of points.

[0252] The example experiments also qualitatively compared Method 500 with baseline methods on the NeRF synthetic dataset. Figure 8As shown, Method 500 generates images with richer details compared to the baseline methods. Notably, unlike the leading baseline method Point-NeRF, this method significantly reduces high-frequency noise. This is particularly evident in areas such as the shiny sphere in the last row and the microphone stand in the fourth row. These results validate the effectiveness of Method 500 in capturing fine details and generating realistic renderings compared to the baseline methods.

[0253] The example experiments also analyze the impact of the number of points in Method 500 on the quality of the rendered image. Evaluation was conducted using the LPIPS metric on a Lego scene test view of the NeRF synthetic dataset. Furthermore, the example experiments compared the performance of Method 500 with DPBRF and Point-NeRF. As shown in Figure 9, this method achieves better perceptual image rendering quality by using more points. Moreover, with the same number of points, Method 500 consistently outperforms the baseline methods significantly. It can also be observed that Method 500 exhibits stronger rendering quality robustness with a reduced number of points compared to the baseline methods. These findings highlight the effectiveness of Method 500 in representing scenes with high rendering quality using a small number of points.

[0254] The example experiment also analyzes ray-based point embedding by progressively removing point positions. and displacement vector This is achieved using equal components. Observation reveals that the removal point location... Subsequently, the learned point cloud noise increased, indicating the importance of introducing this component. Similarly, removing the displacement vector... This can lead to the model failing to learn the correct geometry. These findings validate the effectiveness of point embedding in Method 500 and highlight the crucial role of point location and displacement vectors in obtaining accurate, high-quality results. The analysis is as follows... Figure 10 As shown.

[0255] Although the present invention has been described with reference to certain specific embodiments, it will be apparent to those skilled in the art that various modifications may be made thereto without departing from the spirit and scope of the invention, as set forth in the appended claims.

Claims

1. A processor execution method for accelerating computations at various layers in a machine learning model, the method comprising: It receives row or column vectors of a weight matrix and an input vector, wherein each row or column vector of the weight matrix is ​​associated with a value vector; The row vectors or column vectors of the weight matrix and the input vector are transformed using an embedding function; For each transformed input vector, a search is performed to identify the row or column vectors in the weight matrix that have obtained high activation values ​​after transformation. The similarity between the row or column vectors of the weight matrix returned by the search and their associated input vectors is measured using a kernel function to determine the post-activation value; and The output activation value is used for running the machine learning model.

2. The method of claim 1, wherein determining an output matrix comprises: When the value vector is a column vector of the identity matrix, the output matrix is ​​made equivalent to the post-activation value; Otherwise, when the value vector is not a column vector of the identity matrix, the post-activation value is combined with its associated value vector to generate the output matrix; The output matrix is ​​then output for use by the machine learning model.

3. The method as described in claim 1, wherein the row vectors or column vectors of the weight matrix and the input vector are transformed using the embedding function, including embedding the row vectors or column vectors of the weight matrix and the input vector into a higher-dimensional space.

4. The method of claim 1, wherein the search comprises: Given each query after transformation, for each transformed input vector, use k-nearest neighbor search to identify the row or column of the corresponding weight matrix with higher importance.

5. The method of claim 1, wherein the search comprises one of the following: Returns all points or subsets located within a specific radius of the query vector, or returns points whose similarity to the query vector exceeds a threshold; Identify points that are close to or similar to the query vector; Cluster points that are close to or similar to each other, and return the points that are in the clusters closest to the query vector; Construct a graph with nodes as nodes, connecting nearby or similar nodes with edges, and then traverse the graph; and Estimate the probability density of points and return points whose probability density exceeds a threshold and are close to or similar to the query vector.

6. The method of claim 1, further comprising receiving a mask matrix, wherein the search uses only rows or columns of the unmasked weight matrix in the mask matrix.

7. The method of claim 1, wherein if the search fails to identify a row or column of the weight matrix, the method for determining the corresponding post-activation may include modifying the post-activation.

8. The method of claim 7, wherein for rows or columns of the weight matrix that fail to be identified in the search, the processing method is to treat those rows or columns dependent on the weight matrix and the kernel function of the query as a constant.

9. The method of claim 8, wherein the value of the constant can be fixed, or the value of the constant can be determined by randomly sampling a subset of rows or columns of the weight matrix. In the determination process, the values ​​of the corresponding kernel functions are first obtained through these sampled values ​​of the weight matrix and by querying, and then combined using a certain average method.

10. The method of claim 1, wherein the layer includes attention operations, the post-activation includes attention weights, and the rows or columns of the weight matrix include keys.

11. A system for accelerating layer operations used in machine learning models, the system comprising a processing unit and a data memory, the data memory containing executable instructions that, when executed by the processing unit, the system performs the following operations: It receives either the row vector or column vector of the weight matrix and the input vector, where, Each row vector or column vector of the weight matrix is ​​associated with a value vector; The row vectors or column vectors of the weight matrix and the input vector are transformed using an embedding function; For each transformed input vector, a search is performed to identify the row or column vectors in the weight matrix that have obtained high activation values ​​after transformation. The similarity between the row or column vectors of the weight matrix returned by the search and their associated input vectors is measured using a kernel function to determine the post-activation value; as well as The output activation value is used for running the machine learning model.

12. The system of claim 11, wherein the data memory further includes executable instructions for determining an output matrix, including: When the value vector is a column vector of the identity matrix, the output matrix is ​​made equivalent to the post-activation; Otherwise, when the value vector is not a column vector of the identity matrix, the value vector associated with the subsequent activation is combined to generate the output matrix; The output matrix is ​​then output for use by the machine learning model.

13. The system of claim 11, wherein the embedding function is used to transform the row vectors or column vectors of the weight matrix and the input vector, including embedding the row vectors or column vectors of the weight matrix and the input vector into a higher-dimensional space.

14. The system of claim 11, wherein the search comprises: Given each query after transformation, for each transformed input vector, use k-nearest neighbor search to identify the row or column of the corresponding weight matrix with higher importance.

15. The system of claim 11, wherein the search comprises one of the following: Returns all points or subsets located within a specific radius of the query vector, or returns points whose similarity to the query vector exceeds a threshold; Identify points that are close to or similar to the query vector; Cluster points that are close to or similar to each other, and return the points that are in the clusters closest to the query vector; Construct a graph with nodes as nodes, connecting nearby or similar nodes with edges, and then traverse the graph; and Estimate the probability density of points and return points whose probability density exceeds a threshold and are close to or similar to the query vector.

16. The system of claim 11, wherein determining the corresponding post-activation method for a search failing to identify a row or column of the weight matrix may include modifying the post-activation.

17. The system of claim 16, wherein for rows or columns of the weight matrix that fail to be identified in the search, the processing method is to treat those rows or columns dependent on the weight matrix and the kernel function of the query as a constant.

18. In the system of claim 17, the value of the constant can be fixed, or the value of the constant can be determined by randomly sampling a subset of rows or columns of the weight matrix. In the determination process, the values ​​of the corresponding kernel functions are first obtained through these sampled values ​​of the weight matrix and by querying, and then combined using a certain average method.

19. The system of claim 11, wherein the layer includes attention operations, the post-activation includes attention weights, and the rows or columns of the weight matrix include keys.

20. A processor execution method for learning a three-dimensional (3D) scene representation by utilizing neighbor attention for differentiable point rendering, the method comprising: Receive an input matrix originating from a set of images; Initialize points in the 3D scene; Ray-to-point features are extracted using the points and the input matrix; The ray-to-point features are aggregated to generate a feature map representing surface-ray intersections; Image predictions are generated using a convolutional neural network with the feature map as input. The image prediction is output as a representation of the 3D scene.

Images