Extensible and compressible neural network data storage system

Abstract

The invention relates to a scalable and compressible neural network data storage system. A system for compressed data storage using a neural network. The system includes a memory including a plurality of memory locations configured to store data; a query neural network configured to process a representation of an input data item to generate a query; an invariant key data store comprising key data for indexing a plurality of memory locations; an addressing system configured to process the key data and the query to generate a weighting associated with the plurality of memory locations; a memory read system configured to generate output memory data from the memory based on the generated weights associated with a plurality of memory locations and data stored at the plurality of memory locations; and a memory write system configured to write the received write data to the memory based on the generated weighting associated with the plurality of memory locations.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980063938.9, with an application date of September 27, 2019. Technical Field

[0003] The present disclosure generally relates to scalable and compressible neural network data storage systems. Background Art

[0004] Neural networks are machine learning models that use one or more layers of nonlinear units to predict outputs from received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network, the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters.

[0005] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the network's internal state from previous time steps when computing the output at the current time step. An example of a recurrent neural network is a long short-term (LSTM) neural network, which includes one or more LSTM memory blocks. Each LSTM memory block may include one or more cells, each cell including an input gate, a forget gate, and an output gate, allowing the cell to store the cell's previous state, for example, to be used to generate the current activation or to be provided to other components of the LSTM neural network. Summary of the invention

[0006] This specification describes a system implemented as a computer program on one or more computers in one or more locations that provides a system for storing compressed data using a neural network. The system is capable of encoding data and storing the data in a sparse and distributed manner in memory to achieve a learning data structure suitable for one or more tasks. The learning data structure is highly compressed and greatly reduces memory requirements.

[0007] According to a first aspect, there is provided a system for storing compressed data using a neural network, the system comprising: a memory comprising a plurality of memory locations configured to store data; a query neural network configured to process representations of input data items to generate queries; an invariant key data store comprising key data for indexing the plurality of memory locations; an addressing system configured to process the key data and the queries to generate weights associated with the plurality of memory locations; a memory read system configured to generate output memory data from the memory based on the generated weights associated with the plurality of memory locations and the data stored at the plurality of memory locations; and a memory write system configured to write received write data to the memory based on the generated weights associated with the plurality of memory locations.

[0008] The key data storage is immutable in the sense that the key data is fixed once created and does not change over time or in use. Additionally, while other components of the neural network system may be learned through training, the key data is not learned.

[0009] The key data store may include a plurality of randomly generated keys for indexing a plurality of memory locations, wherein the plurality of keys are immutable after generation.

[0010] Alternatively, the key data store may include invariant seed data for generating multiple invariant keys for indexing multiple memory locations; and the addressing system may also be configured to generate multiple keys based on the seed data. By storing seed data and generating multiple keys on demand rather than storing a complete set of keys, memory requirements may be reduced.

[0011] It will be appreciated that using the same seed data will always result in the same key being generated. It will also be appreciated that in addition to the possibility of a single key, a single seed can also be used to generate multiple keys. For example, a seed can be used to generate a key with an additional offset value added to generate additional keys. In another example, additional keys can be generated by running the generation algorithm for multiple time steps, each of which produces a key.

[0012] The plurality of generated keys may be based on samples of Gaussian random variables. In the case of storing seed data, the seed data includes one or more seeds for generating samples of Gaussian random variables. By using randomly generated keys, data may be stored at memory locations distributed throughout the memory, thereby improving memory utilization.

[0013] The key data store may be implemented as a matrix. The keys may alternatively be referred to as addresses, and the key data store may be referred to as an addressing matrix.

[0014] The addressing system may also be configured to process the key data and the query to generate weights associated with the plurality of memory locations based on similarities between the query and each of the plurality of keys. The similarity may be based on any similarity metric such as cosine distance or Euclidean distance.

[0015] The system may also include an index of multiple keys based on a k-nearest neighbor model. The similarity between the query and each of the multiple keys may also be based on the k-nearest neighbor model. For example, the k-nearest neighbor model may be based on locality sensitive hashing. In this way, the keys most similar to the query may be quickly determined. Considering that the key data store is immutable, the k-nearest neighbor model only needs to be indexed once, compared to prior art methods in which the key data store is not immutable and the key index needs to be regenerated whenever the key data store is modified.

[0016] The query neural network may be a feed-forward neural network.

[0017] The addressing system can also be configured to decorrelate the queries. Processing key data and queries to generate weights associated with multiple memory locations can be based on decorrelated queries. In this way, training of the system to learn appropriate query representations can be made more efficient through faster convergence. Using decorrelated queries encourages the system to maximize the utilization of all available memory locations for storing data, thereby avoiding the selection of only a few specific memory locations for data storage. In addition, decorrelated queries can also produce better optimization during training, thereby producing a data encoding scheme that can be more compressed and thus improve memory usage. Decorrelation can be based on a sphering transform, such as a moving average ZCA transform. The sphering transform can also be called a whitening transform.

[0018] The weighting may be based on a sparse softmax function. That is, the function may be used to select the k most similar keys, and therefore, the corresponding memory locations indexed by those keys. By selecting the k most similar keys and corresponding memory locations, information may be shared between multiple memory locations to increase the information content of the data stored in the memory while also maintaining efficient memory utilization. The value of k may be appropriately selected by one skilled in the art.

[0019] The system may also include an encoder neural network configured to process the input data item to generate a representation of the input data item. In this manner, the representation may provide more informative features for processing than the original input data item. The encoder neural network may be a feedforward neural network or other suitable type of neural network for processing a particular type of input data item.

[0020] The memory reading system may also be configured to generate output memory data based on applying a corresponding weight associated with the memory location to the data stored at the memory location. The application of the weight may be an element-wise multiplication. Thus, the memory reading system may be a content-based soft attention system.

[0021] The memory reading system may also be configured to generate output memory data based on data stored at the memory location having an associated weight greater than a threshold and exclude data stored at other memory locations in the generation of the output memory data. The threshold may be selected so that the k memory locations with the highest weights are selected. The value of k may be appropriately selected by a person skilled in the art.

[0022] The system may also include a write word neural network configured to process a representation of an input data item to generate received write data. In this way, data to be written to the memory may also be learned. The write word neural network may be a feedforward neural network.

[0023] The memory write system may also be configured to generate weighted write data for each memory location based on the corresponding weight associated with the memory location and the received write data; and write the weighted write data additively to each memory location. By writing additively, past data stored in the memory may be maintained. In addition, back propagation through time on sequential writes may also be avoided. Temporal correlations may also be stored in the memory.

[0024] The system may also include an output representation neural network configured to process the output memory data of the memory reading system, the received write data, and the representation of the input data item to generate an output representation of the input data item. This enables the capture of nonlinear interactions between the data stored in the memory. In addition, the input data item can be modeled in part by the data stored in the memory. Because the data stored in the memory can be quickly adapted compared to the parameters of the neural network that change slowly over time, the use of an external memory as described above enables the system to be generalized in the case where a limited number of examples of a set of data items have been presented or in the case where the expected environment or task changes over time. Data items belonging to the same class or set can be mapped to the same memory location, thereby enabling the generalization of data items that have not been seen. The output representation neural network can be a feedforward neural network.

[0025] The system can be trained based on a meta-learning framework. For example, the system can be trained based on a series of tasks with a limited number of training examples in each task. A series of tasks can be sampled from a common distribution. The training data set can include data to be stored in memory and a set of in-distribution queries and a set of out-of-distribution queries, which represent data and queries that the system will likely encounter during use. It should be appreciated that the neural network system is fully differentiable and can be trained end-to-end. The system can be trained based on a cross entropy loss. The training objectives can include binary classification. The system can be trained using back propagation jointly through query operations and memory insertion operations.

[0026] The data stored in the memory may include data indicating one or more data sets. The output memory data of the memory reading system may indicate whether the input data item belongs to a set in one or more data sets. For example, the system may implement a data structure similar to a Bloom filter, which enables the system to quickly determine whether the input data item belongs to a specific data set. The neural network system can be trained to act as a Bloom filter and as a substitute for the Bloom filter in the existing system. However, the system has the ability to learn more compressed and more space-efficient data structures than the Bloom filter for tasks such as approximate set membership. The system is suitable for both static set membership tasks and dynamic set membership tasks. In some prior art neural network systems, the set data is encoded in the weights of the neural network system, and therefore, considering the slow adaptation rate of the weights of the neural network, it is not suitable for dynamic set membership tasks. In the present system, the system memory that can be quickly adapted is used to encode the set data, and is therefore suitable for dynamic set membership tasks.

[0027] The system may also include a backup Bloom filter for handling false negative results.

[0028] According to another aspect, a system for storing compressed data using a neural network is provided, the system comprising: a memory comprising a plurality of memory locations configured to store data; a query neural network configured to process representations of input data items to generate queries; an invariant key data store comprising key data for indexing the plurality of memory locations; an addressing system configured to process the key data and the query to generate weights associated with the plurality of memory locations; and a memory access system configured to access memory data from the memory based on the generated weights associated with the plurality of memory locations.

[0029] According to another aspect, a method for reading data from a system for storing compressed data using a neural network is provided, the method comprising: receiving a representation of an input data item; processing the representation of the input data item by a query neural network to generate a query; retrieving key data for indexing a plurality of memory locations of a memory from an invariant key data store; processing the query and the key data by an addressing system to generate weights associated with the plurality of memory locations; and outputting memory data from a memory read system based on the generated weights associated with the plurality of memory locations and data stored at the plurality of memory locations.

[0030] According to another aspect, a method for using a neural network to write data to a system for storing compressed data is provided, the method comprising: receiving a representation of an input data item; processing the representation of the input data item by a query neural network to generate a query; retrieving key data for indexing multiple memory locations of a memory from an invariant key data store; processing the query and the key data by an addressing system to generate weights associated with the multiple memory locations; receiving write data; and writing the received write data to the memory by a memory write system based on the generated weights associated with the multiple memory locations.

[0031] According to another aspect, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium comprising computer-readable instructions, which when executed, cause one or more computers to implement a system according to any of the above aspects or perform a method according to any of the above aspects.

[0032] The system can be configured to receive any kind of digital data input. For example, the stored data and / or input data items may include images (including video images), images or features that have been extracted from images; characters, words or strings, text sequences or segments in one or more languages, such as text in one language and translations of text into another language; Internet resources (e.g., web pages), documents or portions of documents or features extracted from Internet resources, documents or portions of documents, content items; the flash context of a particular advertisement; features of personalized recommendations for users and / or features characterizing the recommendation context, such as features characterizing previous actions taken by the user; data sequences representing spoken speech and / or transcriptions of speech; scores of any of the foregoing; or any other kind of data items that may be stored in a database. In general, the system's memory can be used to store any type of data, including but not limited to the foregoing.

[0033] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages: The system can learn data structures that are suitable for one or more specific tasks. The system is particularly advantageous in situations where data items can only be examined a limited number of times, or even only once, such as in a data stream, given that the system provides rapid adaptation to new information. The system is also particularly advantageous in settings where tasks or environments change over time. The system can also generalize quickly to unseen data.

[0034] The system is able to learn how to effectively index and query data structures based on input data, and how to effectively encode data and store it in memory. That is, the system is able to use encoding to store information based on a specific memory location where data is stored or specific data stored at a memory location or a combination of where the data is stored and what data is stored in the memory. The system provides greater data compression than a classical data agnostic data structure by learning to exploit the structure of the input data. Therefore, compared with a classical data structure, the system may require fewer bits per element to store data. The system also provides compression across memory locations, thereby achieving greater compression than other prior art memory-enhanced neural networks. The system therefore provides a scalable distributed compression write solution.

[0035] Additionally, the system has low latency due to the system's architecture and efficient operations used to perform queries, addressing, and memory accesses.

[0036] Furthermore, by providing a query mechanism and invariant key data storage as described above, the size of the system's memory can be made independent of the number of trainable parameters of the neural network component, and the system can therefore be more scalable than other prior art memory augmented neural networks. Due to the independence of the size of the memory and the number of trainable parameters, the size of the memory can be modified during use, even after the neural network component has been trained.

[0037] The system may implement a data structure suitable for determining approximate set membership similar to a Bloom filter, or other types of data structures as appropriate.

[0038] For example, the system can be used to quickly determine whether a record exists in a large database to avoid unnecessary disk lookups, to determine whether a data item exists in a cache, to block malicious IP addresses in network security, to serve as a type of classifier for classifying image, video, audio, and text data, and to be used by agents to determine whether a particular environment corresponds to one that has been previously encountered and how to act.

[0039] It will be appreciated that the various aspects may be combined and that features described in the context of one aspect may be combined with other aspects of the invention.

[0040] It will also be appreciated that system components such as memory, addressing systems, memory reading systems, and memory writing systems can be adapted from existing memory and systems for addressing, reading to, and writing from memory included in existing computers. Additionally or alternatively, components of the system can be configured to be compatible with existing memory and memory access systems and other existing computing components.

[0041] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 An example system for compressed data storage using a neural network is shown.

[0043] Figure 2 Another example system for compressed data storage using a neural network is shown.

[0044] Figure 3 An exemplary process for reading data from a neural network memory system is shown.

[0045] Figure 4 An exemplary process for writing data to a neural network memory system is shown. DETAILED DESCRIPTION

[0046] Figure 1 An example system 100 for compressed data storage using a neural network is shown. System 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below can be implemented.

[0047] System 100 includes memory 101. Memory 101 includes a plurality of memory locations 101a…n configured to store data. Memory 101 may store any type of data as appropriate for the particular task being performed. For example, system 100 may implement a data structure equivalent to a Bloom filter for performing an approximate set membership task. Thus, memory 101 may store data indicating one or more data sets. In other examples, the stored data may be image data, an encoding of image data, or data indicating attributes associated with an image. Other modalities may also be stored, such as video or audio data or text data, for example, the data may be related to data in a database.

[0048] The size of the memory 101 may be appropriately selected by a person skilled in the art and based on the task being performed. As an example, for an approximate set membership task, the memory may have 2 to 128 memory locations (inclusive), wherein each memory location stores 2 to 10 elements (inclusive).

[0049] As described in further detail below, system 100 can learn to store data in memory 101 in a compressed and distributed manner, such that memory requirements are greatly reduced compared to conventional methods of storing data.

[0050] The system 100 also includes an invariant key data store 102 that includes key data for indexing a plurality of memory locations 101a ... n. The key data store 102 is invariant in the sense that once the key data has been generated it cannot be modified. The key data can be mapped to a specific memory location and can therefore be considered to provide an addressing mechanism for reading from and writing to the memory 101. Therefore, the key data can also be considered to be address data. Further details about the key data are described below.

[0051] The system 100 also includes an addressing system 103 that is configured to process the key data and the query 104 to generate weights 105 associated with the plurality of memory locations 101a...n. The weights 105 may be used in determining which of the plurality of memory locations 101a...n to access based on a given query 104. In this manner, the weights 105 may also be considered a form of address for accessing the plurality of memory locations 101a...n. Further details regarding the generation of the weights 105 are described below.

[0052] The system 100 also includes a query neural network 106 that is configured to process a representation 107 of an input data item to generate a query 104. The input data item can be any type of data. For example, image data, audio data, sensor data, web data, and text. The representation 107 of the input data item can be an encoding or embedding of the input data item. For example, the representation can be a feature vector that characterizes the input data item or can be a representation or encoding generated by a neural network.

[0053] The system 100 also includes a memory access system configured to access memory data from the memory 101. The access may be based on the generated weights 105 associated with the plurality of memory locations 101a...n. Thus, the system 100 may include a memory read system 108 configured to generate output memory data 109 based on the generated weights 105 associated with the plurality of memory locations 101a...n and the data stored at the plurality of memory locations 101a...n. The system 100 may also include a memory write system 110 configured to write received write data 111 to the memory 101 based on the generated weights 105 associated with the plurality of memory locations 101a...n. As noted above, the weights 105 may act as a form of address for selecting which of the plurality of memory locations 101a...n is to be accessed. The memory access system is described in more detail below and may include one or more neural networks.

[0054] The system 100 with trainable neural network parameters is not only able to learn how to best encode and store data in memory 101, but also how to best address and query memory 101 for efficient access. Since the key data is constant and not learned, the size of memory 101 is independent of the number of trainable parameters and thus improves the scalability of the system. The size of the memory can also be adjusted dynamically without retraining the system.

[0055] Furthermore, the use of memory enables rapid adaptation to new data, compared to the parameters of a neural network which can only change slowly over time. The system is thus able to generalize where only a limited number of examples of a set of data items have been presented, or where the environment or task is expected to change over time. Data items belonging to the same class, distribution, or set can be mapped to the same memory location, thereby enabling generalization to unseen data items.

[0056] Reference now Figure 2 , another example system 200 for compressed data storage using a neural network is shown. Figure 2 The system 200 includes Figure 1 Components of the system 100, whereby similar components have similar reference numerals in the various figures. In this regard, Figure 2 Additional labels are also included, where memory 101 is labeled "M"; invariant key data storage 102 is labeled "A", query 104 is labeled "q", weight 105 is labeled "a", representation of input data item 107 is labeled "z", output memory data 109 is labeled "r", and write data 111 is labeled "w".

[0057] The system 200 also includes an encoder neural network 212 that is configured to process an input data item 213 to generate a representation 107 of the input data item. The representation 107 may provide more informative features than the original input data item 213 for processing and performing a desired task, and such a representation may be learned by training parameters of the encoder neural network 212. Training of the neural network parameters is described in more detail below.

[0058] The encoder neural network 212 can be any type of neural network as deemed appropriate by one skilled in the art. For example, if the input data item is an image, the encoder neural network can be a convolutional neural network or a feed-forward multi-layer perceptron (MLP). If the input data item is a sequence such as a text sequence, a video, an audio sequence, the encoder neural network 212 can be a recurrent neural network such as an LSTM. In one example, the encoder neural network 212 is a three-layer convolutional neural network configured to process image inputs. In another example, the encoder neural network 212 is an LSTM with 128 hidden units configured to process text inputs such as database row keys.

[0059] The system 200 also includes a write word neural network 214 configured to process the representation 107 of the input data item to generate the received write data 111. In this way, the data to be written to the memory 101 can also be learned. Thus, in addition to learning where to write to the memory, the system can also learn what to write to the memory to further provide distributed and compressed encoding of the data for storage.

[0060] The written word neural network 214 can be any type of neural network such as a feed-forward MLP. In one example, the written word neural network 214 is an MLP with a single hidden layer of 128 units with a leaky ReLU nonlinearity.

[0061] The system 200 also includes an output representation neural network 215, which is configured to process the output memory data 109 read from the memory 101, the received write data 111, and the representation 107 of the input data item to generate an output representation 216 of the input data item. The output representation 216 of the input data item 213 can depend on the task being performed by the system 200. For example, if the task is to approximate set membership, the output can be the probability that the input data item belongs to the set of input data items that have been previously seen by the system 200. This can be used in applications such as caching, malicious URL detection, database query, spell checking, and other applications. In another example, the task can be a classification task and the output can be the probability that the input data item, such as an image, belongs to or includes an object of a specific class. In another example, the output representation 216 can be a representation of the input data item as a better encoding of the input data item for use in a separate system such as a classifier or a robot agent. Therefore, the input data item 213 can be modeled in part by the data stored in the memory 101.

[0062] The output representation neural network 215 can be any type of neural network. In one example, the output representation neural network 215 is a three-layer MLP with residual connections. By using the output representation neural network, nonlinear interactions in the data stored in the memory can be captured.

[0063] exist Figure 2 1, no specific distinction is made between the memory reading system 108 and the memory writing system 110. It should be appreciated that the components involved in reading from the memory 101 may form part of the memory reading system 108, and the components involved in writing to the memory 101 may form part of the memory writing system 110. It should also be appreciated that the components may belong to both the memory reading system 108 and the memory writing system 110.

[0064] Further details regarding the key data will now be described. The key data may include a plurality of randomly generated keys for indexing the plurality of memory locations 101a...n. For example, the plurality of keys may be based on samples of a Gaussian random variable. Alternatively, a different random number generation process may be used as appropriate. By using randomly generated keys, data may be stored at memory locations distributed throughout the memory, thereby improving memory utilization.

[0065] As discussed above, after generation, the multiple keys are constant, that is, the multiple keys cannot be modified after creation. In the case of randomly generating multiple keys, the constant key data storage 102 can store seed data for generating multiple keys, rather than the multiple keys themselves. Therefore, the addressing system 103 can be configured to regenerate multiple keys using constant seed data. By storing constant seed data instead of multiple keys, the storage / memory requirements of the constant key data storage 102 can be reduced.

[0066] It will be appreciated that using the same seed data input to the same random generation process will result in the same key being generated. It will also be appreciated that a single seed may be used to generate multiple keys or a single key. For example, a seed may be used to generate a key with an additional offset value added to generate additional keys. In another example, additional keys may be generated by running the generation algorithm for multiple time steps, each of which generates a key.

[0067] The invariant key data store 102 may be implemented as a matrix. As discussed above, the key data may be considered to provide an addressing mechanism for accessing the memory 101. Thus, the key data store 102 may also be referred to as an addressing matrix.

[0068] In order to speed up the similarity comparison between the query 104 and the key data, the system 100, 200 may also include an index of multiple keys. The index may be based on a k-nearest neighbor model or an approximate k-nearest neighbor model. The k-nearest neighbor model may be based on locality sensitive hashing or another similarity preserving hash function. The key most similar to the query may be found by searching the k-nearest neighbor model. Considering that the key data storage 102 is immutable, the k-nearest neighbor model only needs to be indexed once, compared to the prior art method that can index the contents of the memory itself and therefore will need to regenerate the index whenever the memory is modified.

[0069] Reference now Figure 3 , the process for reading data from the neural network memory system will now be described. It should be appreciated that the Figure 1 and Figure 2 The processing is performed by the system 100, 200.

[0070] At step S301, a representation 107 of an input data item is received. The representation 107 of the input data item may be generated using an encoding function as shown below:

[0071] z←f enc (x) (1)

[0072] Where z is the representation of the input data item 107, f enc(.) is the encoding function and x is the input data item. As discussed above, the encoding function may be implemented using the encoder neural network 212.

[0073] At step S302, representation 107 of the input data item is processed by query neural network 106 to generate query 104. Query neural network 106 may implement a function that maps representation 107 to query 104:

[0074] q←f q (z) (2)

[0075] where q is query 104, f q (.) is the mapping function and z is the representation 107 of the input data item.

[0076] At step S303, key data for indexing the plurality of memory locations 101a ... n of the memory 101 is retrieved from the invariant key data store 102. The retrieval may be accessing the invariant key data store 102 to read the key data stored therein. As discussed above, the invariant key data store 102 may store seed data for generating the plurality of keys rather than the plurality of keys themselves. In this case, the retrieval of the key data may also include regenerating the plurality of keys using the invariant seed data. The addressing system 103 may be configured to perform such regeneration of the plurality of keys.

[0077] At step S304, the query 104 and the key data are processed by the addressing system 103 to generate weights 105 associated with the plurality of memory locations 101a...n. The weights 105 may be based on similarities between the query 104 and each of the plurality of keys. For example, the similarity may be based on cosine distance and implemented as:

[0078] d←q T A (3)

[0079] Where A is a row matrix containing multiple keys, q T is a query 104 in the form of a column vector and d is a vector of similarity scores for each of the plurality of keys. Alternatively, similarity may be based on other types of distances, such as Euclidean distance as deemed appropriate.

[0080] The weighting 105 can also be based on sparse softmax. For example, a sparse softmax function can be applied to the similarity score d to generate a probability distribution whereby only a few elements have non-zero values. This can be achieved by ignoring values ​​below a threshold or selecting only the top k scoring elements to have non-zero values. Thus, the weighting 105 can be calculated as:

[0081] a←σ k (q T A) (4)

[0082] where σ k (.) sparse softmax function and q T AAs described above.

[0083] At step S305, output memory data 109 from the memory 101 is output by the memory reading system 108 based on the generated weights 105 associated with the plurality of memory locations 101a...n and the data stored at the plurality of memory locations. This may include applying the corresponding weights 105 associated with the memory locations to the data stored at the memory locations. For example, the output memory data 109 may be generated as follows:

[0084] r←M⊙a (5)

[0085] Where r is the output memory data 109, M is a matrix comprising the data stored in the memory, a is the weight 105 and ⊙ is an element-wise multiplication.

[0086] If the weight 105 is a sparse weight, the weight 105 has the effect of selecting only a few specific memory locations to access based on the query 104. Alternatively, a threshold may be applied to the weight 105 so that only memory locations with associated weights greater than the threshold are selected or the top k memory locations with the highest corresponding weights may be selected. By selecting only a few specific memory locations, the system is able to learn to store data in the memory 101 in a distributed manner, and information may be shared between multiple memory locations to increase the information content of the data stored in the memory 101, while also maintaining efficient memory utilization.

[0087] If the weight 105 is based on the similarity between the key data and the query 104, the weight 105 can be considered as an attention vector for focusing on a specific location in the memory 101, and the addressing mechanism can be considered as a content-based soft attention mechanism. The addressing mechanism can also be considered as a differentiable hashing scheme.

[0088] The method may also include generating an output representation 216 of the input data item 213. Generating may be based on processing the output memory data 109 read from the memory 101 at step S305, the received write data 111, and the representation 107 of the input data item received at step S301 using the output representation neural network 215. For example, the output representation 216 may be generated as follows:

[0089] o←f out ([r,w,z]) (6)

[0090] Where o is the output representation 216, f out(.) is the function implemented by the output representation neural network 215, r is the output memory data 109, w is the received write data 111 and z is the representation 107 of the input data item.

[0091] As discussed above, the output representation 216 of the input data item 213 may depend on the task being performed and may be, for example, a probability score, or a final output encoding of the input data item 213, or other data characterizing the input data item for use in another system.

[0092] The query 104 generated at step S302 may optionally undergo further processing before being used at step S304. For example, the query 104 may be a vector and may undergo a decorrelation operation to decorrelate the elements of the vector. The decorrelation operation may be any suitable type of decorrelation, such as a spherification or whitening transformation. For example, the decorrelation may be based on a moving ZCA average as shown below:

[0093]

[0094] Σ t+1 ←γΣ t +(1-γ)s T s second moment (8)

[0095] U,s,_←svd(Σ-μ 2 ) Calculate singular values ​​(9)

[0096]

[0097] θ sca ←ηθ sca +(1-η)W ZCA moving average (11)

[0098] q←qθ sca Decorrelate queries (12)

[0099] In another example, the decorrelation operation can be based on PCA. The addressing system 103 can be configured to perform decorrelation. Decorrelation is particularly beneficial in cases where sparse and distributed addressing systems are desired. Using decorrelation queries encourages the system to maximize the utilization of all available memory locations in the memory 101, thereby avoiding the selection of only a few specific memory locations for data storage. In addition, decorrelation queries can also produce faster convergence and better optimization during training, thereby producing a learning data encoding scheme that can be more compressed and therefore improves on memory usage.

[0100] Reference now Figure 4 , the process for writing data to the neural network memory system will now be described. It should be appreciated that the Figure 1and Figure 2 The processing is performed by the system 100, 200.

[0101] Figure 4 Steps S401 to S404 are Figure 3 1 to S304 of the memory 101, and therefore, the above operations described with respect to steps S301 to S304 also apply to steps S401 to S404. Thus, in brief, at step S401, a representation 107 of an input data item is received; at step S402, the representation 107 of the input data item is processed by the query neural network 106 to generate a query 104; at step S403, key data for indexing a plurality of memory locations 101a...n of the memory 101 is retrieved from the invariant key data store 102; and at step S404, the query 104 and the key data are processed by the addressing system 103 to generate weights 105 associated with the plurality of memory locations 101a...n.

[0102] At step S405, write data 111 is received. The write data 111 may be generated using the write word neural network 214 based on the representation 107 of the input data. For example, the write data 111 may be generated as follows:

[0103] w←f w (z) (13)

[0104] Where w is the written data 111, f w (.) is a function implemented by the write word neural network 214 and z is the representation of the input data item 107. Alternatively, the write data 111 may be received externally or may simply be the representation of the input data item 107 or the input data item 213 itself without modification.

[0105] At step S406, the received write data 111 is written to the memory 101 by the memory write system 110 based on the generated weights 105 associated with the plurality of memory locations 101a...n. This may be performed by generating weighted write data for each memory location based on the corresponding weights 105 associated with the memory location and the received write data 111, and additively writing the weighted write data to each memory location. For example, the writing may be performed as follows:

[0106] M t+1 ←m t +wa T (14)

[0107] Among them, M t+1 is the memory 101 after writing, M tis the memory 101 before writing, w is the received write data 111 and a T is weight 105. Writing to memory 101 can therefore be considered an additive write, as the new data is added to the existing data in memory 101. This enables past information to be maintained in memory. Note that writing does not require multiplicative gating or squashing as in some prior art neural network memory systems.

[0108] You can refer to the above Figure 3 The weights 105 are generated as described. The weights 105 can be used to select which specific memory locations of the memory 101 are to be written to and can therefore be considered as addresses. The weights 105 can specify that all of the multiple memory locations 101a...n are to be written to. Alternatively, the weights 105 can be sparse and specify a small subset of the memory locations to be written to. In this way, data can be stored across memory locations in a compressed and distributed manner. In addition, in the case of randomly generated key data, this combined with the sparse weights also ensures that the stored data is distributed across the memory to maximize memory utilization, rather than only writing data to a few frequently used memory locations. This is in contrast to some prior art neural network memory systems that are only able to write to one memory location in one time step or, in the case of being able to write to multiple memory locations, only very specific addresses are selected in practice.

[0109] It should be appreciated that while the above process is presented as being performed in a particular order, it is not intended to be limited to any particular ordering of the steps, and the above steps may be performed in a different order.

[0110] Can be trained based on a meta-learning framework Figure 1 and Figure 2 The neural network of the system 100, 200. In the meta-learning framework, training can initially be based on sampled tasks that are related to the tasks that the system will perform before undergoing final training to adapt the system to the desired task. This is beneficial in situations where there is limited training data available for the desired task, but other training data may be available. Therefore, the system 100, 200 can be trained with limited training data, such as in a few-shot or one-shot learning setting. For example, it is possible to provide the data as a temporary data stream, whereby each data item is seen only once and must be learned based on observing this single instance.

[0111] The system 100, 200 can implement a data structure similar to a Bloom filter for the task of approximating set membership. Such a system can be trained using a training data set that includes a distribution of different data sets, each of which corresponds to a different set membership task. At each iteration of training, one of the data sets is selected as described above and the data including the set is input to the system 100, 200 for storage. Sample data queries are generated and provided as input to the system 100, 200. For each query, the system provides an output to indicate whether the query data is part of the data set input to the system 100, 200.

[0112] The system can then be trained based on a cross entropy loss function. For example, the loss function can take the following form:

[0113]

[0114] Where L is the loss, j is the index of the query data, t is the number of queries, and y is the loss if the query data is in the set. j = 1 or if the query data is not in the set, y j =0,o j is the output of the system 100, 200 and can be calculated as shown in equation (6).

[0115] The parameters of the neural network of the system 100, 200 can then be updated using stochastic gradient descent by computing the gradient of the loss with respect to the parameters of the neural network and back-propagating through the queries and writes performed. It should be appreciated that the above-described neural network system is fully differentiable and end-to-end trainable.

[0116] Further iterations of training may be performed until a suitable stopping criterion is reached, such as when a maximum number of training iterations is reached. The trained system may then be further trained on real task data. It is possible that the system may have only one opportunity to observe a particular data item and learn that it is part of a data set. This is in contrast to prior art methods that may have unlimited access to real task data and learn to model a data set by conventional means of traversing the entire task data many times until satisfactory performance is obtained before putting it to use.

[0117] As discussed above, key data is not learned. The system may be initialized with key data before training begins and remain fixed throughout the training process and in use.

[0118] In the case where the system uses additive writes, backpropagation through time over sequential writes can be avoided. The addition operator is commutative and therefore invariant to the ordering of writes to memory, and the computation of gradients can be performed in parallel. Thus, the system can be trained more efficiently and is scalable, whereby prior art neural network systems with memory may not be able to train or train poorly if backpropagation through time over very long sequences is required.

[0119] Systems 100, 200 can be used as a replacement for Bloom filters. However, Bloom filters guarantee that no false negatives are generated. That is, given a query data item, if the query data item is not in the set, the Bloom filter is guaranteed not to erroneously output that the query data item is in the set. Systems 100, 200 cannot guarantee that no false negatives are generated, and therefore, a backup Bloom filter can be used to store false negatives issued during training. Table 1 shows a comparison of the space requirements for a database task of storing 5000 row key strings for a target false positive rate in an exemplary embodiment of the system (labeled "neural Bloom filter") and a conventional Bloom filter and a cuckoo filter.

[0120]

[0121] Table 1. Database tasks. Store 5000 row key strings for target false positive rate.

[0122] Table 1 shows that even when a backup Bloom filter is needed, the combined space requirements of the system's memory 101 and the backup Bloom filter are significantly lower than the space requirements of a conventional Bloom filter or cuckoo filter for storing the entire data. The system thus has the ability to learn more compressed and more space efficient data structures than conventional data structures. The system is able to learn to exploit structure in the input data distribution to provide encodings of data that are significantly more compact than conventional data structures.

[0123] The systems 100, 200 can be used for both static and dynamic set membership tasks. In some prior art neural network systems, set membership is encoded in the weights of one or more neural networks, and therefore is not suitable for dynamic set membership tasks given the slow adaptation rate of the neural weights. The memory 101 of the systems 100, 200 provides the system with the ability to quickly adapt to new data, and is therefore suitable for dynamic set membership tasks.

[0124] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0125] The embodiments of the subject matter and functional operations described in this specification may be implemented with digital electronic circuits, with tangibly implemented computer software or firmware, with computer hardware including the structures disclosed in this specification and their structural equivalents, or with a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not a propagation signal, however.

[0126] The term "data processing apparatus" encompasses all kinds of apparatus, devices and machines for processing data, including by way of example a programmable processor, a computer or a plurality of processors or computers. The apparatus may include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus may include, in addition to hardware, code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0127] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files storing portions of one or more modules, subroutines or codes). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0128] As used in this specification, "engine" or "software engine" refers to a software-implemented input / output system that provides an output that is different from the input. An engine can be a functional coding block, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine can be implemented on a computing device of any suitable type including one or more processors and a computer-readable medium, such as a server, a smart phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smart phone, or other fixed or portable device. Additionally, two or more of these engines can be implemented on the same computing device or on different computing devices.

[0129] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows may also be performed by special purpose logic circuits, and the apparatus may also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). For example, the processes and logic flows may be performed by a graphics processing unit (GPU) and the apparatus may also be implemented as a graphics processing unit (GPU).

[0130] Computers suitable for executing computer programs include, as examples, a central processing unit that can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include or be operationally coupled to receive data from one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or transfer data to the one or more large-capacity storage devices, or both. However, a computer does not have to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0131] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0132] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by a user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0133] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser that a user can use to interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0134] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0135] Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may be removed from the combination in some cases, and a claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0136] Similarly, although operations are depicted in the accompanying drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired effect. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0137] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

Claims

1. A method performed by one or more computers, the method comprising: receiving a representation of an input data item; processing said representation of said input data item by a query neural network to generate a query; generating a respective weight for each of a plurality of memory locations in a memory from (i) the query and (ii) a respective key for each of a plurality of memory locations in the memory, wherein the memory stores data representing a set of data items; generating output memory data from the memory based on respective weights of the plurality of memory locations and data stored at the plurality of memory locations; as well as An output is generated from at least the output memory data indicating whether the input data item belongs to the set of data items.

2. The method according to claim 1, further comprising: The input data item is processed using an encoder neural network to generate the representation of the input data item.

3. The method according to claim 1, wherein: Generating output memory data from the memory based on respective weights of the plurality of memory locations and data stored at the plurality of memory locations comprises: For each memory location, a product is calculated between the data stored in the memory location and the corresponding weight of the memory location.

4. The method according to claim 1, wherein: Generating an output from at least the output memory data indicating whether the input data item belongs to the set of data items comprises: At least the output memory data is processed by an output representation neural network to generate the output indicating whether the input data item belongs to the set of data items.

5. The method according to claim 4, further comprising: processing said representation of said input data item using a written word neural network to generate written data; And among them, Processing at least the output memory data by an output representation neural network to generate the output indicating whether the input data item belongs to the set of data items includes: processing at least the write data and the output memory data by the output representation neural network.

6. The method according to claim 5, wherein: Processing at least the output memory data by an output representation neural network to generate the output indicating whether the input data item belongs to the set of data items includes: processing the write data, the representation of the input data item and the output memory data by the output representation neural network.

7. The method according to claim 1, wherein: Generating a respective weight for each of a plurality of memory locations in the memory from (i) the query and (ii) a respective key for each of a plurality of memory locations in the memory comprises: generating a decorrelated query from the query; and A respective weight for each of a plurality of memory locations in the memory is generated from (i) the decorrelated query and (ii) a respective key for each of a plurality of memory locations in the memory.

8. The method according to claim 1, wherein: The corresponding weights are sparse weights that are non-zero only for a proper subset of memory locations in the memory.

9. The method according to claim 1, further comprising: receiving a first representation of a first data item in the set of data items; processing, by the query neural network, the first representation of the first data item to generate a first query; generating a respective first weight for each of the plurality of memory locations in the memory from (i) the first query and (ii) a respective key for each of the plurality of memory locations in the memory; generating write data from said first representation of said first data item; as well as The obtained write data is written into the memory based on the corresponding first weight.

10. The method according to claim 9, wherein: Generating write data from the first representation of the first data item comprises: The first representation is processed using a written word neural network to generate the written data.

11. The method according to claim 9, wherein: Writing the obtained write data to the memory based on the corresponding first weight comprises, for each memory location: A product of the write data and the corresponding first weight of the memory location is added to the data stored at the memory location.

12. A system, the system comprising one or more computers and one or more storage devices, the one or more storage devices storing instructions, the instructions when executed by the one or more computers causing the one or more computers to perform operations comprising: receiving a representation of an input data item; processing said representation of said input data item by a query neural network to generate a query; generating a respective weight for each of a plurality of memory locations in a memory from (i) the query and (ii) a respective key for each of a plurality of memory locations in the memory, wherein the memory stores data representing a set of data items; generating output memory data from the memory based on respective weights of the plurality of memory locations and data stored at the plurality of memory locations; as well as An output is generated from at least the output memory data indicating whether the input data item belongs to the set of data items.

13. The system of claim 12, the operations further comprising: The input data item is processed using an encoder neural network to generate the representation of the input data item.

14. The system according to claim 12, wherein: Generating output memory data from the memory based on respective weights of the plurality of memory locations and data stored at the plurality of memory locations comprises: For each memory location, a product is calculated between the data stored in the memory location and the corresponding weight of the memory location.

15. The system of claim 12, wherein: Generating an output from at least the output memory data indicating whether the input data item belongs to the set of data items comprises: At least the output memory data is processed by an output representation neural network to generate the output indicating whether the input data item belongs to the set of data items.

16. The system of claim 15, the operations further comprising: processing said representation of said input data item using a written word neural network to generate written data; And among them, Processing at least the output memory data by an output representation neural network to generate the output indicating whether the input data item belongs to the set of data items includes: processing at least the write data and the output memory data by the output representation neural network.

17. The system of claim 16, wherein: Processing at least the output memory data by an output representation neural network to generate the output indicating whether the input data item belongs to the set of data items includes: processing the write data, the representation of the input data item and the output memory data by the output representation neural network.

18. The system of claim 12, wherein: Generating a respective weight for each of a plurality of memory locations in the memory from (i) the query and (ii) a respective key for each of a plurality of memory locations in the memory comprises: generating a decorrelated query from the query; and A respective weight for each of a plurality of memory locations in the memory is generated from (i) the decorrelated query and (ii) a respective key for each of a plurality of memory locations in the memory.

19. The system of claim 12, wherein: The corresponding weights are sparse weights that are non-zero only for a proper subset of memory locations in the memory.

20. The system of claim 12, the operations further comprising: receiving a first representation of a first data item in the set of data items; processing, by the query neural network, the first representation of the first data item to generate a first query; generating a respective first weight for each of the plurality of memory locations in the memory from (i) the first query and (ii) a respective key for each of the plurality of memory locations in the memory; generating write data from said first representation of said first data item; as well as The obtained write data is written into the memory based on the corresponding first weight.

21. The system of claim 20, wherein: Generating write data from the first representation of the first data item comprises: The first representation is processed using a written word neural network to generate the written data.

22. The system of claim 20, wherein: Writing the obtained write data to the memory based on the corresponding first weight comprises, for each memory location: A product of the write data and the corresponding first weight of the memory location is added to the data stored at the memory location.

23. A non-transitory computer-readable medium storing instructions which, when executed by one or more computers, cause the one or more computers to perform the method according to any one of claims 1 to 11.

Images