Memory network initialization

Abstract

A memory network is constructed using at least memory write weights, memory read weights, and at least one read vector, where the memory write weights parameterize the neural network's memory write operations to the memory matrix and the memory read weights parameterize the neural network's memory read operations from the memory matrix. At least one of the elements of the memory write weights, memory read weights, and at least one read vector is initialized to have sparsity or a low-discrepancy sampling pattern, or both. The memory network is trained to perform a task.

Description

[Technical field]

[0001] This application relates generally to initialization of memory networks, and more particularly to machine learning and machine learning systems coupled with memory components. [Background technology]

[0002] Neural Turing Machines (NTMs), Memory Augmented Neural Networks (MANNs), Differentially Computable Neural Computers (DNCs) and other memory networks can couple artificial neural networks to external memories. Coupling allows memory networks to outperform recurrent neural networks, such as long short-term memories (LSTMs), in tasks that require learning complex data transformations over time, such as sorting, associative recall, and few-shot learning. However, training these memory networks is not easy, and existing approaches are not reliable. For example, training can often break down reading and writing to a single memory slot. Training memory networks with existing approaches can be very slow to converge. Summary of the Invention

[0003] A computer-implemented method according to one aspect of the invention includes constructing a memory network with at least memory write weights, memory read weights, and at least one read vector. The memory write weights parameterize memory write operations of the neural network to an external memory component of the memory network. The memory read weights parameterize memory read operations of the neural network from the external memory component of the memory network. The method also includes initializing at least one selected from a group of elements of at least the memory write weights, the memory read weights, and the at least one read vector to have sparsity. The method further includes training the memory network with at least one selected from a group of elements of at least the memory write weights, the memory read weights, and the at least one read vector initialized to have sparsity properties. The training step updates the memory write weights, the memory read weights, and the at least one read vector.

[0004] In another aspect, a computer-implemented method of training a memory network may include constructing the memory network with at least memory write weights, memory read weights, and at least one read vector. The memory write weights parameterize memory write operations of the neural network to an external memory component of the memory network. The memory read weights parameterize memory read operations of the neural network from the external memory component of the memory network. The method may also include initializing at least the memory write weights, the memory read weights, and at least one selected from the group of elements of the at least one read vector to configured values ​​that follow a low discrepancy sequence. The method may also include training the memory network with at least the memory write weights, the memory read weights, and at least one selected from the group of elements of the at least one read vector with initial configured values ​​that follow a low discrepancy sequence. The training step updates the memory write weights, the memory read weights, and at least one read vector.

[0005] In one aspect, a system for training a memory network includes a hardware processor and a memory device coupled to the hardware processor. The memory device stores a memory matrix. The hardware processor is configured to construct the memory network with at least memory write weights, memory read weights, and at least one read vector. The memory write weights parameterize memory write operations of the neural network to the memory matrix. The memory read weights parameterize memory read operations of the neural network from the memory matrix. The hardware processor is configured to initialize at least one selected from the group of the memory write weights, the memory read weights, and the elements of the at least one read vector to have sparsity. The hardware processor is also configured to train the memory network with at least one selected from the group of the memory write weights, the memory read weights, and the elements of the at least one read vector, initialized to have sparsity properties. Training includes updating the memory write weights, the memory read weights, and the at least one read vector.

[0006] A computer readable storage medium storing a program of instructions executable by a machine to perform one or more of the methods described herein may also be provided.

[0007] Further features, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings, where like reference numbers indicate identical or functionally similar elements. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates components of a system in one embodiment. [Diagram 2]1 illustrates a method for training a memory network according to one embodiment. [Diagram 3] 4 is a flow diagram illustrating a method according to another embodiment. [Figure 4] In one embodiment, an example of a method for generating low discrepancy sequences is shown, in which a computer program is used to generate a Sobol sequence. [Diagram 5] FIG. 1 illustrates components of a system for training a memory network, according to one embodiment. [Figure 6] 1 illustrates an overview of an exemplary computer or processing system on which the system may be implemented in one embodiment. [Figure 7] FIG. 1 illustrates a cloud computing environment in one embodiment. [Figure 8] 1 illustrates a set of functional abstraction layers provided by a cloud computing environment in one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Systems, methods and techniques may be provided that enable fast and reliable initialization of memory networks. The systems may include, for example, computer systems. The methods may include, for example, computer-implemented methods. The memory networks combine machine learning models, such as artificial neural networks, with memory components that can read from and write to them. In the memory networks, the machine learning models may be trained to work with the memory components. Weights are parameters in the machine learning models. Training a model in machine learning involves learning or estimating weights (e.g., among other parameters), for example, by gradient descent backpropagation or other techniques.

[0010] In an embodiment, the system initializes memory write weights, read weights, or read vectors, or combinations thereof, of the memory network with sparse values, e.g., one or a few nonzero elements, instead of, e.g., small pseudorandom values ​​for many or all elements. In an embodiment, the system also generates such sparse initializations using low discrepancy sequences (e.g., instead of pseudorandom sequences). Examples of low discrepancy sequences include, but are not limited to, Sobol sequences, Halton sequences, Hammersley sequences, and van der Corput sequences.

[0011] An artificial neural network (ANN) is also referred to herein as a neural network. An embodiment of an implementation of an artificial neural network may include a series of interconnected neuron layers such that the output signal of a neuron in one layer is weighted and sent to a neuron in the next layer. A neuron Ni in a given layer may be connected to one or more neurons Nj in the next layer, and a different weight wij may be associated with each neuron-neuron connection Ni-Nj to weight the signal sent from Ni to Nj. A neuron Nj generates an output signal depending on its accumulated input, and the weighted signal may be propagated through successive layers of the network from the input to the output neuron layer. An artificial neural network machine learning model may undergo a training phase in which a set of weights associated with each neuron layer is determined. The network is exposed to a set of training data in an iterative training scheme, where the weights are iteratively updated as the network learns from the training data. The resulting learned model, together with the weights defined through the training operation, may be applied to perform a task based on new data.

[0012] The memory network includes a controller and a memory component. The controller may be a neural network using states, such as, for example, but not limited to, a long short-term memory (LSTM). The controller may also be another network, such as a feed-forward network. The controller may access the memory with read and write operations, which may be parameterized with weights. A network output that parameterizes the read and write operations with weights is referred to as a head. For example, the write weights define the extent to which the write head writes to a memory location, and the read weights define the extent to which the read head reads from a memory location. The read vector returned by the read head is a combination of a memory matrix and a read weighting vector (e.g., a weighted sum of the memory vectors). The number of read and write heads may be configured to have at least one read head and one write head, for example, based on the architecture of the memory network. Training the memory network includes learning the memory write weights, read weights, and read vectors of the memory network. The memory state of a memory network can be defined as a combination (e.g., concatenation) of read weights, write weights, read vectors, and memory matrices. Depending on the architecture of the memory network, additional parameters, such as but not limited to, usage vectors, precedence weights, and link matrices, may be defined and included in the memory state.

[0013] The external memory access mechanism may include content-based addressing and location-based addressing, which may provide memory addresses or cells for storing or retrieving information. Controller states such as read weights and write weights and other system parameters related to memory access are trainable. Training the weights may take into account factors such as the current input (e.g., content addressing by similarity), previous focus (interpolation), potential shifts in focus (convolution shifts) and blurring (sharpening). The process of training the memory network initializes these weights and vectors, which are updated throughout the learning process until the error converges, e.g., to a minimum acceptable value or other criterion.

[0014] Systems, methods and techniques in accordance with embodiments provide techniques for initializing weights and / or vectors that enable efficient and rapid convergence, thus enabling more efficient and faster training of memory network models.

[0015] Embodiments of the systems described herein train a memory network by initializing write weights, read weights, or read vectors with sparse values. The write weights, or weights, parameterize the memory cells where information is stored. The read weights, or weights, parameterize the memory cells where information is read.

[0016] As an example, the system initializes the write weights such that a single randomly selected weight for one write head is initialized to 1 and the other weights are initialized to 0. The system initializes all read weights, read vectors and memory matrices to 0. Experiments performed with simulation parameters (e.g., 100 LSTM units, 128x20 memory matrix, training sequence length 1-20, batch size (epoch number = sequence number) 1, 1 read head, 1 write head) show that roughly 80% of training trials converge fairly quickly (e.g., 12-30 minutes on an Intel® Core i7-5600U central processing unit (CPU) at 2.6 GHz with AVX2 (SIMD) support for TensorFlow software).

[0017] In an embodiment, the system uses a low discrepancy sequence to generate such a sparse initialization. A low discrepancy sequence refers to a sequence that has the property that its subsequences x1,...,xN have low discrepancy for all values ​​of N. A low discrepancy sequence is also referred to as a pseudorandom sequence or a quasi-random sequence. A low discrepancy sequence covers a more uniform space than a pseudorandom sequence for a finite number of points. For example, a low discrepancy sequence has a more uniform distribution or a higher degree of equidistribution with respect to a pseudorandom sequence. Examples include, but are not limited to, Sobol sequences, Halton sequences, Hammersley sequences, and van der Corput sequences.

[0018] For example, the system in an embodiment selects or picks sparse initial values ​​from a low discrepancy sequence. Simulation experiments in which a single write weight is drawn from a (40,127) Sobol sequence and initialized to 1, other write weights are initialized to 0, and read weights, read vectors, and memory matrices are all initialized to 0 show that with example simulation parameters of 100 LSTM units, 127×20 memory matrix, 1-20 training sequence lengths, batch size=1 (so epoch number=sequence number), 1 read head, and 1 write head, all training trials (e.g., 15 runs) converge faster, e.g., within 12-30 minutes. Observations from the experiments show that memory networks trained with training sequence lengths between 1 and 20 generalize well to unknown sequences of sequence lengths of 40, 80, and 120.

[0019] The following example illustrates a possible way to implement a method in one embodiment. This example is not meant to be limiting. Rather, it is provided as an example and for illustrative purposes only. The example code is written in Python, but may be written in any other programming language as well. Line numbers do not correspond to actual line numbers in the code, but are provided only for ease of reference in the following description.

[0020]

number

[0021] In line number 2, "seq_wrap" is a variable that represents the number of times the generated Sobol sequence is repeated (wrapped around) to generate a longer sequence. Since the number and length of the generated Sobol sequences are limited and the total length of the concatenated sequence is much smaller than the number of iterations used to train the neural network, one possible way to provide a larger sequence is to repeat the generated sequence. "iterations" represents the number of iterations used for training. "batch_size" represents the batch size used in training. "write_heads" represents the number of write heads. "mem_words" represents the number of memory words (e.g., the size of N in an NxM memory matrix, where N represents the number of memory locations). "sobol_len" represents the number of generated sobol sequences. The parameters "batch_size", "write_heads" and "mem_words" are configurable parameters in the memory network architecture.

[0022] In line 3, a Sobol string of number "sobol_len" and length "mem_words" is generated and stored in "mem_seed" as an array with "mem_words" rows and "sobol_len" columns. For example, mem_words=127, sobol_len=40 and mem_seed=[[64,96,32,48,...,65,1],[64,32,96,48,...,21,85],...,[64,32,96,80,...,43,107]] represents a 127x40 array, with 40 Sobol strings each of length 127 (i.e., a total of 127x40=5080 elements).

[0023] In line 4, the array stored in "mem_seed" is reformed as an array with one row and "mem_words*sobol_len" columns (equivalent to a row vector of length "mem_words*sobol_len"), and the result is stored in "mem_seed". For example, if before execution of line 4, mem_seed=[[64,96,32,48,...,65,1], [64,32,96,48,...,21,85],...,[64,32,96,80,...,43,107]], then after execution of line 4, mem_seed=[64,96,32,48,...,65,1,64,32,96,48,...,21,85,...,64,32,96,80,...,43,107].

[0024] In line number 5, the array stored in "mem_seed" is repeated along its first axis (e.g., rows) "seq_wrap" times, resulting in an array with one row and "mem_words*sobol_len*seq_wrap" columns, which is stored in "mem_seed". As an illustrative example, if seq_wrap=2 (although "seq_wrap" can be much greater than 2) and before execution of line number 5, mem_seed=[64,96,32,48,...,43,107], then after execution of line number 5, mem_seed=[64,96,32,48,...,43,107,64,96,32,48,...,43,107].

[0025] At line number 7, a loop over the training iterations begins. "count" represents the current iteration (e.g., the current loop level, starting from 1).

[0026] In line number 9, "write_weightings" is a three-dimensional array of sizes "batch_size", "mem_words", and "write_heads" for the first, second, and third dimensions, respectively, that represent the write weightings. All elements of "write_weightings" are set to zero. "float32" represents a 32-bit floating point (also referred to as single precision) numeric format. Other numeric formats (including but not limited to 16-bit floating point) may be used as well.

[0027] In lines 10-12, some elements of “write_weightings” are set to 1 while the rest remain 0, as determined by the sequence stored in “mem_seed”. For example, if mem_seed=[64, 96, 32, 48, 112, 80, 16, 24, 88, 120, 56, 40, 104, 72, …, 43, 107] and batch_size=3 and write heads=2 for the first training iteration (e.g., count=1),

number

number

[0028] In line number 13, "read_weightings" is a three-dimensional array of sizes "batch_size", "mem_words", and "read_heads" for the first, second, and third dimensions, respectively, that represents read weightings. All elements of "read_weightings" are set to zero. "read_heads" represents the number of read heads.

[0029] In line 14, "memory_matrix" is a three-dimensional array of size "batch_size", "mem_words", and "mem_width" for the first, second, and third dimensions, respectively, that represents the memory matrix. All elements of "memory_matrix" are set to zero. "mem_width" represents the memory width (e.g., in an NxM memory matrix, the size of M, the vector size in memory locations).

[0030] In line 15, "read_vectors" is a three-dimensional array of sizes "batch_size", "mem_width", and "read_heads" for the first, second, and third dimensions, respectively, that represents read vectors. All elements of "read_vectors" are set to zero.

[0031] Line number 16 represents a subsequent portion of code (not shown) where training for each iteration is performed using the write weights, read weights, memory matrix and read vector initialized for that iteration as described above.

[0032] In the above example, the write weights are initialized as sparse arrays following the pattern of the low discrepancy sequence (Sobol sequence in this example) as described above. In an embodiment, both sparsity and low discrepancy may be utilized. In another embodiment, sparsity and low discrepancy may be implemented as two separate features. For example, in line number 9, if the write weights are initialized as small random numbers instead of zero and some elements of the write weights are set to 1 as described in lines numbers 10-12, the resulting write weights are not sparse, but the write weights set to 1 follow the pattern of the low discrepancy sequence and may have low discrepancy characteristics.

[0033] In another example, the write weights are initialized to zero as in line 9 above, but instead of setting some elements of the write weights to 1 based on the low discrepancy column as in lines 10-12, the write weights are randomly set to 1. The resulting write weights may not have the low discrepancy properties, however, they are still sparse.

[0034] In the above example, in line 12, exactly one element of a given set of write weights is set to 1. In other examples, a larger number may be set to 1, for example two or more elements of a given set.

[0035] In the above example, in line number 12, some elements of the write weighting are set to 1. In other examples, these elements may be set to different non-zero values, for example 0.5, 2, or 5, or another value.

[0036] In the above example, the write weights are initialized as sparse and low discrepancy arrays, while the read weights and read vectors are initialized to zero. In another example, the read weights are initialized as sparse and low discrepancy arrays, while the write weights and read vectors are initialized to zero. For example, in an embodiment, at least one of the write weights, read weights, and read vectors may be initialized as sparse and / or low discrepancy arrays.

[0037] In the above example, in lines 2-5, the generated low discrepancy sequence is flattened (i.e., converted from a two-dimensional array to a one-dimensional array, i.e., vector) and wrapped (i.e., repeated a desired number of times). The resulting sequence is then used as a template for setting elements of the write weights to one, as described above. In various embodiments, various other arrangements may be used. For example, a concatenation of low discrepancy sequences may be used, where a given low discrepancy sequence appears at random positions and a random number of times within a concatenated low discrepancy sequence of a desired length. In another example, a one-dimensional array (i.e., vector) of low discrepancy sequences, such as the one generated in line 4, may be provided. Then, at the start of the training iteration for each training iteration, a random element of the vector may be selected and assumed to be the start of the vector. The vector may be read from its start and used as a template for setting the write weights to one, as described above. If the end of the vector is reached within a training iteration, the next element may be read from the beginning of the vector. In yet another example, a two-dimensional array of low discrepancy columns may be provided, such as the one generated in row number 3. Then, at the start of a training iteration (e.g., for each of the training iterations), a random number sequence may be selected from the array and used as a template to set the write weights to 1, as described above. When the end of the sequence is reached, another random number sequence may be selected from the array.

[0038] FIG. 1 illustrates components of a system in one embodiment. The illustrated components include, for example, computer-implemented components implemented and / or executed on or coupled with one or more hardware processors. The one or more hardware processors may include components such as, for example, programmable logic devices, microcontrollers, memory devices, or other hardware components, or combinations thereof, that may be configured to perform the respective tasks described in this disclosure. The associated memory devices may be configured to selectively store instructions executable by the one or more hardware processors. For example, the processor may be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), other suitable processing components or devices, or one or more combinations thereof. The processor may be coupled with a memory device. The memory device may include a random access memory (RAM), a read only memory (ROM), or other memory device, and may store data and / or processor instructions for implementing various functions associated with the methods and / or systems described herein. The processor can execute computer instructions that are stored in memory or received from other computer devices or media.

[0039] The memory network may include a controller 102, such as an artificial neural network, and a memory component 104, such as a memory external to the neural network. The memory component may also be referred to as a memory matrix. The controller 102 may access the memory 104 with read and write operations, which may be parameterized with weights (e.g., write weights for write operations and read weights for read operations). Training the memory network includes having the memory network learn the memory write weights of the memory network for writing to selective locations of the memory 104 and read weights for reading from selective locations of the memory 104. The read vectors are returned as a combination of row vectors in the memory, determined based on the read weights. That is, for each instance of data (e.g., training sample), M read vectors are generated for each read head, where M is the width of the memory matrix.

[0040] For example, a computer processor, shown at 106, may construct a memory network with at least one memory write weight, a memory write weight, and at least one read vector. The computer processor 106 may also initialize a memory matrix 104. The memory write weights parameterize memory write operations to the memory matrix 104 of the neural network (e.g., 102). The memory write weights parameterize memory read operations from the memory matrix 104 of the neural network (e.g., 102). The computer processor 106 may initialize at least one of the write weights, read weights, or read vectors to be sparse or have sparsity properties. The computer processor 106 may train the memory network 108 with at least one of the write weights, read weights, or read vectors initialized to have sparsity properties. The training updates the write weights, read weights, and read vectors.

[0041] In an embodiment, the computer processor 106 may use a low discrepancy sequence to generate a sparse set for applying to at least one of the write weights, the read weights, or the elements of at least one read vector. Examples of low discrepancy sequences include, but are not limited to, Sobol sequences, Halton sequences, Hammersley sequences, and van der Corput sequences.

[0042] For example, in an embodiment, the computer processor 106 may initialize at least one of the write weights, read weights, or read vectors to be sparse by setting the write weights of the sparse numbers to a non-zero value and setting the remaining write weights, read weights, and elements of the at least one read vector to zero values. In an embodiment, the write weights of the sparse numbers that are set to have non-zero values ​​follow a pattern of a low discrepancy sequence.

[0043] As another example, in an embodiment, the computer processor 106 may initialize at least one of the write weights, read weights, or read vectors to be sparse by setting the read weights of the sparse numbers to a non-zero value and setting the remaining read weights, the write weights, and the elements of the at least one read vector to have zero values. In an embodiment, the read weights of the sparse numbers that are set to non-zero follow a pattern of a low discrepancy sequence.

[0044] 2 is a method of training a memory network according to an embodiment. The method may be implemented or executed on a hardware processor, e.g., a computer processor. The method includes, at 202, constructing a memory network with at least memory write weights, memory read weights, and at least one read vector. In one aspect, the memory write weights parameterize memory write operations of the neural network to an external memory component of the memory network, and the memory read weights parameterize memory read operations of the neural network from the external memory component of the memory network.

[0045] The method includes initializing at least one of the write weights, read weights, or read vectors to have sparsity at 204. For example, the method picks out a write weight, read weight, or read vector for selection. The selected one is then initialized to have the sparse characteristic. For example, in an embodiment, the elements of the write weights, read weights, and read vector may be set to a first value, such as "0". The selected one (write weight, read weight, or read vector) is then set to a sparse (small) number of its elements to have a second value, such as "1". Values ​​other than "0" and "1" are used to indicate the sparse characteristic. In an embodiment, the method uses a low discrepancy sequence to generate a sparse set of values.

[0046] As an example, the method may set the write weights of the sparse numbers to a non-zero value and set the remainder of the write weights, the read weights, and at least one element of the read vector to zero values. In one embodiment, the write weights of the sparse numbers that are set to non-zero values ​​follow a pattern of a low discrepancy sequence.

[0047] As another example, the method may set the read weights of the sparse numbers to a non-zero value and set the remainder of the read weights, the write weights, and at least one element of the read vector to have zero values. In one embodiment, the read weights of the sparse numbers that are set to non-zero follow the pattern of a low discrepancy sequence.

[0048] As yet another example, the method may set a sparse number of elements of the read vector to a non-zero value and set the remainder of the elements of the read vector, the read weights and the write weights, to have zero values. In one embodiment, the sparse number of elements of the read vector that are set to non-zero follow the pattern of a low discrepancy sequence.

[0049] In an embodiment, the sparse number is 1. In other embodiments, the sparse number may be greater than 1, but less than a configurable number.

[0050] The method includes, at 206, training the memory network with at least one of the write weights, read weights, or read vectors initialized to have a sparse characteristic. The training updates the write weights, read weights, and at least one read vector. The write weights and read weights can be considered as connection weights between the neural network and an external memory. In an aspect, the memory network may be trained by gradient descent.

[0051] 3 is a flow diagram illustrating a method according to another embodiment. The method may be implemented or executed on a hardware processor, e.g., a computer processor. The method includes, at 302, constructing a memory network with at least memory write weights, memory read weights, and at least one read vector. The memory write weights parameterize memory write operations of the neural network to an external memory component of the memory network. The memory read weights parameterize memory read operations of the neural network from an external memory component of the memory network.

[0052] The method includes initializing at least one selected from a group of at least a write weight, a read weight, and at least one read vector to a configured value that follows a low discrepancy sequence, for example initialized by a low discrepancy sampling pattern, at 304. For example, the method includes selecting a write weight, a read weight, or a read vector. The method may set an element of the selection to a configured value, such as "1", that follows the low discrepancy sequence. The initializing step may also include setting elements of the write weight, read weight, and read vector that are not selected for the low discrepancy sequence to have a value other than the configured value. As an example, the initializing step may include setting elements of the write weight, read weight, and read vector that are not selected to follow the low discrepancy sequence to have a value of zero. As yet another example, the initializing step may include setting elements of the write weight, read weight, and read vector that are not selected to follow the low discrepancy sequence to have a random number other than the configured value.

[0053] The method includes, at 306, training the memory network with initialized write weights, read weights, and read vectors, where at least one of the write weights, read weights, and read vectors are set to include configured values ​​that follow a low discrepancy sequence. The training updates the write weights, read weights, and at least one read vector. The write weights and read weights can be considered as connection weights between the neural network and an external memory. In an aspect, the memory network may be trained by gradient descent.

[0054] Figure 4 shows an example of generating a Sobol sequence using a Python (computer programming language) interpreter as an example method for generating low discrepancy sequences. Language commands are shown preceded by ">>>" followed by the output that results from the executed command. The first command "import sobol_seq" imports the available Sobol sequence package. The second command creates 40 length 2 sequences. 7 It outputs a matrix with a Sobol column of -1=127, i.e. sobol(40,127). The third command outputs the first column of the matrix output by the second command.

[0055] FIG. 5 illustrates components of a system in one embodiment for training a memory network. One or more hardware processors 502, such as a central processing unit (CPU), a graphic processing unit (GPU) or a field programmable gate array (FPGA), an application specific integrated circuit (ASIC) or other processor or combination thereof, may be coupled with a memory device 504 to construct and train the memory network. The memory device 504 may include a random access memory (RAM), a read only memory (ROM) or other memory device and may store data and / or processor instructions for performing various functions associated with the methods and / or systems described herein. The one or more processors 502 may execute computer instructions stored in the memory 504 or received from other computer devices or media. The memory device 504 may store, for example, instructions and / or data for functioning the one or more hardware processors 502 and may include an operating system and other instructions and / or data programs. The one or more hardware processors 502 can configure the memory network and initialize parameters of the memory network. For example, the one or more hardware processors 502 can initialize at least one of the write weights, read weights, or read vectors to have sparsity. In another aspect, the one or more hardware processors can initialize at least one of the write weights, read weights, or read vectors to values ​​configured according to a low discrepancy sequence. In yet another aspect, the one or more hardware processors can initialize at least one of the write weights, read weights, or read vectors to have sparsity and follow a low discrepancy sequence in setting the sparse values.Input or training data may be stored in the storage device 504 or received via the network interface 508 from a remote device and temporarily loaded into the memory device 504 to construct or generate a trained memory network. In one aspect, the generated or trained memory network may be stored in the memory device 504 for execution by one or more hardware processors 502, e.g., to perform tasks. The one or more hardware processors 502 may be coupled with interface devices, e.g., a network interface 508 for communicating with remote systems over a network, an input / output interface 510 for communicating with input devices, output devices, or input / output devices, e.g., a keyboard, mouse, display, etc.

[0056] FIG. 6 is a schematic diagram of an exemplary computer or processing system on which a system may be implemented in one embodiment. The computer system is merely one example of a suitable processing system and is not intended to suggest any limitation as to the scope of use or functionality of the method embodiments described herein. The illustrated processing system may be operated in many other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments or configurations or combinations thereof suitable for use with the processing system illustrated in FIG. 6 may include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, distributed cloud computing environments including any of the above systems or devices.

[0057] A computer system may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types. The computer system may also be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0058] Components of a computer system may include, but are not limited to, one or more processors or processing units 12, a system memory 16, and a bus 14 that couples various system components, including the system memory 16, to the processor 12. The processor 12 may include modules 30 that perform the methods described herein. The modules 30 may be programmed into integrated circuits of the processor 12 or may be loaded from the memory 16, the storage device 18, or a network 24, or a combination thereof.

[0059] Bus 14 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, including, by way of example and without limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0060] The computer system may include a variety of computer system readable media. Such media may be any available media that can be accessed by the computer system and may include both volatile and nonvolatile media, removable and non-removable media.

[0061] System memory 16 may include computer system readable media in the form of random access memory (RAM) and / or cache memory or other volatile memory. The computer system may further include other removable / non-removable volatile / non-volatile computer system storage media. By way of example, storage system 18 may be provided for reading from and writing to a non-removable non-volatile magnetic medium (e.g., a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), an optical disk drive for reading from or writing to a removable non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media may be provided. In such cases, each may be connected to bus 14 by one or more data media interfaces.

[0062] The computer system may also communicate with one or more external devices 26, such as a keyboard and / or pointing device, a display 28, and / or any devices that enable the computer system to communicate with one or more other computer systems (e.g., network cards, modems, etc.). Such communication may occur through an input / output (I / O) interface 20.

[0063] Additionally, the computer system may communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN) or a public network (e.g., the Internet) or combinations thereof, via a network adapter 22. As shown, the network adapter 22 may communicate with other components of the computer system via the bus 14. It is understood that other hardware and / or software components, not shown, may also be used in the computer system. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives and data archive storage systems.

[0064] Although this disclosure may include a description of cloud computing, it is to be understood that implementation of the teachings detailed herein is not limited to a cloud computing environment. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or hereafter developed. Cloud computing is a model of service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0065] The characteristics are as follows:

[0066] On-Demand Self-Service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, as they need it automatically, without the need for human interaction with the service provider.

[0067] Broadband Network Access: Capabilities are available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, PDAs).

[0068] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided to them, but there is a sense of location independence in the sense that they can specify location at a higher level of abstraction (e.g. country, state, or data center).

[0069] Rapid Elasticity: Capacity can be rapidly and elastically, in some cases automatically, provisioned to quickly scale out and rapidly released to quickly scale in. To the consumer, the provisioned available capacity often appears externally unlimited, and any amount can be purchased at any time.

[0070] Measured Services: Cloud systems automatically control and optimize resource usage by utilizing metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, number of active users). Resource usage is monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0071] The service model is as follows:

[0072] Software as a Service (SaaS): The capability offered to the consumer is to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities with the potential exception of limited user-specific application configuration settings.

[0073] Platform as a Service (PaaS): The capability offered to a consumer is to deploy consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and possibly the configuration of the application hosting environment.

[0074] Infrastructure as a Service (IaaS): The capability provided to the consumer is to provide processing, storage, networking, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over selected networking components (e.g., host firewalls).

[0075] The deployment model is as follows:

[0076] Private Cloud: The cloud infrastructure is used exclusively for one organization. It may be managed by the organization or a third party and may be on-premise or off-premise.

[0077] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with common concerns (e.g., mission, security requirements, policy and compliance considerations). It may be managed by the organization or a third party and may exist on-premise or off-premise.

[0078] Public Cloud: The cloud infrastructure is available to the general public or large industry entities and is owned by organizations that sell cloud services.

[0079] Hybrid Cloud: A cloud infrastructure is a blend of two or more clouds (private, community or public) that remain unique entities but are joined by standardized or proprietary technologies that allow data and application portability (e.g. cloud bursting for load balancing between clouds).

[0080] Cloud computing environments are service-oriented with an emphasis on statelessness, low coupling, modularity and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0081] Referring now to FIG. 7, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, a PDA or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped (not shown) in one or more networks, such as private, community, public, or hybrid clouds, as described above, or combinations thereof. This enables the cloud computing environment 50 to provide infrastructure, platform, or software, or combinations thereof, as a service, for which the cloud consumer does not need to maintain resources on a local computing device. The types of computing devices 54A-54N shown in FIG. 7 are for illustrative purposes only, and it will be understood that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network, network addressable connection (e.g., using a web browser), or both.

[0082] Referring now to Figure 8, there is shown a set of functional abstraction layers provided by cloud computing environment 50 (Figure 7). It will be understood that the components, layers and functions shown in Figure 8 are for illustrative purposes only and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0083] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, servers based on RISC (Reduced Instruction Set Computing) architecture 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0084] The virtualization layer 70 provides an abstraction layer from which examples of virtualized entities such as virtualized servers 71, virtualized storage 72, virtualized networks including virtual private networks 73, virtualized applications and operating systems 74, and virtual clients 75 are provided.

[0085] In one example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides tracking of costs as resources are utilized within the cloud computing environment and billing or invoicing for consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and fulfillment 85 provides pre-allocation and procurement of cloud computing resources in anticipation of future demand in accordance with SLAs.

[0086] The Workload Layer 90 provides examples of functionality for which cloud computing environments are utilized. Examples of workloads and functions provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and memory, network, and training processing 96.

[0087] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0088] A computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Illustrative enumeration of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, punch cards or mechanically encoded devices such as ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. Computer-readable storage media, as used herein, is not to be construed as a transitory signal per se, such as an electric wave, a freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0089] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.

[0090] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented languages ​​such as Smalltalk®, C++ or the like, procedural languages ​​such as the C programming language or similar programming languages. The computer readable program instructions may be executed as a stand-alone software package, entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, the electrical circuitry may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electrical circuitry, which may include, for example, programmable logic circuitry, a field programmable gate array (FPGA), or a programmable logic array (PLA), to perform aspects of the invention.

[0091] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer readable program instructions.

[0092] These computer readable program instructions, when provided to a computer processor or other programmable data processing apparatus, cause the instructions to execute on the computer processor or other programmable data processing apparatus to produce a machine-generated means for implementing the functions / acts identified in the block or blocks of the flowchart diagrams and / or blocks. These computer readable program instructions may also be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus or other device or combination thereof to function in a particular manner, such that a computer readable storage medium having instructions stored thereon includes an article of manufacture including instructions that implement aspects of the functions / acts identified in the block or blocks of the flowchart diagrams and / or blocks.

[0093] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to generate a computer implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement aspects of the functions / actions identified in a block or blocks of the flowchart and / or blocks.

[0094] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment or portion of instructions, including one or more executable instructions for implementing a particular logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order depicted in the drawings. For example, two blocks shown in succession may in fact be realized as one step, may be executed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, or the blocks may be executed in reverse order depending on the functionality involved. It should be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations may be implemented by a special-purpose hardware-based system that performs a particular function or action or implements a combination of special-purpose hardware and computer instructions.

[0095] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural unless the context clearly indicates otherwise. As used herein, the term "or" is an inclusive operator and means "and / or" unless the context explicitly or clearly indicates otherwise. Furthermore, it is to be understood that the terms "comprise", "comprises", "comprising", "include", "includes", "including" or "having" or combinations thereof, as used herein, specify the presence of stated features, integers, steps, operations, elements or components or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components or groups or combinations thereof. As used herein, the phrase "in an embodiment" does not necessarily refer to the same embodiment, although it may. As used herein, the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it may be so. As used herein, the phrase "in another embodiment" does not necessarily refer to a different embodiment, although it may be so. Furthermore, embodiments and / or elements of embodiments may be freely combined with each other, unless they are mutually exclusive.

[0096] The corresponding structures, materials, acts, and equivalents of all means or step-plus-function elements, if any, in the following claims are intended to include any structures, materials, or acts for performing a function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The embodiments have been selected and described in order to best explain the principles and practical applications of the invention, and to enable others skilled in the art to understand the invention in various embodiments with various modifications suitable for the particular applications intended.

Claims

1. 1. A computer-implemented method for training a memory network, comprising: constructing a memory network with at least memory write weights, memory read weights, and at least one read vector, the memory write weights parameterizing memory write operations of the neural network to an external memory component of the memory network, and the memory read weights parameterizing memory read operations of the neural network from the external memory component of the memory network; initializing at least one selected from the group of the memory write weights, the memory read weights, and the elements of the at least one read vector to have sparsity; training the memory network using at least one selected from a group of elements of the memory write weights, the memory read weights, and the at least one read vector initialized to have a sparse characteristic, the training step updating the memory write weights, the memory read weights, and the at least one read vector; A method comprising:

2. 2. The method of claim 1 , wherein initializing the at least one selected from the group of the memory write weightings, the memory read weightings, and the elements of the at least one read vector to have sparsity comprises setting a sparse number of the memory write weightings to a non-zero value and setting the remainder of the memory write weightings, the memory read weightings, and the elements of the at least one read vector to zero values.

3. The method of claim 2 , wherein the memory write weightings of the sparse numbers that are set to non-zero values ​​follow a pattern of low discrepancy amounts.

4. 2. The method of claim 1 , wherein initializing the at least one selected from the group of the memory write weightings, the memory read weightings, and the elements of the at least one read vector to have sparsity comprises setting a sparse number of the memory read weightings to a non-zero value and setting the remainder of the memory read weightings, the memory write weightings, and the elements of the at least one read vector to have zero values.

5. The method of claim 4 , wherein the memory read weightings of the sparse numbers that are set to non-zero values ​​follow a pattern of low discrepancy columns.

6. 2. The method of claim 1, wherein initializing the at least one selected from the group of the memory write weights, the memory read weights, and the elements of the at least one read vector to have sparsity comprises setting a sparse number of elements of the at least one read vector to a non-zero value and setting the remainder of the elements of the at least one read vector, the memory read weights, and the memory write weights, to have zero values.

7. The method of claim 6 , wherein the sparse number of elements of the at least one read vector that are set to a non-zero value follow a pattern of low discrepancy sequences.

8. 2. The method of claim 1, further comprising: using a low discrepancy sequence to generate at least one sparse set selected from a group of the memory write weights, the memory read weights, and the at least one read vector element.

9. The method according to any one of claims 2 to 7, wherein the sparse number is one.

10. The method of any one of claims 2 to 7, wherein the sparse number is greater than one.

11. 1. A computer-implemented method for training a memory network, comprising: constructing a memory network with at least memory write weights, memory read weights, and at least one read vector, the memory write weights parameterizing memory write operations of the neural network to an external memory component of the memory network, and the memory read weights parameterizing memory read operations of the neural network from the external memory component of the memory network; initializing at least one selected from the group of the memory write weights, the memory read weights, and the elements of the at least one read vector to configured values ​​that follow a low discrepancy sequence; training the memory network using at least one selected from a group of elements of the memory write weights, the memory read weights, and the at least one read vector with initial configured values ​​that follow a low discrepancy sequence, the training step updating the memory write weights, the memory read weights, and the at least one read vector; A method comprising:

12. The initializing step further comprises: setting the memory write weights, the memory read weights and the elements of the at least one read vector not selected for the low discrepancy columns to have values ​​other than the configured values. The method of claim 11 , comprising:

13. The initializing step further comprises: setting the memory write weights, the memory read weights and the elements of the at least one read vector that are not selected to follow the low discrepancy sequence to have a value of zero. The method of claim 11 , comprising:

14. The initializing step further comprises: setting the memory write weights, the memory read weights and the elements of the at least one read vector that are not selected to follow the low discrepancy sequence to have random values ​​other than the configured values. The method of claim 11 , comprising:

15. 1. A system for training a memory network, comprising: a hardware processor; a memory device coupled to said hardware processor for storing a memory matrix; The hardware processor includes at least constructing a memory network with at least memory write weights, memory read weights, and at least one read vector, the memory write weights parameterizing memory write operations of the neural network to the memory matrix, and the memory read weights parameterizing memory read operations of the neural network from the memory matrix; initializing at least one selected from the group of the memory write weights, the memory read weights, and the elements of the at least one read vector to have sparsity; training the memory network using at least one selected from a group of elements of the memory write weights, the memory read weights, and the at least one read vector initialized to have a sparse characteristic, the training comprising updating the memory write weights, the memory read weights, and the at least one read vector; A system configured to:

16. The hardware processor includes: Initializing the at least one selected from the group of the memory write weights, the memory read weights, and the elements of the at least one read vector to have a sparse characteristic by setting a sparse number of the memory write weights to a non-zero value and setting the remaining memory write weights, the memory read weights, and the elements of the at least one read vector to a zero value. The system of claim 15 configured to:

17. 20. The system of claim 16, wherein the memory write weightings of the sparse numbers that are set to non-zero values ​​follow a pattern of low discrepancy amounts.

18. The hardware processor includes: Initializing the at least one selected from the group of the memory write weights, the memory read weights, and the elements of the at least one read vector to have a sparse characteristic by setting a sparse number of the memory read weights to a non-zero value and setting the remainder of the memory read weights, the memory write weights, and the elements of the at least one read vector to have zero values. The system of claim 15 configured to:

19. 20. The system of claim 18, wherein the memory read weightings of the sparse numbers that are set to non-zero values ​​follow a pattern of low discrepancy columns.

20. The hardware processor includes: generating at least one sparse set selected from a group of the memory write weights, the memory read weights, and the at least one read vector element using a low discrepancy sequence; The system of claim 15 configured to:

21. 1. A system for training a memory network, comprising: a hardware processor; a memory device coupled to said hardware processor for storing a memory matrix; The hardware processor includes at least constructing a memory network having at least memory write weights, memory read weights, and at least one read vector, the memory write weights parameterizing memory write operations of the neural network to an external memory component of the memory network, and the memory read weights parameterizing memory read operations of the neural network from the external memory component of the memory network; initializing at least one selected from a group of the memory write weights, the memory read weights, and the elements of the at least one read vector to configured values ​​according to a low discrepancy sequence; training the memory network using at least one selected from a group of elements of the memory write weights, the memory read weights, and the at least one read vector with initial configured values ​​following a low discrepancy sequence, the training updating the memory write weights, the memory read weights, and the at least one read vector; A system configured to:

Images