Initialization method and system for memory network

Abstract

A memory network can be constructed using at least a memory write weight, a memory read weight, and at least one read vector. The memory write weight parameterizes a neural network's memory write operation on a memory matrix, and the memory read weight parameterizes a neural network's memory read operation from the memory matrix. At least one of the write weight, the read weight, or an element of the at least one read vector can be initialized to have a sparse and / or low-discrepancy sampling pattern. The memory network can be trained to perform a task.

Description

Technical Field

[0001] The present application relates generally to initialization of memory networks, and more particularly to machine learning and machine learning systems coupled with memory components. Background Art

[0002] Memory networks such as neural Turing machines (NTMs), memory-augmented neural networks (MANNs), and differentiable neural computers (DNCs) can couple artificial neural networks to external memories. This coupling can allow memory networks to outperform recurrent neural networks (RNNs) such as long short-term memory (LSTM) in tasks that require learning complex data transformations over long periods of time (e.g., copying, classification, associative recall, and few-trigger learning). However, training these memory networks is not straightforward, and existing methods are unreliable. For example, training may often cause the reading and writing of a single memory slot to crash. Training memory networks using existing methods can be very slow to converge. Summary of the Invention

[0003] According to one aspect of the present invention, a computer-implemented method includes constructing a memory network using at least memory write weights, memory read weights, and at least one read vector. The memory write weights parameterize a neural network's memory write operations on an external memory component of the memory network. The memory read weights parameterize a neural network's memory read operations from the external memory component of the memory network. The method also includes initializing at least one selected from a group of elements having at least the write weights, the read weights, and the at least one read vector to have sparsity. The method also includes training the memory network using one selected from a group of elements having at least the write weights, the read weights, and the at least one read vector initialized to have sparse properties. The training updates the write weights, the read weights, and the at least one read vector.

[0004] In another aspect, a computer-implemented method for training a memory network includes constructing a memory network using at least memory write weights, memory read weights, and at least one read vector. The memory write weights parameterize memory write operations of a neural network to an external memory component of the memory network. The memory read weights parameterize memory read operations of the neural network from the external memory component of the memory network. The method may also include initializing at least one selected from a group of elements having at least the write weights, the read weights, and the at least one read vector to a configuration value that follows a low-discrepancy sequence. The method may also include training the memory network using one selected from a group of elements having at least the write weights, the read weights, and the at least one read vector having an initial configuration value that follows a low-discrepancy sequence. The training updates the write weights, the read weights, and the at least one read vector.

[0005] In one aspect, a system for training a memory network includes a hardware processor and a memory device coupled to the hardware processor. The memory device stores a memory matrix. The hardware processor is configured to construct a memory network using at least a memory write weight, a memory read weight, and at least one read vector. The memory write weight parameterizes a neural network's memory write operation on the memory matrix. The memory read weight parameterizes a neural network's memory read operation from the memory matrix. The hardware processor may also be configured to initialize at least one element selected from a group consisting of at least the write weight, the read weight, and the at least one read vector to have sparsity. The hardware processor may also be configured to train the memory network using an element selected from a group consisting of at least the write weight, the read weight, and the at least one read vector initialized to have sparsity. The training updates the write weight, the read weight, and the at least one read vector.

[0006] A computer-readable storage medium storing a program of instructions executable by a machine to perform one or more of the methods described herein may also be provided.

[0007] Further features, structure, and operation of various embodiments are described in detail below with reference to the accompanying drawings. In the drawings, like reference numerals indicate identical or functionally similar elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a diagram illustrating components of a system in one embodiment.

[0009] Figure 2 is a method for training a memory network according to one embodiment.

[0010] Figure 3is a flow chart illustrating a method in another embodiment.

[0011] Figure 4 An example of using a computer program to generate a Sobol sequence in one embodiment is shown as an example method for generating a low-discrepancy sequence.

[0012] Figure 5 is a diagram illustrating components of a system in one embodiment for training a memory network.

[0013] Figure 6 A schematic diagram shows an example computer or processing system in which the system may be implemented in one embodiment.

[0014] Figure 7 A cloud computing environment is shown in one embodiment.

[0015] Figure 8 A set of functional abstraction layers provided by a cloud computing environment in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Systems, methods, and techniques can be provided that allow for rapid and reliable initialization of memory networks. For example, the system can include a computer system. For example, the method can include a computer-implemented method. A memory network combines a machine learning model (e.g., an artificial neural network) with a memory component that can be read and written. In the memory network, the machine learning model can be trained to operate with the memory component. Weights are parameters in the machine learning model. Training a model in machine learning involves learning or estimating weights (e.g., in addition to other parameters), for example, by gradient descent backpropagation or another technique.

[0017] In one embodiment, the system initializes the memory write weights, read weights, and / or read vectors of such a memory network using sparse values, such as one or only a few non-zero elements, rather than, for example, small pseudo-random values for many or all elements. In one embodiment, the system also uses a low-discrepancy sequence to generate such sparse initialization (e.g., instead of a pseudo-random sequence). Examples of low-discrepancy sequences include, but are not limited to, Sobol, Halton, Hammersley, and van der Corput sequences.

[0018] Artificial neural networks (ANNs) are also referred to herein as neural networks. An embodiment of an implementation of an artificial neural network may include a series of neuron layers that are interconnected so that the output signals of the neurons in one layer are weighted and transmitted to the neurons in the next layer. A neuron Ni in a given layer can be connected to one or more neurons Nj in the next layer, and a different weight wij can be associated with each neuron-neuron connection Ni-Nj to weight the signal transmitted from Ni to Nj. Neuron Nj generates an output signal based on its accumulated input, and the weighted signal can propagate from the input to the output neuron layer on successive layers of the network. An artificial neural network machine learning model can go through a training phase in which a set of weights associated with the corresponding neuron layer is determined. In an iterative training scheme, the network is exposed to a set of training data, where the weights are repeatedly updated as the network "learns" from the training data. The resulting trained model with weights defined via the training operation can be applied to perform tasks based on new data.

[0019] A memory network includes a controller and a memory component. The controller can be a neural network, for example, having a state such as, but not limited to, a long short-term memory (LSTM). The controller can also be another network, such as a feedforward network. The controller can access memory using read and write operations, which can be parameterized using weights. The network output that parameterizes the read and write operations using weights is called a head. For example, a write weight defines the extent to which a write head writes to a memory location; a read weight defines the extent to which a read head reads from a memory location. The read vector returned by the read head is a combination of a memory matrix and a read weight vector (e.g., a weighted sum of the memory vectors). For example, the number of read and write heads can be configured based on the architecture of the memory network, using at least one read head and one write head. Training the memory network includes learning the memory write weights, read weights, and read vectors of the memory network. The memory state of the memory network can be defined as a combination (e.g., a cascade) of the read weights, write weights, read vector, and memory matrix. Depending on the architecture of the memory network, additional parameters can be defined and included in the memory state, including, but not limited to, usage vectors, priority weights, and link matrices.

[0020] Access mechanisms to external memory may include content-based addressing and position-based addressing, which may provide memory addresses or locations to store or retrieve information. The state of the controller (e.g., read and write weights) and other system parameters associated with memory access are trainable. The training of weights may take into account factors such as the current input (e.g., content addressing by similarity), previous focus (interpolation), possible focus shift (convolution shift), and blur (sharpening). The process of training the memory network initializes those weights and vectors, which are updated through a learning process until the error converges to, for example, a minimum acceptance value or another criterion.

[0021] The systems, methods, and techniques of the embodiments provide weight and / or vector initialization techniques that can allow for efficient and faster convergence, thereby allowing for more efficient and faster memory network model learning.

[0022] Embodiments of the system described herein initialize write weights, read weights, and / or read vectors with sparse values when training a memory network. Write weights or weight parameterize memory cells that store information. Read weights or weight parameterize memory cells that read information.

[0023] As an example, the system initializes write weights, where a single randomly selected weight for each write head is initialized to 1, and all other weights are initialized to 0. The system initializes all read weights, read vectors, and memory matrices to 0. Experiments performed with simulation parameters (e.g., 100 LSTM units, 128×20 memory matrix, training sequence lengths 1-20, batch size 1 (epoch number = sequence number), 1 read head, and 1 write head) show that approximately 80% of training attempts converge very quickly (e.g., 12-30 minutes on a Core i7-5600U central processing unit (CPU) at 2.6 GHz with AVX2 (SIMD) support in TensorFlow software).

[0024] In one embodiment, the system uses low-discrepancy sequences to generate this sparse initialization. A low-discrepancy sequence is a sequence with the following property: for all values of N, its subsequences x1, ..., xN have low discrepancy. Low-discrepancy sequences are also referred to as quasi-random or sub-random sequences. For a finite number of points, low-discrepancy sequences cover space more evenly than pseudo-random sequences. For example, low-discrepancy sequences have a more even distribution or a higher degree of equal distribution than pseudo-random sequences. Examples include, but are not limited to, Sobol, Halton, Hammersley, and van der Corput sequences.

[0025] For example, in an embodiment, the system selects or picks sparse initial values from a low-discrepancy sequence. Simulation experiments performed show that using the example simulation parameters of 100 LSTM units, a 127×20 memory matrix, training sequence lengths 1-20, batch size = 1 (thus, epoch number = sequence number), 1 read head and 1 write head, all training attempts (e.g., 15 runs) converged faster, for example, within 12-30 minutes. In this simulation experiment, a single write weight was pulled from a (40, 127) Sobol sequence initialized to 1, the other write weights were initialized to 0, and the read weights, read vectors, and memory matrix were all initialized to 0. It was observed from the experiments that the memory network trained on sequence lengths between 1 and 20 generalized well to previously unseen sequences of lengths 40, 80, and 120.

[0026] The following example illustrates one possible way to implement this method in one embodiment. This example is not meant to be limiting. Rather, it is provided by way of example and for illustrative purposes only. The example code is written in Python, but could also be written in any other programming language. The line numbers do not correspond to the actual line numbers in the code; they are provided only for ease of reference in the following description.

[0027]

[0028] In line 2, "seq_wrap" is a variable that represents the number of times the generated Sobol sequence is repeated (wrapped) to generate a longer sequence. Since the number and length of the generated Sobol sequences are limited, and the total length of the concatenated sequences is much smaller than the number of iterations used in the training of the neural network, one possible way to provide a larger sequence is to repeat the generated sequence. "iterations" represents the number of iterations used for training. "batch_size" represents the batch size used for training. "write_head" represents the number of write heads. "mem_words" represents the number of memory words (for example, in an N×M memory matrix, the size of N, where N represents the number of memory locations). "sobol_len" represents the number of Sobol sequences generated. Parameters, batch_size, write_head, and mem_words are configurable parameters in the memory network architecture.

[0029] In line 3, a Sobol sequence of number "sobol_len" and length "mem_words" is generated and stored in "mem_seed" as an array with "mem_words" rows and "sobol_len" columns. For example, mem_words = 127, sobol_len = 40, and mem_seed = [[64, 96, 32, 48, ..., 65, 1], [64, 32, 96, 48, ..., 21, 85], ..., [64, 32, 96, 80, ..., 43, 107]] is a 127×40 array, representing 40 Sobol sequences each of length 127 (i.e., a total of 127×40=5080 elements).

[0030] In line 4, the array stored in “mem_seed” is reshaped into an array with 1 row and “mem_words*Sobol_len” columns (equivalent to a row vector of length “mem_words*Sobol_len”), and the result is stored in “mem_seed”. For example, if before executing line 4, mem_seed = [[64, 96, 32, 48, …, 65, 1], [64, 32, 96, 48, …, 21, 85], …, [64, 32, 96, 80, …, 43, 107]], then after executing line 4, mem_seed = [64, 96, 32, 48, …, 65, 1, 64, 32, 96, 48, …, 21, 85, …, 64, 32, 96, 80, …, 43, 107].

[0031] In line 5, the array stored in "mem_seed" is repeated "seq_wrap" times along its first axis (e.g., rows), resulting in an array with 1 row and "mem_words*sobol_len*seq_wrap" columns, which is stored in "mem_seed". As an illustrative example, if seq_wrap = 2 (note that "seq_wrap" can be much larger than 2) and mem_seed = [64, 96, 32, 48, ..., 43, 107] before executing line 5, then after executing line 5, mem_seed = [64, 96, 32, 48, ..., 43, 107, 64, 96, 32, 48, ..., 43, 107].

[0032] In line 7, we start looping over the training iterations. “count” represents the current iteration (e.g., the current loop level starting from 1).

[0033] In line 9, "write_weightings" is a 3-dimensional array with sizes "batch_size," "mem_words," and "write_heads" for the first, second, and third dimensions, respectively, and represents write weighting. All elements of "write_weighting" are set to zero. "float32" represents a 32-bit floating point (also known as single precision) numerical format. Other numerical formats (including but not limited to 16-bit floating point) may also be used.

[0034] In lines 10-12, some elements of “write_weighting” are set to 1 while the rest remain 0, as determined by the sequence stored in “mem_seeds”, e.g., if for the first training iteration (e.g., count=1), mem_seeds=[64, 96, 32, 48, 112, 80, 16, 24, 88, 120, 56, 40, 104, 72, ..., 43, 107], batch_size=3 and write_heads=2:

[0035] write_weightings[0]

[63] [0]=1

[0036] write_weightings[0]

[95] [1]=1

[0037] write_weightings[1]

[31] [0]=1

[0038] write_weightings[1]

[47] [1]=1

[0039] write_weightings[2]

[111] [0]=1

[0040] write_weightings[2]

[79] [1]=1

[0041] The other elements of write_weightings remain zero. Similarly, for the second training iteration (e.g., count=2):

[0042] write_weightings[0]

[15] [0]=1

[0043] write_weightings[0]

[23] [1]=1

[0044] write_weightings[1]

[87] [0]=1

[0045] write_weightings[1]

[119] [1]=1

[0046] write_weightings[2]

[55] [0]=1

[0047] write_weightings[2]

[39] [1]=1

[0048] etc.

[0049] In line 13, "read_weightings" is a 3-dimensional array having sizes "batch_size", "mem_words", and "read_heads" for the first, second, and third dimensions, respectively, and represents read weighting. All elements of "read_weightings" are set to zero. "read_heads" represents the number of read heads.

[0050] In line 14, 'memory_matrix' is a 3-dimensional array having sizes 'batch_size', 'mem_words', and 'mem_width' for the first, second, and third dimensions, respectively, and represents a memory matrix. All elements of 'memory_matrix' are set to zero. 'mem_width' represents the memory width (e.g., in an N×M memory matrix, the size of M, the size of the vector at the memory location).

[0051] In line 15, "read_vectors" is a 3-dimensional array with sizes "batch_size", "mem_width", and "read_heads" for the first, second, and third dimensions, respectively, and represents a read vector. All elements of "read_vector" are set to zero.

[0052] Line 16 represents the following portion of the code (not shown) where training for each iteration is performed using the write weights, read weights, memory matrix, and read vector initialized as described above.

[0053] In the above example, the write weights are initialized to a sparse array that follows the pattern of the low-discrepancy sequence (in this case, the Sobol sequence). In an embodiment, both sparsity and low-discrepancy can be exploited. In another embodiment, sparsity and low-discrepancy can be implemented as two separate features. For example, if at line 9, the write weights are initialized to small random numbers instead of zero, and some elements of the write weights are set to 1, as described in lines 10-12, the resulting write weights may not be sparse, but still have the low-discrepancy characteristic because the write weights that are set to 1 follow the pattern of the low-discrepancy sequence.

[0054] In another example, as in line 9 above, the write weight is initialized to zero, but instead of setting some elements of the write weight to 1 based on a low-discrepancy sequence as in lines 10-12, the write weight is randomly set to 1, and the resulting write weight may not have the low-discrepancy characteristic, but is still sparse.

[0055] In the above example, in row 12, exactly one element of a given set of write weights is set to 1, in other examples a larger number of elements, for example 2 or more elements of a given set may be set to 1.

[0056] In the above example, in row 12, some elements of the write weight are set to 1. In other examples, these elements may be set to different non-zero values, such as 0.5, 2, or 5, or another value.

[0057] In the above example, the write weights are initialized to a sparse and low-discrepancy array, while the read weights and read vectors are initialized to zero. In another example, the read weights can be initialized to a sparse and low-discrepancy array, while the write weights and read vectors are initialized to zero. For example, in one embodiment, at least one of the write weights, read weights, and read vectors can be initialized to a sparse and / or low-discrepancy array.

[0058] In the above example, in rows 2-5, the low-discrepancy sequence generated is flattened (i.e., converted from a 2-dimensional array to a 1-dimensional array, i.e., a vector) and packed (i.e., repeated the desired number of times). Subsequently, the resulting sequence is used as a template for setting the element of the write weight to 1 as described above. In various embodiments, various other arrangements can be used. For example, a cascade of low-discrepancy sequences can also be used, wherein a given low-discrepancy sequence appears a random number of times at a random position within the cascaded low-discrepancy sequence of the desired length. In another example, a 1-dimensional array (i.e., a vector) of low-discrepancy sequences can be provided, such as the array generated in row 4. Then, at the beginning of each training iteration of the training iteration, a random element of the vector can be selected and assumed to be the starting point of the vector. The vector can be read from the starting point and used as a template for setting the write weight to 1 as described above. If the end of the vector is reached in the training iteration, the next element can be read from the beginning of the vector. In another example, a 2-dimensional array of low-discrepancy sequences can be provided, such as the array generated in row 3. Then, at the beginning of a training iteration (e.g., for each training iteration), a random sequence can be selected from the array and used as a template to set the write weight to 1 as described above. If the end of the sequence is reached, another random sequence can be selected from the array.

[0059] Figure 1 is a diagram illustrating components of a system in one embodiment. The components shown include, for example, computer-implemented components implemented and / or run on one or more hardware processors or coupled to one or more hardware processors. One or more hardware processors may, for example, include components such as programmable logic devices, microcontrollers, memory devices, and / or other hardware components that can be configured to perform the corresponding tasks described in this disclosure. The coupled memory devices can be configured to selectively store instructions that can be executed by one or more hardware processors. For example, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), another suitable processing component or device, or one or more combinations thereof. The processor can be coupled to a memory device. The memory device can include a random access memory (RAM), a read-only memory (ROM), or another memory device, and can store data and / or processor instructions for implementing various functions associated with the methods and / or systems described herein. The processor can execute computer instructions stored in the memory or received from another computer device or medium.

[0060] The memory network can include a controller 102, such as an artificial neural network, and a memory component 104, such as an external memory component of the neural network. The memory component is also referred to as a memory matrix. The controller 102 can access the memory 104 using read and write operations, which can be parameterized using weights (e.g., a write weight for a write operation and a read weight for a read operation). Training the memory network can include causing the memory network to learn memory write weights for writing to selected locations in the memory 104 and read weights for reading from selected locations in the memory 104. A read vector is returned as a combination of row vectors in the memory determined based on the read weights. That is, for each data instance (e.g., a training example), each read head generates M read vectors, where M is the width of the memory matrix.

[0061] For example, the computer processor shown at 106 may construct a memory network using at least memory write weights, memory read weights, and at least one read vector. Computer processor 106 may also initialize memory matrix 104. The memory write weights parameterize memory write operations of a neural network (e.g., 102) to memory matrix 104. The memory read weights parameterize memory read operations of a neural network (e.g., 102) from memory matrix 104. Computer processor 106 may initialize at least one of the write weights, read weights, or read vectors to be sparse or have a sparsity property. Computer processor 106 may train memory network 108 using at least one of the write weights, read weights, or read vectors initialized to have a sparse property. Training updates the write weights, read weights, and read vectors.

[0062] In one embodiment, the computer processor 106 may use a low-discrepancy sequence to generate a sparse set for application to at least one of the write weights, read weights, or elements of at least one read vector. Examples of low-discrepancy sequences include, but are not limited to, Sobol, Halton, Hammersley, and van der Corput sequences.

[0063] For example, in an embodiment, the computer processor 106 may set a sparse number of write weights to non-zero values and set the remaining write weights, read weights, and at least an element of the read vector to zero values to initialize at least one of the write weights, read weights, or read vectors to be sparse. In one embodiment, the sparse number of write weights set to non-zero values follows a pattern of a low-discrepancy sequence.

[0064] As another example, in an embodiment, the computer processor 106 may set a sparse number of read weights to a non-zero value and set the remaining read weights, write weights, and elements of at least one read vector to zero values to initialize at least one of the write weights, read weights, or read vectors to have sparsity. In one embodiment, the sparse number of read weights set to non-zero follows a pattern of low-discrepancy sequences.

[0065] Figure 2 A method for training a memory network according to one embodiment is provided. The method may be executed or run on a hardware processor, such as a computer processor. At 202, the method includes constructing a memory network using at least a memory write weight, a memory read weight, and at least one read vector. In one aspect, the memory write weight parameterizes a memory write operation of a neural network to an external memory component of the memory network; and the memory read weight parameterizes a memory read operation of a neural network from an external memory component of the memory network.

[0066] At 204, the method includes initializing at least one of the write weighting, read weighting, or read vector to have sparsity. For example, the method may select a write weighting, a read weighting, or a read vector. Then, the selection is initialized to have a sparse property. For example, in one embodiment, the elements of the write weighting, read weighting, and read vector may be set to a first value, such as "0." The selected one (e.g., write weighting, read weighting, or read vector) may be set to a sparse (small) number of its elements to a second value, such as "1." Values other than "0" and "1" may be used to represent a sparse property. In one embodiment, the method may use a low-discrepancy sequence to generate a sparse set of values.

[0067] As an example, the method can set a sparse number of write weights to non-zero values and set the remaining write weights, read weights, and elements of at least one read vector to zero values. In one embodiment, the sparse number of write weights set to non-zero values follows a pattern of low-discrepancy sequences.

[0068] As another example, the method can set a sparse number of read weights to a non-zero value and set the remaining read weights, write weights, and elements of at least one read vector to have zero values. In one embodiment, the sparse number of read weights set to non-zero follows a pattern of low-discrepancy sequences.

[0069] As yet another example, the method can set a sparse number of elements of the read vector to non-zero values and set the remaining elements of the read vector, the read weight, and the write weight to have zero values. In one embodiment, the sparse number of elements of the read vector that are set to non-zero follow a pattern of a low-discrepancy sequence.

[0070] In one embodiment, the sparsity number is 1, in another embodiment, the sparsity number is greater than 1, and may be less than a configurable number.

[0071] At 206, the method includes training the memory network using at least one of a write weight, a read weight, or a read vector initialized to have a sparse property. The training updates the write weight, the read weight, and the at least one read vector. The write weight and the read weight can be considered as connection weights between the neural network and the external memory. In one aspect, the memory network can be trained using gradient descent.

[0072] Figure 3 is a flow chart illustrating a method in another embodiment. The method may be executed or run on a hardware processor, such as a computer processor. At 302, the method may include constructing a memory network using at least a memory write weight, a memory read weight, and at least one read vector. The memory write weight parameterizes a neural network's memory write operations to an external memory component of the memory network. The memory read weight parameterizes a neural network's memory read operations from an external memory component of the memory network.

[0073] At 304, the method includes initializing at least one selected from the group of at least a write weight, a read weight, and at least one read vector to a configuration value that follows a low-disparity sequence (e.g., initialized by a low-disparity sampling pattern). For example, the method includes selecting a write weight, a read weight, or a read vector. The method can set the selected elements to configuration values such as "1", which follow a low-disparity sequence. Initialization can also include setting the elements of the write weight, read weight, and read vector that are not selected for the low-disparity sequence to values different from the configuration value. As an example, initialization can include setting the elements of the write weight, read weight, and read vector that are not selected to follow the low-disparity sequence to have a zero value. As another example, initialization can include setting the elements of the write weight, read weight, and read vector that are not selected to follow the low-disparity sequence to random numbers that are different from the configuration value.

[0074] At 306, the method includes training the memory network using initialized write weights, read weights, and read vectors, wherein at least one of the write weights, read weights, or read vectors is set to include a configuration value that follows a low-discrepancy sequence. Training updates the write weights, read weights, and at least one read vector. The write weights and read weights can be considered connection weights between the neural network and the external memory. In one aspect, the memory network can be trained via gradient descent.

[0075] Figure 4An example of generating a Sobol sequence using the Python (computer programming language) interpreter is shown as an example method for generating low-discrepancy sequences. Commands in this language are shown preceded by ">>>" and followed by the output generated from the executed command. The first command "import sobol_seq" imports the available Sobol sequence package. The second command outputs a length of 2 7 The matrix of 40 Sobol sequences with -1=127 is Sobol(40,127). The third command outputs the first column of the matrix output by the second command.

[0076] Figure 5is a diagram illustrating components of a system for training a memory network in one embodiment. One or more hardware processors 502, such as a central processing unit (CPU), a graphics processing unit (GPU), and / or a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or another processor, can be coupled to a memory device 504 and design and train a memory network. The memory device 504 may include a random access memory (RAM), a read-only memory (ROM), or another memory device and may store data and / or processor instructions for implementing various functions associated with the methods and / or systems described herein. The one or more processors 502 can execute computer instructions stored in the memory 604 or received from another computer device or medium. The memory device 504 may, for example, store instructions and / or data for the functions of the one or more hardware processors 502 and may include an operating system and other programs for instructions and / or data. The one or more hardware processors 502 can construct a memory network and initialize the parameters of the memory network. For example, the one or more hardware processors 502 can initialize at least one of the write weights, read weights, or read vectors to have sparsity. On the other hand, one or more hardware processors can initialize at least one of the write weight, read weight, or read vector to a configuration value that follows a low-discrepancy sequence. On the other hand, one or more hardware processors can initialize at least one of the write weight, read weight, or read vector to have sparsity, and also follow a low-discrepancy sequence when setting the sparse value. Input or training data can be stored in the memory device 504, or received from a remote device via the network interface 508, and can be temporarily loaded into the memory device 504 for building or generating a trained memory network. In one aspect, the generated or trained memory network can be stored on the memory device 504, for example, for execution by one or more hardware processors 502 to perform a task. One or more hardware processors 502 can be coupled to an interface device, such as a network interface 508 for communicating with a remote system, for example, via a network, and an input / output interface 510 for communicating with input and / or output devices such as a keyboard, mouse, display, and / or other devices.

[0077] Figure 6 A schematic diagram of an example computer or processing system in which the system may be implemented in one embodiment is shown. The computer system is only one example of a suitable processing system and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the method described herein. The processing system shown is operational with numerous other general purpose or special purpose computing system environments or configurations. Figure 6Examples of well-known computing systems, environments, and / or configurations for use with the illustrated processing system may include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe systems, and distributed cloud computing environments that include any of the above systems or devices.

[0078] Computer systems may be described in the general context of computer system-executable instructions, such as program modules, executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer systems may be practiced in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0079] The components of the computer system may include, but are not limited to, one or more processors or processing units 12, a system memory 16, and a bus 14 that couples various system components, including the system memory 16, to the processor 12. The processor 12 may include a module 30 for performing the methods described herein. The module 30 may be programmed into the integrated circuits of the processor 12 or loaded from the memory 16, the storage device 18, or the network 24, or a combination thereof.

[0080] Bus 14 may represent one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, such as, by way of example and not limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0081] The computer system may include a variety of computer system readable media. Such media can be any available media that can be accessed by the computer system, and it can include volatile and non-volatile media, removable and non-removable media.

[0082] The system memory 16 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory or other. The computer system may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 18 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (e.g., a "hard drive"). Although not shown, a disk drive for reading from and writing to a removable, non-volatile disk (e.g., a "floppy disk") may be provided, as well as an optical drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM, or other optical media. In this case, each may be connected to the bus 14 via one or more data medium interfaces.

[0083] The computer system may also communicate with one or more external devices 26, such as a keyboard, pointing device, display 28, or the like; one or more devices that enable a user to interact with the computer system; and / or any device that enables the computer system to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via input / output (I / O) interface 20.

[0084] In addition, the computer system can communicate with one or more networks 24, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via a network adapter 22. As shown, the network adapter 22 communicates with the other components of the computer system via bus 14. It should be understood that, although not shown, other hardware and / or software components may also be used in conjunction with the computer system. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0085] It is understood in advance that although the present disclosure may include descriptions regarding cloud computing, implementation of the teachings set forth herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed. Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0086] Features are as follows:

[0087] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities, such as server time and network storage, as needed without manual interaction with the service provider.

[0088] Wide Area Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0089] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location independence, as consumers typically do not control or know the exact location of the provided resources, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0090] Rapid elasticity: In some cases, the ability to scale out quickly and in quickly can be provided quickly and elastically. To the consumer, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0091] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the provider and consumer of the utilized service.

[0092] The service model is as follows:

[0093] Software as a Service (SaaS): The ability provided to consumers is to use the provider's applications running on a cloud infrastructure. Applications are accessed from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0094] Platform as a Service (PaaS): The capability provided to consumers is to deploy consumer-created or acquired applications onto cloud infrastructure. These applications are built using programming languages and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but do have control over the deployed applications and possibly the configuration of the application hosting environment.

[0095] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but do have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0096] The deployment model is as follows:

[0097] Private cloud: The cloud infrastructure is operated solely for the organization. It can be managed by the organization or a third party and can exist inside or outside the building.

[0098] Community cloud: Cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0099] Public cloud: Cloud infrastructure is available to the general public or large industrial groups and is owned by the organization that sells cloud services.

[0100] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a unique entity but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0101] The cloud computing environment is service-oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure consisting of a network of interconnected nodes.

[0102] Now refer to Figure 7 , depicts an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which a local computing device used by a cloud consumer can communicate, such as a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service for which the cloud consumer does not need to maintain resources on a local computing device. It should be understood that Figure 7The types of computing devices 54A-54N shown in FIG are intended for illustration only, and computing node 10 and cloud computing environment 50 may communicate with any type of computing device over any type of network and / or network-addressable connection (eg, using a web browser).

[0103] Now refer to Figure 8 , showing the cloud computing environment 50 ( Figure 7 ) provides a set of functional abstraction layers. It should be understood in advance that Figure 8 The components, layers, and functions shown in are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0104] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: host 61; server 62 based on RISC (Reduced Instruction Set Computer) architecture; server 63; blade server 64; storage device 65; and network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0105] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71 ; virtual storage 72 ; virtual networks 73 , including virtual private networks; virtual applications and operating systems 74 ; and virtual clients 75 .

[0106] In one example, the management layer 80 may provide the functionality described below. Resource provisioning 81 provides for dynamic procurement of computing and other resources for performing tasks within a cloud computing environment. Metering and pricing 82 provides for cost tracking when utilizing resources in a cloud computing environment, as well as billing or invoicing for the consumption of those resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. A user portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides for allocation and management of cloud computing resources so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides for pre-scheduling and procurement of cloud computing resources, where future demand is anticipated based on the SLA.

[0107] The workload layer 90 provides examples of functionality that can take advantage of a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and memory network training processing 96.

[0108] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0109] A computer-readable storage medium can be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination thereof. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0110] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0111] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages (including object-oriented programming languages, such as Smalltalk, C++, etc.) and procedural programming languages (such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In some embodiments, in order to perform various aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0112] Aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowcharts and / or block diagrams and the combination of blocks in the flowcharts and / or block diagrams can be implemented by computer-readable program instructions.

[0113] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can direct the computer, programmable data processing device and / or other equipment to operate in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0114] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0115] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for realizing the specified logical function. In some alternative embodiments, the functions noted in the frame may not occur in the order noted in the figure. For example, the two frames shown in succession can actually be implemented as a step, simultaneously, substantially simultaneously, in a manner that overlaps part or all of the time, or these frames can sometimes be performed in reverse order, depending on the functions involved. It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0116] The terms used herein are only for the purpose of describing specific embodiments, rather than for limiting the present invention. As used herein, the singular forms "one", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. As used herein, the term "or" is an inclusive operator and can represent "and / or", unless the context clearly or clearly indicates otherwise. It will also be understood that the terms "comprise", "include" and / or "have" can specify the existence of stated features, integers, steps, operations, elements and / or components when used in this article, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. As used herein, the phrase "in an embodiment" does not necessarily refer to the same embodiment, although it can refer to the same embodiment. As used herein, the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it can refer to the same embodiment. As used herein, the phrase "in another embodiment" does not necessarily refer to different embodiments, although it can refer to different embodiments. In addition, the parts of the embodiments and / or embodiments can be freely combined with each other unless they are mutually exclusive.

[0117] The corresponding structure, material, action and equivalent (if any) of all means or step plus function elements in the following claims are intended to include any structure, material or action for performing a function in combination with other elements claimed as specifically claimed. A description of the present invention has been given for the purpose of illustration and description, but this description is not exhaustive or limits the present invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the invention. The embodiments are selected and described in order to best explain the principles of the present invention and practical applications, and to enable other persons of ordinary skill in the art to understand the various embodiments of the present invention with various modifications that are suitable for the specific purposes envisioned.

Claims

1. A computer-implemented method for training a memory network, comprising: constructing a memory network using at least a memory write weight, a memory read weight, and at least one read vector, wherein the memory write weight parameterizes a memory write operation of a neural network to an external memory component of the memory network, and the memory read weight parameterizes a memory read operation of the neural network from the external memory component of the memory network; initializing at least one selected from the group consisting of at least the write weight, the read weight, and an element of the at least one read vector to have sparsity; as well as The memory network is trained using the at least one element selected from the group of elements including at least the write weight, the read weight, and the at least one read vector that is initialized to have a sparse property, and the training updates the write weight, the read weight, and the at least one read vector.

2. The method according to claim 1, wherein Initializing the one selected from the group having the write weights, the read weights, and elements of the at least one read vector to have sparsity includes setting a sparse number of the write weights to non-zero values and setting the remaining write weights, the read weights, and elements of the at least one read vector to zero values. 3 . The method of claim 2 , wherein the sparse number of the write weights set to non-zero values follow a pattern of low-discrepancy sequences.

4. The method according to claim 1, wherein Initializing the one selected from the group having the write weight, the read weight, and the elements of the at least one read vector to have sparsity includes setting a sparse number of read weights to non-zero values and setting the remaining read weights, the write weight, and the elements of the at least one read vector to have zero values.

5. The method according to claim 4, wherein The read weights for the sparse number set to non-zero follow the pattern of low-discrepancy sequences.

6. The method according to claim 1, wherein Initializing the one selected from the group having the write weight, the read weight, and the elements of the at least one read vector to have sparsity includes setting a sparse number of elements of the at least one read vector to non-zero values and setting the remaining elements of the at least one read vector, the read weight, and the write weight to have zero values. 7 . The method of claim 6 , wherein the sparse number of elements of the at least one read vector that are set to non-zero follow a pattern of a low-discrepancy sequence. 8 . The method of claim 1 , further comprising using a low-discrepancy sequence to generate a sparse set having the at least one selected from the group of the write weight, the read weight, and elements of the at least one read vector.

9. The method according to any one of claims 2 to 7, wherein the sparse number is one.

10. The method according to any one of claims 2 to 7, wherein the sparsity number is greater than one.

11. A computer-implemented method for training a memory network, comprising: constructing a memory network using at least a memory write weight, a memory read weight, and at least one read vector, wherein the memory write weight parameterizes a memory write operation of a neural network to an external memory component of the memory network, and the memory read weight parameterizes a memory read operation of the neural network from the external memory component of the memory network; initializing at least one selected from the group of elements having at least the write weight, the read weight, and the at least one read vector to a configuration value that follows a low-discrepancy sequence; as well as The memory network is trained using the at least one selected from the group of elements having at least the write weight, the read weight, and the at least one read vector having an initial configuration value that follows a low-discrepancy sequence, the training updating the write weight, the read weight, and the at least one read vector. 12 . The method of claim 11 , wherein the initializing further comprises setting the write weights, the read weights, and elements of the at least one read vector that are not selected to follow the low-disparity sequence to have values different from the configured values. 13 . The method of claim 11 , wherein the initializing further comprises setting elements of the write weights, the read weights, and the at least one read vector that are not selected to follow the low-disparity sequence to have a value of zero. 14 . The method of claim 11 , wherein the initializing further comprises setting elements of the write weight, the read weight, and the at least one read vector that are not selected to follow the low-disparity sequence to random numbers having different values than the configured values.

15. A system for training a memory network, comprising: Hardware processor; a memory device coupled to the hardware processor, the memory device storing a memory matrix; The hardware processor is configured to: A memory network is constructed using at least a memory write weight, a memory read weight, and at least one read vector, wherein the memory write weight parameterizes a memory write operation of a neural network to the memory matrix, and the memory read weight parameterizes a memory read operation of the neural network from the memory matrix; initializing at least one selected from the group consisting of at least the write weight, the read weight, and an element of the at least one read vector to have sparsity; as well as The memory network is trained using the at least one selected from the group consisting of at least the write weight, the read weight, and the at least one read vector initialized to have a sparse property, and the training updates the write weight, the read weight, and the at least one read vector.

16. The system according to claim 15, wherein: The hardware processor is configured to set a sparse number of the write weights to non-zero values and set the remaining write weights, the read weights, and the elements of the at least one read vector to zero values to initialize the one selected from the group having the write weights, the read weights, and the elements of the at least one read vector to have sparsity.

17. The system of claim 16, wherein the sparse number of write weights set to have non-zero values follows a pattern of low-discrepancy sequences.

18. The system according to claim 15, wherein: The hardware processor is configured to set a sparse number of read weights to non-zero values and set the remaining read weights, the write weights, and elements of the at least one read vector to have zero values to initialize the one selected from the group having the write weights, the read weights, and elements of the at least one read vector to have sparsity.

19. The system according to claim 18, wherein: The weights of reads with a sparse number set to non-zero follow the pattern of low-discrepancy sequences.

20. The system of claim 15, wherein the hardware processor is configured to generate the sparse set of the at least one selected from the group consisting of the write weight, the read weight, and elements of the at least one read vector using a low-discrepancy sequence.

21. A system for training a memory network, comprising: Hardware processor; a memory device coupled to the hardware processor, the memory device storing a memory matrix; The hardware processor is configured to at least: constructing a memory network using at least a memory write weight, a memory read weight, and at least one read vector, wherein the memory write weight parameterizes a memory write operation of a neural network to an external memory component of the memory network, and the memory read weight parameterizes a memory read operation of the neural network from the external memory component of the memory network; initializing at least one selected from the group of elements having at least the write weight, the read weight, and the at least one read vector to a configuration value that follows a low-discrepancy sequence; as well as The memory network is trained with the at least one selected from the group of elements having at least the write weight, the read weight, and at least one read vector having initial configuration values following a low-discrepancy sequence, the training updating the write weight, the read weight, and the at least one read vector.

Images