A method and system for implementing a memory network based on imprinting precipitation

Abstract

The present invention discloses a method and system for implementing a memory network based on imprint precipitation. First, a memory network model including imprint units is constructed. The memory network model at least includes a feature extraction network, a mapping network, a group of imprint units, and a memory judgment network. The feature extraction network is used to extract the features of the input image, and the same-category features are clustered by the idea of exchange. Then, the features are input into a hash network, and the mapping relationship between the features and the imprint units is established through the hash network. The precipitation function is used to store the features in the imprint units, and at the same time, a data set is constructed using the feature sequence and the imprint sequence for training the subsequent stress function. Finally, the stress function network is trained using the made data set and the trained model is used for memory. The method of the present invention targets part of the imprint units with the extracted features and performs persistent memory, and uses a siamese network to judge the distribution similarity of the feature sequence and the imprint sequence to answer whether the input is in memory, realizing the memory judgment process.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and particularly relates to a method and system for implementing a memory network based on imprint precipitation. Background Art

[0002] An artificial neural network is a computational model aimed at simulating the human brain's nervous system. It constructs a complex structure through a large number of nodes and connections, enabling it to learn and extract patterns from input data. Compared with traditional computational models, neural networks have become a widely used technology with their high computational power, plasticity, and adaptability. They have achieved great success particularly in fields such as image recognition, speech recognition, and natural language processing. Their development has provided us with new perspectives and opportunities to understand the nature of memory.

[0003] Existing artificial neural networks contain various memory modules, but most of their designs aim to capture global information and are not similar to memories such as storing a certain picture or a certain audio, such as RNN and LSTM. The most relevant and outstanding current solution to memory like a human is to couple external memory resources with the neural network, and the neural network will interact with the external memory resources in the form of attention, such as NTM. This novel and flexible method has led to extensive divergent research and has achieved some success in the neural network's simulation of human cognition, but its memory method is contrary to the mainstream ideas of modern biology.

[0004] The external memory uses individual memory cells to store memories, and the memories are independent of each other without interaction. In addition, memory and association are highly closely related, and the mechanism of association should be based on memory. However, mutually independent memories are not a very suitable structure. Therefore, how to implement a memory mechanism similar to that of humans and enable the network to answer whether the input is in memory is still a challenging problem. Summary of the Invention

[0005] The present invention precisely aims at the problems of low memory judgment efficiency and an inapplicable memory structure for association research in the prior art, and provides a method and system for implementing a memory network based on imprint precipitation. First, a memory network model including imprint units is constructed. The memory network model at least includes a feature extraction network, a mapping network, a group of imprint units, and a memory judgment network. The feature extraction network extracts the features of the input picture and clusters the features of the same category using the idea of exchange. Then, the features are input into a hash network, and a mapping relationship between the features and the imprint units is established through the hash network. A precipitation function is used to store the features in the imprint units, and at the same time, a data set is constructed using the feature sequence and the imprint sequence for training the subsequent stress function. Finally, the stress function network is trained using the made data set and the trained model is used for memory. The method of the present invention targets part of the imprint units with the extracted features and performs persistent memory, and uses a siamese network to judge the distribution similarity between the feature sequence and the imprint sequence to answer whether the input is in memory, realizing the memory judgment process.

[0006] To achieve the above object, the technical solution adopted by the present invention is: A method for implementing a memory network based on imprint precipitation, including the following steps:

[0007] S1: Construct a memory network model including imprint units. The memory network model at least includes a feature extraction network, a mapping network, a group of imprint units, and a memory judgment network. The feature extraction network filters the features of the input picture to obtain a high-dimensional abstract representation of the picture. The mapping network establishes a one-way mapping relationship between the picture features and the imprint units. The group of imprint units simulates the human brain neuron group and is the main carrier of memory. The memory judgment network judges whether the input picture is in memory or judges whether the input is highly correlated with the content of memory.

[0008] S2: Use the feature extraction network to extract the features of the input picture and cluster the features of the same category using the idea of exchange.

[0009] S3: Input the features obtained in step S2 into a hash network, and establish a mapping relationship between the features and the imprint units through the hash network.

[0010] S4: Use a precipitation function to store the features in the imprint units, and at the same time, construct a data set using the feature sequence and the imprint sequence for training the subsequent stress function.

[0011] S5: Use the made data set to train the stress function network and use the trained model for memory.

[0012] As an improvement of the present invention, in step S2, the clustering is specifically: Input a pair of pictures at the same time, extract the features respectively, then exchange some features at the same position of the two and then perform reconstruction. The changed model loss function is:

[0013]

[0014] where x i is the input image, z i is the extracted feature, g(x) represents the decoder for image reconstruction, and L rec represents the reconstruction loss function, where L rec (x i , g(z i )) is the reconstruction loss of the original image, and L rec (x i , g(z′ i )) is the reconstruction loss after feature swapping.

[0015] As an improvement of the present invention, in step S2, in addition to two pictures of the same category, an additional picture of a different category needs to be introduced, and the same feature extraction steps and reconstruction work are performed. The segmentation operation is performed but does not participate in the swapping, and the three form a triple relationship: (anchor, positive, negative), where anchor and positive are pictures of the same category, and negative is a picture of a different category. The triple loss function used is:

[0016]

[0017] where ‖*‖ is the Euclidean distance, represents the Euclidean distance metric between positive and anchor, represents the Euclidean distance metric between negative and anchor, and α refers to the distance between x a and x n and the distance between x a and x p has a minimum interval. + means that when the value in [*] is greater than 0, the value is taken as the loss, and when it is less than 0, the loss is taken as 0.

[0018] As another improvement of the present invention, for the hash network in step S3, only the features of the swapped part are taken as the input features, and normalization processing is performed before output. The output dimension is the number of imprinting units responsible for memorizing one picture; the specific calculation method of the mapping is:

[0019]

[0020] where n i is the subscript of the corresponding imprinting unit, z i is the feature, f map represents the hash network, and N represents the total number of imprinting units.

[0021] As another improvement of the present invention, the precipitation function in the step S4 is specifically:

[0022]

[0023] In the formula, a and k are respectively used to control the single - time enhancement degree and the imprint precipitation saturation level;

[0024] The emergency function is implemented by a siamese network, which is used to judge whether the input is in memory. The judgment basis is the feature extracted from the input picture and the distribution of the corresponding imprinting units.

[0025] As a further improvement of the present invention, during the memory process of the siamese network, the feature sequence z = {z 1 , z 2 ,..., z n} extracted from the picture and the corresponding imprint precipitation sequence u = {u 1 , u 2 ,..., u n} are obtained. Before memory, z and u are not similar, and they are labeled as "unseen" and combined together to form a negative sample for training:

[0026] [{z 1 , z 2 ,..., z n}, {u 1 , u 2 ,..., u n}, unseen]

[0027] Subsequently, after storing the current memory through the precipitation function, z and u are similar. At this time, they are labeled as "seen" and combined together to form a positive sample for training:

[0028] [{z 1 , z 2 ,..., z n}, {u 1 , u 2 ,..., u n}, seen]

[0029] After completing one memory, the changed imprinting units may affect the pictures that have been memorized previously because the imprint sequence corresponding to the previous pictures may have changed. The pictures affected by the changed imprinting units are "reviewed", and their feature sequences and the current corresponding imprint sequences are labeled as "seen" again and added to the training samples:

[0030] [{z 1 , z 2 ,..., z n}, {u 1,u 2 ,...,u n} new , seen].

[0031] To achieve the above object, the technical solution adopted by the present invention is also: a memory network implementation system based on imprint precipitation, including a computer program, and when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0032] Compared with the prior art, the technical advantages and technical effects of the present invention are:

[0033] (1) During feature extraction, memory pictures are input into the encoder in pairs to extract features. After exchanging some features at the same positions of the two, reconstruction is performed, forcing the network to extract the common features of the two pictures into the exchanged feature part to obtain picture semantic information and achieve feature clustering. This exchange idea is novel, easy to operate, and has a significant clustering effect, and can be extended to tasks in other similar scenarios as a general method;

[0034] (2) The hash network establishes a mapping relationship between features and imprint units. The hash network realizes linear transformation and can comprehensively consider the extracted features and feature order to achieve one-way mapping. In practical applications, this hash method can support the network to complete a memory judgment within the time complexity of O(1), and the calculation efficiency is better than the existing attention mechanism method;

[0035] (3) Using a group of imprint units to simulate human brain neurons and serving as the carrier of memory not only has better biological interpretability, but also this memory structure can solve the problem that memories in the existing structure are independent of each other, and can provide a feasible structural basis for further associative research in the future;

[0036] (4) Using a siamese network for memory judgment, the dataset required for training the network itself can be automatically constructed by the memory process, making the training of the siamese network closer to unsupervised, and the network itself is not restricted to a specific similarity judgment function and is more flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flowchart of the steps of a method for implementing a memory network based on imprint precipitation according to the present invention;

[0038] Figure 2 It is a schematic diagram of extracting picture features using the exchange idea according to the present invention;

[0039] Figure 3 It is a schematic diagram of establishing a mapping relationship using a hash network according to the present invention;

[0040] Figure 4 It is a schematic diagram of implementing a stress mechanism based on a siamese network according to the present invention. Detailed implementation manners

[0041] The present invention will be further clarified below in conjunction with the accompanying drawings and specific implementation manners. It should be understood that the following specific implementation manners are only used to illustrate the present invention and not to limit the scope of the present invention.

[0042] Embodiment 1

[0043] A method for implementing a memory network based on imprinting precipitation, as Figure 1 shown, includes the following steps:

[0044] Step S1: Construct a memory network model including imprinting units; the memory network model includes: a feature extraction network, a mapping network, a group of imprinting units, and a memory judgment network;

[0045] For the feature extraction network, because it involves image reconstruction, the structure of an autoencoder is used, and the encoder part is used to implement feature extraction; the mapping network needs to implement a one-way linear mapping, and its input dimension and output dimension are the same, which is determined by the extracted feature dimension. Before the final output, normalization is performed to facilitate subsequent mapping to the imprinting units. The output dimension means how many imprinting units are used to be responsible for the memory of a picture; the group of imprinting units is a group of persistent media, which is defined to simulate the imprinting cell clusters in the human brain. For the convenience of mapping and observation, it can be regarded as a one-dimensional floating-point array with a fixed length, which is the storage carrier of memory; the memory judgment network acts as a similarity judgment. By analyzing the distribution similarity between the feature sequence and the imprinting sequence, it judges whether the input is in the memory. Using a siamese network can avoid manually selecting a specific similarity calculation formula and can handle the case of different input dimensions.

[0046] Step S2: Use the feature extraction network to extract the features of the input picture, and cluster the features of the same category with the idea of exchange; specifically as shown in Figure 2.

[0047] Figure 2 It is a schematic diagram for extracting the features of the picture with the idea of exchange; the encoder adopted in this embodiment is a custom three-layer convolutional neural network, and the decoder is a symmetric three-layer convolutional neural network. The input image comes from the CIFAR10 dataset, and the image size is 32×32 pixels. After the convolutional calculation by the encoder, the size of the feature map obtained is (256, 4, 4). In this embodiment, the division of public features and private features is 1:1, and it is artificially specified that the upper half is used for exchange. Therefore, the division is as follows:

[0048]

[0049] In the formula, the first dimension is the sample sequence, the second dimension is the channel, and the third and fourth dimensions are the height and width respectively; each convolutional kernel is a different feature extraction method, corresponding to a channel of the feature map. Dividing on the channel means that the network will let this part of the convolutional kernels focus on the similar features in the two pictures, and then let the remaining convolutional kernels process the dissimilar feature parts.

[0050] After the feature segmentation, exchange, and recombination, there are currently four feature maps, two original feature maps and two spliced feature maps, all with dimensions of (256, 4, 4). Using the decoder to perform the reconstruction task and choosing the loss as the mean square error MSE, the loss function is:

[0051]

[0052] The formula includes the reconstruction loss of the original image and the reconstruction loss after feature exchange. x i is the input image, z i is the extracted feature, and g(x) represents the decoder, which is the process of image reconstruction.

[0053] In the actual process, the neural network will slack off during training. The goal of this method is to force the network to extract the common features to the upper half of the feature map. The ideal state is that the upper halves of the feature maps of the two pictures are exactly the same, so that the exchange will not have a bad impact. However, in the actual process, the network will set the upper half to 0 to achieve the purpose of making the upper half exactly the same, and only use the lower half to extract features and perform reconstruction, so that the method fails and the network degenerates into an ordinary autoencoder network. One of the reasons is that the current reconstruction task is relatively simple. To address this issue, this method adds a new loss term to constrain the lazy behavior of the network.

[0054] Based on the above steps, in addition to two pictures of the same category, an additional picture of a different category needs to be introduced. Perform the same feature extraction steps and reconstruction work, and perform the segmentation operation but do not participate in the exchange. The purpose is to distinguish it from the pictures of the same category. The three form the following triple relationship:

[0055] (anchor, positive, negative)

[0056] In the formula, anchor and positive are pictures of the same category, and negative is a picture of a different category. The learning objective is to make the distance between positive and anchor as small as possible, and the distance between negative and anchor as large as possible. The triple loss function used in this example is:

[0057]

[0058] In the formula, ‖*‖ is the Euclidean distance, so It represents the Euclidean distance metric between positive and anchor. It represents the Euclidean distance metric between negative and anchor, and α refers to x a With x n The distance between a With x p There is a minimum interval between the distances. + means that when the value in [*] is greater than 0, the value is taken as the loss, and when it is less than 0, the loss is 0.

[0059] Therefore, the loss function finally used in this embodiment is:

[0060]

[0061] Where m and n are used to control the reconstruction loss L rec and the ternary loss L triplet The weight ratio of .

[0062] Step S3: Establish the mapping relationship between features and imprint units through the hash network, as follows Figure 3 As shown, Figure 3 Schematic diagram of establishing mapping relationships using a hash network.

[0063] The hash network is essentially a hash function implemented by an artificial neural network, and its purpose is to establish a mapping relationship between input features and imprint units. The input features only take common features, that is, the features of the exchanged part. In this embodiment, the dimension of the common features is (128, 4, 4). If it is processed into a one-dimensional vector, the dimension is 2048, which means that the input and output dimensions of the hash network are both 2048. To put it more concretely, the memory of each picture is responsible for 2048 imprint units. The hash network only plays a mapping role and does not require training itself. The hash network used in this embodiment is a three-layer fully connected network, and the parameter initialization obeys the standard normal distribution with an expectation of 0 and a standard deviation of 0.03.

[0064] The hash network will perform normalization before the final output. The purpose of normalization is to limit the data to [0,1] to facilitate subsequent mapping. The mapping calculation method is:

[0065]

[0066] Where n i is the subscript of the corresponding imprint unit, z i As the characteristic, f map represents the hash network, and N represents the total number of imprint units, which is 100,000 in this embodiment.

[0067] Step S4: Use the precipitation function to store the features in the imprinting unit, and at the same time, construct a data set using the feature sequence and the imprinting sequence for training the subsequent stress function;

[0068] The precipitation function is responsible for persisting the changed mapped imprinting unit, that is, the precipitation function is responsible for storing the feature values in the corresponding imprinting unit. A single imprinting unit may be changed several times, but its value should not expand infinitely and there should be upper and lower bounds. For this, the precipitation function is:

[0069]

[0070] In the formula, a and k are used to control the degree of single - time enhancement and the imprinting precipitation saturation level respectively. In this embodiment, a takes the value of 1 and k takes the value of 0.5.

[0071] The stress function is used to determine whether the input is in memory. The basis for judgment is the distribution of the features extracted from the input picture and the corresponding imprinting unit. Therefore, the stress function essentially makes a similarity judgment. In the present invention, a siamese network is used to implement the stress function, and its advantage is that it is not limited to a specific similarity calculation formula and can handle the situation of different input dimensions.

[0072] Furthermore, the training data of the siamese network can be obtained from the memory process, which is a supervised learning process. In the memory process, the feature sequence z = {z 1 ,z 2 ,...,z n} extracted from the picture and the corresponding imprinting precipitation sequence u = {u 1 ,u 2 ,...,u n} can be obtained. When constructing the data set, the memory of a single picture is divided into three parts: before memory, after memory, and the impact caused. Before memory, z and u are not similar, and they are labeled as "not seen" and combined together to form a negative sample for training:

[0073] [{z 1 ,z 2 ,...,z n},{u 1 ,u 2 ,...,u n}, not seen]

[0074] Subsequently, after storing the current memory via the precipitation function, z and u are similar. At this time, they are labeled as "seen" and combined together to form a positive sample for training:

[0075] [{z 1 ,z 2 ,...,z n},{u 1,u 2 ,...,u n}, seen]

[0076] After completing a memory, the changed imprinting units may affect the previously memorized pictures because the imprinting sequences corresponding to the previous pictures may have changed. Therefore, in addition to the training samples generated by the above method, there should be a recollection "consolidation" process after each memory, that is, "review" the pictures affected by the changed imprinting units, re-label the feature sequences and the current corresponding imprinting sequences with "seen" and add them to the training samples:

[0077] [{z 1 ,z 2 ,...,z n ,{u 1 ,u 2 ,...,u n} new , seen]

[0078] Step S5: Train the stress function network using the made dataset and test the memory effect using the trained model.

[0079] Figure 4 This is a schematic diagram of the stress mechanism implemented by the present invention based on the Siamese network. The stress mechanism refers to judging whether the input picture is in memory by judging the distribution similarity between the input features and the imprinting units. In this embodiment, the input of the Siamese network is the relationship pair of (z, u). After passing through three fully connected networks respectively in the forward propagation forward, the two are subtracted and then the absolute value is taken, and then calculated through two fully connected networks. Finally, a two-dimensional output is obtained to represent the judgment result. Use the made dataset above to train the Siamese network, divide it into a training set and a test set according to the ratio of 8:2, and the loss function is the commonly used cross-entropy loss function CrossEntropyLoss.

[0080] When the model is trained, input any image to the network, obtain the corresponding imprinting sequence through feature extraction and mapping, and hand it over to the Siamese network for judgment. If the network outputs 1, it means it is in memory. If the output is 0, it means it is a strange memory.

[0081] A method for implementing a memory network based on imprint precipitation disclosed by the present invention extracts the common semantic features of pictures through the idea of exchange to achieve feature clustering; uses a hash network for mapping to achieve memory judgment within constant time; by proposing the concept of imprinting units as the carrier of memory, it is more in line with the current mainstream view of memory, has better biological interpretability, and at the same time provides an effective structural basis for subsequent research on association.

[0082] It should be noted that the above content only illustrates the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. For those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements all fall within the protection scope of the claims of the present invention.

Claims

1. A memory network implementation method based on imprint precipitation, characterized in that , including the following steps: S1: Construct a memory network model including engram units, wherein the memory network model at least includes a feature extraction network, a mapping network, an engram unit group and a memory judgment network; the feature extraction network filters input image features to obtain a high-dimensional abstract representation of the image; the mapping network establishes a one-way mapping relationship between image features and engram units; the engram unit group simulates a human brain neuron group and is the main carrier of memory; the memory judgment network judges whether the input image is in memory or whether the input is highly associated with the content of memory; S2: Use the feature extraction network to extract input image features and cluster features of the same category using the idea of ​​exchange; S3: Input the features obtained in step S2 into the hash network, and establish a mapping relationship between the features and the imprint units through the hash network; S4: Using a precipitation function to store features into engram units, and using feature sequences and engram sequences to construct a data set for training subsequent stress functions; the stress function is implemented by a twin network and is used to determine whether the input is in memory, based on the features extracted from the input image and the distribution of the corresponding engram units; S5: Use the prepared dataset to train the stress function network and use the trained model for memorization.

2. A method for implementing a memory network based on imprint precipitation according to claim 1, characterized in that: In step S2, clustering is specifically as follows: input a pair of images at the same time, extract features respectively, then exchange some features at the same position between the two images and reconstruct them. The changed model loss function is: In the formula, x i is the input image, z i is the extracted feature, g(x) represents the decoder for image reconstruction, L rec Represents the reconstruction loss function, where L rec (x i ,g(z i )) is the reconstruction loss of the original image, L rec (x i ,g(z′ i )) is the reconstruction loss after feature exchange.

3. A memory network implementation method based on imprint precipitation as claimed in claim 2, characterized in that: In step S2, in addition to the two pictures of the same category, an additional picture of a different category needs to be introduced, and the same feature extraction steps and reconstruction work are performed. The segmentation operation is performed but no exchange is involved. The three form a triple relationship: (anchor, positive, negative), where anchor and positive are pictures of the same category, and negative is a picture of a different category. The ternary loss function used is: In the formula, ‖*‖ is the Euclidean distance, It represents the Euclidean distance metric between positive and anchor. It represents the Euclidean distance metric between negative and anchor, and α refers to x a With x n The distance between a With x p There is a minimum interval between the distances. + means that when the value in [*] is greater than 0, the value is taken as the loss, and when it is less than 0, the loss is 0.

4. A method for implementing a memory network based on imprint precipitation as claimed in claim 3, characterized in that: The hash network of step S3 takes only the exchanged part of the input features, performs normalization before output, and the output dimension is the number of engram units responsible for the memory of an image; the calculation method of the mapping is specifically: Where n i is the subscript of the corresponding imprint unit, z i As the characteristic, f map represents the hash network, and N represents the total number of imprint units.

5. The method for implementing a memory network based on imprint precipitation according to claim 4, characterized in that: The precipitation function in step S4 is specifically: Where a and k are used to control the single enhancement degree and the imprinting precipitation saturation level, respectively.

6. A memory network implementation method based on imprint precipitation as claimed in claim 5, characterized in that: The twin network obtains the feature sequence z={z1,z2,...,z n }, the corresponding imprinted precipitation sequence u={u1,u2,...,u n }, before memorization, z is not similar to u, so they are labeled as "unseen" and combined together to form a negative sample for training: [{z1,z2,...,z n },{u1,u2,...,u n }, never seen] Then, after the current memory is stored by the precipitation function, z is similar to u, and they are labeled as "seen" and combined together to form a positive sample for training: [{z1,z2,...,z n },{u1,u2,...,u n }, seen] After completing a memory, the changed engram unit may affect the previously memorized picture, because the engram sequence corresponding to the previous picture may have changed. The picture affected by the changed engram unit is "reviewed", and its feature sequence and the current corresponding engram sequence are labeled "seen" again and added to the training sample: [{z1,z2,...,z n },{u1,u2,...,u n } new , met].

7. A memory network implementation system based on imprint precipitation, including a computer program, characterized in that: When the computer program is executed by a processor, the steps of any of the above methods are implemented.

Images