Feature processing method based on attention and memory mechanism and related equipment

Abstract

The invention provides a feature processing method based on an attention and memory mechanism and related equipment. The method comprises the steps of obtaining to-be-input data corresponding to a current moment of a target deep learning unit; performing coding processing and dimension compression on the current input data to obtain a plurality of compression vectors; calculating attention weights corresponding to the plurality of compression vectors based on the query projection matrix, the key projection matrix and the value projection matrix, and obtaining a local feature vector according to the plurality of attention weights; calculating a plurality of global feature vectors corresponding to the local feature vectors based on a memory matrix, and performing weighted calculation on the plurality of global feature vectors to obtain a plurality of memory feature vectors; and feature fusion and dimension reduction are performed on the plurality of memory feature vectors to obtain reconstructed data, so that the target deep learning unit performs training based on the reconstructed data and executes a corresponding deep learning task, the long-term dependency relationship in the long-term input data can be effectively captured, and the model performance is further improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of deep learning network training, and in particular to a feature processing method based on attention and memory mechanisms and related equipment. Background Art

[0002] At present, with the development of deep learning network technology, the scale and complexity of input data that deep learning network models need to process during training and reasoning are increasing. Since current deep learning network models generally only rely on short-term data for feature analysis and extraction to achieve subsequent model training and reasoning processes, current deep learning network models are unable to effectively capture long-term dependencies in long-term input data such as network traffic behavior when performing long-term dependent training and reasoning tasks such as network anomaly detection. The lack of long-term dependency modeling capabilities leads to poor situational awareness performance of current deep learning network models and the inability to effectively perform corresponding training and reasoning tasks. For example, in network anomaly detection tasks, potential security threats are easily missed and they are unable to cope with complex and changeable network environments and attack methods. Summary of the invention

[0003] The embodiment of the present application provides a feature processing method based on attention and memory mechanism, which can perform feature extraction processing on the input data corresponding to the current moment of the target deep learning unit based on the attention and memory mechanism, so that the target deep learning unit can effectively capture the long-term dependencies in the long-term input data, thereby improving its situation awareness performance.

[0004] To achieve the above-mentioned purpose, the first aspect of an embodiment of the present application provides a feature processing method based on an attention and memory mechanism, including: obtaining the to-be-input data corresponding to the target deep learning unit at the current moment; encoding the to-be-input data to obtain multiple input feature vectors, and dimensionally compressing the multiple input feature vectors to obtain multiple compressed vectors; performing attention matrix reconstruction training based on the multiple compressed vectors to obtain a query projection matrix, a key projection matrix, and a value projection matrix; calculating the attention weights corresponding to the multiple compressed vectors based on the query projection matrix, the key projection matrix, and the value projection matrix, and obtaining local feature vectors based on the multiple attention weights; according to the memory matrix generated according to the historical input data corresponding to the target deep learning unit at the historical moment, calculating multiple global feature vectors corresponding to the local feature vector based on the memory matrix, and performing weighted calculation on the multiple global feature vectors to obtain multiple memory feature vectors; performing feature fusion and dimensionality restoration on the multiple memory feature vectors to obtain reconstructed data, and the reconstructed data is used to train the target deep learning unit as the current input data at the current moment.

[0005] In some embodiments, the attention weights corresponding to the multiple compressed vectors are calculated based on the query projection matrix, the key projection matrix and the value projection matrix, including: calculating the query vectors corresponding to the multiple compressed vectors according to the query projection matrix; calculating the key vectors corresponding to the multiple compressed vectors according to the key projection matrix; calculating the value vectors corresponding to the multiple compressed vectors according to the value projection matrix; performing a dot product operation on each compressed vector according to the transposed matrix of the query vector and the key vector to obtain a query-key similarity index; and normalizing the multiple query-key similarity indices to obtain the attention weights corresponding to each compressed vector.

[0006] In some embodiments, obtaining a local feature vector based on multiple attention weights includes: performing a dot product operation between the corresponding value vector and the attention weight for each compression vector to obtain an attention feature; and obtaining a local feature vector based on the attention feature corresponding to each compression vector.

[0007] In some embodiments, the memory matrix includes multiple memory units, each of which contains a historical feature vector generated based on the historical input data, and the number of the memory units and the feature dimensions of the historical feature vector are determined based on the input data scale of the target deep learning unit.

[0008] In some embodiments, the calculation of multiple global feature vectors corresponding to the local feature vector based on the memory matrix includes: performing a dot product operation on each memory unit in the memory matrix according to the transposed matrix of the local feature vector and the historical feature vector to obtain a global feature vector.

[0009] In some embodiments, performing weighted calculation on the multiple global feature vectors to obtain multiple memory feature vectors includes: performing dot product operation on each memory unit in the memory matrix according to the global feature vector and the historical feature vector to obtain a memory feature vector.

[0010] In some embodiments, the feature fusion and dimensionality restoration of the multiple memory feature vectors to obtain reconstructed data includes: performing feature fusion according to the memory feature vectors corresponding to the multiple memory units to obtain a fused feature vector; and performing dimensionality restoration of the fused feature vector based on the data dimension corresponding to the data to be input to obtain reconstructed data.

[0011] In some embodiments, encoding the data to be input to obtain multiple input feature vectors includes: preprocessing the data to be input to obtain preprocessed data; encoding the preprocessed data to obtain multiple input feature vectors; wherein the preprocessing includes normalization or standardization.

[0012] In some embodiments, the dimension compression of the multiple input feature vectors to obtain multiple compressed vectors includes: obtaining current feature dimensions corresponding to the multiple input feature vectors; determining a dimensionality reduction neural network based on the current feature dimensions and a preset target feature dimension; and dimensionally compressing the multiple input feature vectors based on the dimensionality reduction neural network to obtain multiple compressed vectors.

[0013] In some embodiments, the target deep learning unit is used to perform network anomaly detection, and the data to be input includes network traffic data used to perform the network anomaly detection.

[0014] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application provides a feature processing device based on an attention and memory mechanism, including: a data encoding module, used to obtain the to-be-input data corresponding to the target deep learning unit at the current moment; encode the to-be-input data to obtain multiple input feature vectors, and perform dimension compression on the multiple input feature vectors to obtain multiple compressed vectors; an attention module, used to perform attention matrix reconstruction training based on the multiple compressed vectors to obtain a query projection matrix, a key projection matrix and a value projection matrix; calculate the attention weights corresponding to the multiple compressed vectors based on the query projection matrix, the key projection matrix and the value projection matrix, and obtain local feature vectors based on the multiple attention weights; a memory module, used to generate a memory matrix based on the historical input data corresponding to the target deep learning unit at the historical moment, calculate multiple global feature vectors corresponding to the local feature vector based on the memory matrix, and perform weighted calculation on the multiple global feature vectors to obtain multiple memory feature vectors; perform feature fusion and dimension restoration on the multiple memory feature vectors to obtain reconstructed data, and the reconstructed data is used to train the target deep learning unit as the current input data at the current moment.

[0015] To achieve the above-mentioned objectives, the third aspect of an embodiment of the present application provides a computer device, comprising: at least one processor; at least one memory for storing at least one program; when at least one of the programs is executed by at least one of the processors, a feature processing method based on attention and memory mechanism as described in any one of the first aspects is implemented.

[0016] To achieve the above-mentioned objectives, the fourth aspect of an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the feature processing method based on attention and memory mechanism as described in any one of the fourth aspects.

[0017] The embodiment of the present application provides a feature processing method and related equipment based on attention and memory mechanism, which can obtain the input data corresponding to the target deep learning unit at the current moment; encode the input data to obtain multiple input feature vectors, and compress the multiple input feature vectors to obtain multiple compressed vectors; perform attention matrix reconstruction training based on the multiple compressed vectors to obtain a query projection matrix, a key projection matrix and a value projection matrix; calculate the attention weights corresponding to the multiple compressed vectors based on the query projection matrix, the key projection matrix and the value projection matrix, and obtain local feature vectors according to the multiple attention weights; calculate multiple global feature vectors corresponding to the local feature vector based on the memory matrix according to the historical input data corresponding to the target deep learning unit at the historical moment, and perform weighted calculation on the multiple global feature vectors to obtain multiple memory feature vectors; perform feature fusion and dimensionality restoration on the multiple memory feature vectors to obtain reconstructed data, and the reconstructed data is used as the current input data at the current moment to train the target deep learning unit so that the target deep learning unit can train the target deep learning unit based on the current input data at the current moment. Reconstruct data to perform corresponding deep learning tasks; wherein, the present application extracts features from the original data to perform local feature vector extraction based on the attention mechanism, and uses the memory mechanism on the basis of the attention mechanism to perform global feature vector extraction, and performs feature reconstruction to obtain reconstructed data, so that the corresponding target deep learning unit can perform various downstream tasks such as classification, regression, and anomaly detection based on the reconstructed data. It can be understood that since the memory mechanism in the embodiment of the present application can effectively handle long-term dependency problems in the data and can be applicable to multivariate time series and anomaly detection scenarios, the target deep learning unit can effectively capture the long-term dependency relationship in long-term input data such as network traffic behavior based on the reconstructed data when performing long-term dependent training and reasoning tasks such as network anomaly detection, improve the modeling ability of long-term dependency, and improve the potential perception performance of the corresponding deep learning network model state, so as to effectively perform corresponding training and reasoning tasks, for example, effectively identify potential security threats in network anomaly detection tasks, and be able to cope with complex and changeable network environments and attack methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a feature processing method based on attention and memory mechanism provided in one embodiment of the present application;

[0019] Figure 2A flow chart for calculating attention weight in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application;

[0020] Figure 3 A flow chart of obtaining a local feature vector in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application;

[0021] Figure 4 A flow chart of obtaining a global feature vector in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application;

[0022] Figure 5 A flow chart of obtaining a memory feature vector in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application;

[0023] Figure 6 A flow chart of obtaining reconstructed data in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application;

[0024] Figure 7 A flow chart of obtaining multiple compressed vectors in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of a device corresponding to a feature processing method based on attention and memory mechanism provided in an embodiment of the present application;

[0026] Fig. 9 A schematic diagram of modules corresponding to a feature processing method based on attention and memory mechanism provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] In some embodiments, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms first, second, etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0030] In addition, unless otherwise clearly specified and limited, the term "connected" should be understood in a broad sense. For example, it can be a fixed connection or a movable connection, a detachable connection or a non-detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or the two can communicate with each other; it can be a direct connection or an indirect connection through an intermediate medium.

[0031] In the description of the embodiments of the present application, the description of reference terms "one embodiment / implementation", "another embodiment / implementation" or "certain embodiments / implementations", "in the above embodiments / implementations", etc. means that the specific features, structures, materials or characteristics described in combination with the implementation or example are included in at least two embodiments or implementations disclosed in the present application. In the disclosure of this application, the schematic representation of the above terms does not necessarily refer to the same embodiment or implementation. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that in the flowchart.

[0032] At present, with the development of deep learning network technology, the scale and complexity of input data that deep learning network models need to process during training and reasoning are increasing. Since current deep learning network models generally only rely on short-term data for feature analysis and extraction to achieve subsequent model training and reasoning processes, current deep learning network models are unable to effectively capture long-term dependencies in long-term input data such as network traffic behavior when performing long-term dependent training and reasoning tasks such as network anomaly detection. The lack of long-term dependency modeling capabilities leads to poor situational awareness performance of current deep learning network models and the inability to effectively perform corresponding training and reasoning tasks. For example, in network anomaly detection tasks, potential security threats are easily missed and they are unable to cope with complex and changeable network environments and attack methods.

[0033] It can be understood that in order to overcome the limitations of the prior art, the embodiments of the present application provide a feature processing method based on attention and memory mechanisms, which can perform feature extraction processing on the input data corresponding to the target deep learning unit at the current moment based on the attention and memory mechanisms, so that the target deep learning unit can effectively capture the long-term dependencies in the long-term input data, thereby improving its situational awareness performance.

[0034] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0035] refer to Figure 1 , Figure 1A flowchart of a feature processing method based on attention and memory mechanism provided in an embodiment of the present application; To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a feature processing method based on attention and memory mechanism, which is used for feature extraction processing of high-dimensional data, and the quality of feature extraction is evaluated by the ability to identify open unknown samples, and the method includes but is not limited to the following steps:

[0036] Step S110, obtaining the to-be-input data corresponding to the current moment of the target deep learning unit;

[0037] Step S120, encoding the input data to obtain a plurality of input feature vectors, and dimensionality compressing the plurality of input feature vectors to obtain a plurality of compressed vectors;

[0038] Step S130, performing attention matrix reconstruction training based on multiple compressed vectors to obtain a query projection matrix, a key projection matrix, and a value projection matrix;

[0039] Step S140, calculating attention weights corresponding to a plurality of compressed vectors based on the query projection matrix, the key projection matrix, and the value projection matrix, and obtaining a local feature vector according to the plurality of attention weights;

[0040] Step S150, according to the memory matrix generated by the historical input data corresponding to the target deep learning unit at the historical moment, multiple global feature vectors corresponding to the local feature vector are calculated based on the memory matrix, and the multiple global feature vectors are weighted to obtain multiple memory feature vectors;

[0041] Step S160, performing feature fusion and dimensionality restoration on multiple memory feature vectors to obtain reconstructed data.

[0042] Among them, the reconstructed data is used as the current input data at the current moment to train the target deep learning unit, so that the target deep learning unit performs the corresponding deep learning task based on the reconstructed data.

[0043] In some embodiments, corresponding to steps S110 to S160, the present application extracts features from the original data to perform local feature vector extraction based on the attention mechanism, and uses the memory mechanism based on the attention mechanism to extract global feature vectors, and reconstructs the input data to obtain reconstructed data, so that the corresponding target deep learning unit can perform various downstream tasks such as classification, regression, and anomaly detection based on the reconstructed data. It can be understood that since the memory mechanism in the embodiment of the present application can effectively handle long-term dependency problems in the data and can be applicable to multivariate time series and anomaly detection scenarios, the target deep learning unit can effectively capture the long-term dependency relationship in long-term input data such as network traffic behavior based on the reconstructed data when performing long-term dependent training and reasoning tasks such as network anomaly detection, improve the modeling ability of long-term dependency, and improve the potential perception performance of the corresponding deep learning network model state, so as to effectively perform the corresponding training and reasoning tasks, for example, effectively identify potential security threats in network anomaly detection tasks, and be able to cope with complex and changeable network environments and attack methods.

[0044] Among them, the target deep learning unit can be a model or module used to perform deep learning tasks, such as a part of a neural network model, responsible for tasks such as feature extraction, classification, and regression. Specifically, it can be a network anomaly detection module that is used to perceive abnormal trends in multivariate time series data, analyze network traffic data, user behavior logs and other data through a neural network model, extract features, and thus discover potential network security threats.

[0045] Among them, the input data corresponding to the current moment can be the original data that the target deep learning unit currently needs to process, such as network traffic data, user behavior logs, etc. This application converts the original input data into a format suitable for deep learning model processing, such as through normalization, standardization and other data preprocessing, feature extraction and other steps, to obtain the input feature vector. The input feature vector is the feature representation obtained after encoding processing, and each feature vector contains partial feature information of the input data.

[0046] In some embodiments, corresponding to steps S120 to S140, the present application can convert the high-dimensional data of the input feature vector into a low-dimensional feature representation of a compressed vector by using dimensionality reduction techniques such as convolutional neural networks (CNN) and fully connected networks (FCN) to reduce data complexity and extract key features, and then generate the query projection matrix, key projection matrix and value projection matrix required for the attention mechanism through a training process. These matrices are used to calculate the attention weights. The query projection matrix, key projection matrix and value projection matrix are used to convert the input feature vector into a query vector, a key vector and a value vector, which are respectively used to calculate the attention weights and generate local feature vectors.

[0047] Among them, the attention weight represents the importance of each element in the input feature vector, which is calculated through the attention mechanism and used to weightedly combine the input feature vector to generate a local feature vector; the local feature vector represents the feature vector generated by combining the attention weight, emphasizing the important features of the input data at the current moment.

[0048] In some embodiments, corresponding to step S150, the memory matrix is ​​for storing global feature information of historical input data. The memory matrix may include multiple memory units, each memory unit records a global feature, which is used to capture long-term dependencies in the data, and then the global feature vector calculated based on the memory matrix can reflect the global historical information of the input data; further, the present application can obtain a memory feature vector by combining the information of local features and global features with the feature vector obtained by weighted calculation of the global feature vector, and combine the local feature vector with the memory feature vector to generate a comprehensive feature representation; further, the fused feature vector is restored to the same dimension as the input data for use in subsequent deep learning tasks.

[0049] In some embodiments, the embodiments of the present application provide a feature processing method based on attention and memory mechanism, which aims to improve the accuracy of feature extraction and the situational awareness performance of the model by combining local features and global historical information. It can be used for the training and reasoning tasks of network security detection models. Specifically, the present application obtains the original input data to be processed, such as network traffic data or user behavior logs; encodes the input data corresponding to the current moment, extracts multiple input feature vectors, and compresses the dimensions of these feature vectors through a convolutional neural network (CNN) or a fully connected network (FCN) to obtain multiple compressed vectors; performs attention matrix reconstruction training based on multiple compressed vectors to generate a query projection matrix, a key projection matrix and a value projection matrix; these matrices are used for subsequent attention weight calculations; the query projection matrix, the key projection matrix and the value projection matrix are used to calculate the attention weights corresponding to the multiple compressed vectors, according to These attention weights generate local feature vectors to emphasize the important features of the current moment; a memory matrix is ​​generated based on the historical input data corresponding to the target deep learning unit at the historical moment, and the matrix stores the global historical information; multiple global feature vectors corresponding to the local feature vectors are calculated based on the memory matrix, and these global feature vectors are weighted to obtain multiple memory feature vectors; multiple memory feature vectors are feature fused to generate a comprehensive feature representation; then the fused feature vectors are dimensionally restored through a fully connected network or a deconvolution network to obtain reconstructed data of the same dimension as the input data, so that the target deep learning unit performs the corresponding deep learning tasks based on the reconstructed data, such as classification, regression, anomaly detection, etc.; through the above steps, this method combines the attention mechanism and the memory mechanism, effectively captures the long-term dependencies and local features in the data, and improves the accuracy of feature extraction and the situational awareness performance of the model.

[0050] In some embodiments, the historical moment in the present application represents a historical time period before the current moment. It can be understood that before the data to be input is input into the target deep learning unit, the present application scheme can perform feature extraction on the input data currently to be input into the target deep learning unit based on the recorded input data of the target deep learning unit at the historical moment; that is, the present application scheme triggers the feature extraction optimization based on historical data for the current data to be input into the model before the data is input into the target deep learning unit.

[0051] refer to Figure 2 , Figure 2 A flowchart of calculating attention weights in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application; in some embodiments, the attention weights corresponding to multiple compressed vectors are calculated based on the query projection matrix, the key projection matrix, and the value projection matrix, including but not limited to the following steps:

[0052] Step S210, calculating a query vector corresponding to a plurality of compressed vectors according to a query projection matrix;

[0053] Step S220, calculating key vectors corresponding to the plurality of compression vectors according to the key projection matrix;

[0054] Step S230, calculating value vectors corresponding to the plurality of compressed vectors according to the value projection matrix;

[0055] Step S240, performing a dot product operation on each compressed vector according to the transposed matrix of the query vector and the key vector to obtain a query-key similarity index;

[0056] Step S250, normalize the multiple query key similarity indexes to obtain the attention weight corresponding to each compression vector.

[0057] In some embodiments, the present application can calculate the query vector corresponding to each compressed vector according to the query projection matrix and multiple compressed vectors, calculate the key vector corresponding to each compressed vector according to the key projection matrix and multiple compressed vectors, and calculate the value vector corresponding to each compressed vector according to the value projection matrix and multiple compressed vectors, and then for each compressed vector, perform a dot product operation according to the transposed matrix of the query vector and the key vector to obtain a query key similarity index, and normalize the multiple query key similarity indices to obtain an attention weight corresponding to each compressed vector; wherein it can be understood that the above process summarizes the query projection matrix, the key projection matrix and the value projection matrix, and the compressed vector can be converted into a query vector, a key vector and a value vector, and these vectors can more accurately represent the characteristics of the input data, thereby improving the feature extraction ability of the model; further, through the dot product operation and normalization processing, the attention weight of each compressed vector can be calculated, and these weights reflect the importance of each compressed vector in the current context, so that the model can more effectively focus on important features and ignore unimportant parts. The attention mechanism allows the model to dynamically select important features in different contexts, thereby improving the flexibility and adaptability of the model, especially when processing complex and changeable data, it can better capture key information

[0058] In some embodiments, the attention matrix reconstruction training is performed based on multiple compressed vectors to obtain the query projection matrix, the key projection matrix and the value projection matrix, which can include the following steps: first, the query projection matrix, the key projection matrix and the value projection matrix are initialized, and these matrices can usually be initialized from random distribution. Then, when there are multiple compressed vectors from any part of the input data, such as word embedding of text, feature map of image, etc., each compressed vector can be projected to the query space, key space and value space respectively by using the initialized projection matrix, so as to calculate the similarity score between the query vector and the key vector using the dot product method, and the learnable query projection matrix, the learnable key projection matrix and the learnable value projection matrix are repeatedly reconstructed, and the attention output can be used for downstream tasks (such as classification, regression, etc.), and the loss function is calculated. Then, the parameters of the projection matrix are updated by the back propagation algorithm to minimize the loss function, and the above steps are repeated until the model converges or reaches a predetermined training round. In each iteration, the projection matrix will be gradually optimized to obtain the query projection matrix, key projection matrix and value projection matrix used in the above embodiment process of the present application, so that the attention mechanism can better capture the important features in the input data.

[0059] refer to Figure 3 , Figure 3 A flowchart of obtaining a local feature vector in a feature processing method based on an attention and memory mechanism provided in an embodiment of the present application; in some embodiments, obtaining a local feature vector according to multiple attention weights includes but is not limited to the following steps:

[0060] Step S310, for each compressed vector, perform a dot product operation according to the corresponding value vector and the attention weight to obtain an attention feature;

[0061] Step S320, obtaining a local feature vector according to the attention feature corresponding to each compression vector.

[0062] In some embodiments, the present application can obtain attention features by performing dot product operations on each compression vector according to the corresponding value vector and the attention weight, and generate a local feature vector according to the attention features corresponding to each compression vector. Specifically, the local feature vector can be obtained by weighted summing all attention features; it can be understood that by performing dot product operations on the attention weights and the value vectors, attention features can be generated, which emphasize the important parts of the input data, thereby enhancing the expressive power of the features. The local feature vector integrates all attention features, highlights the most relevant information in the input data, enables the model to focus more on local features, improves the accuracy of feature extraction and the performance of the model, and effectively reduces the interference of noise and irrelevant information.

[0063] In some embodiments, the memory matrix includes multiple memory units, each of which contains historical feature vectors generated based on historical input data. The number of memory units and the feature dimensions of the historical feature vectors are determined based on the input data scale of the target deep learning unit. For example, if the time span of the input data is longer and the feature dimension is higher, larger hyperparameters can be used as the number of memory units and the dimension value of the feature dimension. It can be understood that the memory matrix can effectively capture the long-term dependencies in the input data by storing historical feature vectors, so that the corresponding deep learning model can effectively process time series data, natural language processing and other tasks, and can improve the model's ability to model long-term dependencies, so that the model can better capture key information and improve the accuracy of detection and prediction when processing tasks such as network anomaly detection and time series prediction.

[0064] In some embodiments, specifically, given that the input compressed feature vector is h, the local feature vector determination process in the above embodiment is as follows: Generate query Q, key K, value V through the attention mechanism:

[0065] Q=W Q h,K=W K h,V=W V h, where W Q ,W K ,W V is a learnable projection matrix; further, the attention weight is calculated: Then the local feature vector h' is obtained by the weighted value V:

[0066] In some embodiments, before calculating the attention weight, multi-scale feature extraction is performed on the compressed vector. For example, convolution kernels or pooling windows of different sizes can be used to extract features of different scales, and the multi-scale features are fused to generate a multi-scale feature vector. Feature fusion can be performed using weighted summation, splicing, etc. Attention weights are calculated based on the multi-scale feature vector, and a local feature vector is generated. Multi-scale feature fusion can capture the features of the input data at different scales and improve the richness and comprehensiveness of the feature representation. Through multi-scale feature fusion, the model can better capture the complex patterns in the input data and improve the performance of the model.

[0067] refer to Figure 4 , Figure 4 A flow chart of obtaining a global feature vector in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application; in some embodiments, multiple global feature vectors corresponding to local feature vectors are calculated based on a memory matrix, including but not limited to the following steps:

[0068] Step S410, for each memory unit in the memory matrix, perform a dot product operation based on the transposed matrix of the local eigenvector and the historical eigenvector to obtain a global eigenvector.

[0069] In some embodiments, global feature vectors can be obtained by performing dot product operations on local feature vectors and historical feature vectors in a memory matrix. These global feature vectors not only contain information about current local features, but also combine long-term dependencies in historical data, thereby enhancing the richness and comprehensiveness of feature representation. The historical feature vectors in the memory matrix can preserve information over a long time span, allowing the model to better capture long-term dependencies in input data, which is particularly important for tasks such as processing time series data and natural language processing.

[0070] refer to Figure 5 , Figure 5 A flowchart of obtaining a memory feature vector in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application; in some embodiments, weighted calculation is performed on multiple global feature vectors to obtain multiple memory feature vectors, including but not limited to the following steps:

[0071] Step S510, for each memory unit in the memory matrix, a dot product operation is performed according to the global feature vector and the historical feature vector to obtain a memory feature vector.

[0072] Among them, the memory matrix is ​​used to capture the long-term dependencies in the input data, helping the model to better understand and process time series data or long-term dependent tasks; the local feature vector is the feature vector extracted from the current input data through the attention mechanism, emphasizing the important part of the current input data. The local feature vector provides an immediate feature representation of the current input data and is the focus of the model; the historical feature vector is the feature vector stored in the memory matrix, which records the feature information of the past input data. The historical feature vector provides the long-term historical information of the input data and helps the model capture long-term dependencies; the transposed matrix refers to a new matrix obtained by swapping the rows and columns of the matrix. In the dot product operation, the use of the transposed matrix can make the matrix multiplication meet the requirements of mathematical operations and ensure Ensure the correctness of the dot product operation; the dot product operation is the inner product operation between two vectors, and the result is a scalar. The dot product operation is used to calculate the similarity between two vectors. The larger the result, the more similar the two vectors are; the global feature vector is a feature vector generated by combining the local feature vector and the historical feature vector, which contains comprehensive information of the current input data and the historical data. The global feature vector provides a comprehensive feature representation of the input data to help the model better understand and process complex input data; the memory feature vector is a feature vector generated by combining the global feature vector and the historical feature vector, which further enhances the comprehensiveness and richness of the feature representation. The memory feature vector provides a comprehensive feature representation of the input data to help the model better understand and process complex input data.

[0073] In some embodiments, a memory feature vector can be generated for each memory unit through dot product operations and weighted calculations. These feature vectors can more finely select and emphasize the important parts of the input data corresponding to the current moment, thereby improving the accuracy of feature selection; in addition, since the generation process of the memory feature vector is dynamic, the importance of the feature can be adjusted in real time according to the changes in the input data corresponding to the current moment, so that the model can better adapt to different input scenarios.

[0074] refer to Figure 6 , Figure 6 In the feature processing method based on attention and memory mechanism provided in one embodiment of the present application, a flowchart of obtaining reconstructed data is provided; in some embodiments, feature fusion and dimensionality reduction are performed on multiple memory feature vectors to obtain reconstructed data, including but not limited to the following steps:

[0075] Step S610, performing feature fusion according to the memory feature vectors corresponding to the multiple memory units to obtain a fused feature vector;

[0076] Step S620, restore the dimension of the fused feature vector based on the data dimension corresponding to the input data to obtain reconstructed data.

[0077] In some embodiments, the feature fusion process merges multiple memory feature vectors into a fused feature vector, integrating the information of different memory units, making the feature representation more comprehensive and integrated; the dimensionality restoration process ensures that the dimension of the fused feature vector is consistent with the dimension of the input data, which facilitates subsequent processing and application and avoids problems caused by dimensionality mismatch. It can be understood that the reconstructed data not only retains the key features of the input data, but also combines historical information and global information, thereby improving the performance of the model in feature extraction and task processing.

[0078] In some embodiments, the local feature vector determination process in the above embodiment is as follows: given a local feature vector h', determine that the memory module includes a memory matrix M∈R N×d , used to store global historical information, N represents the number of memory units, d is the feature dimension, and the correlation between the input feature and the memory unit is calculated by dot product to generate a weighted global feature representation: s = h′·m T , where s is the correlation between feature h and memory unit m. Then, each memory unit is weighted according to the correlation to obtain the weighted memory feature: r = s m. Finally, all weighted memory unit features are fused to obtain the global feature vector representation h″:

[0079] In some embodiments, during the generation process of the memory matrix, an external knowledge base can also be introduced. By introducing an external knowledge base, such as domain expert knowledge, historical data statistics, etc., as auxiliary information, the information in the external knowledge base is fused with the historical feature vector in the memory matrix to generate an enhanced memory feature vector, and the global feature vector is calculated based on the enhanced memory feature vector; it can be understood that the introduction of the external knowledge base can provide additional information and enhance the expressiveness of the model. By fusing external knowledge, the model can better understand and process input data and improve the accuracy of the model; in addition, in some embodiments, when updating the memory matrix, an attention mechanism can be used to determine which memory units need to be updated, and the attention weight of each memory unit is calculated according to the current input data and the output of the model, and the memory units are weighted updated according to the attention weight. The memory units with high attention weights have a larger update amplitude, and the memory units with low attention weights have a smaller update amplitude. The calculation method of the attention weight is dynamically adjusted during the training process to optimize the update effect of the memory matrix, and then through the attention mechanism, unnecessary memory unit updates can be reduced and the efficiency of the model can be improved.

[0080] In some embodiments, encoding is performed on the input data to obtain multiple input feature vectors, including: preprocessing the input data to obtain preprocessed data; encoding is performed on the preprocessed data to obtain multiple input feature vectors; wherein the preprocessing includes normalization or standardization. It can be understood that the normalization or standardization in the preprocessing step can eliminate the dimensional differences in the data, making the model more stable during the training process, improving the convergence speed and model performance, and the encoding process can convert the original input data into a more advanced feature representation, which can better capture the intrinsic structure and pattern of the input data and improve the feature extraction capability of the model.

[0081] In some embodiments, when preprocessing the data to be input corresponding to the current moment, raw data such as time series, images or other multidimensional data can be input into the neural network model for preprocessing. The data can be unlabeled raw input so that features can be extracted in an unsupervised learning scenario. Specifically, the sample of the data to be input corresponding to the current moment can be an image collected by a camera, behavioral data collected by behavioral sensors such as IMU, or gesture action states in a synthetic gesture action sequence. The data acquired by the unit is transmitted to the abnormal perception part through a readable and writable storage medium or a communicative data transmission medium. Data preprocessing operations such as normalization or standardization are performed on the input data to ensure the consistency of feature distribution and reduce data noise.

[0082] refer to Figure 7 , Figure 7 A flowchart of obtaining multiple compressed vectors in a feature processing method based on attention and memory mechanism provided in an embodiment of the present application; in some embodiments, multiple input feature vectors are dimensionally compressed to obtain multiple compressed vectors, including but not limited to the following steps:

[0083] Step S710, obtaining current feature dimensions corresponding to multiple input feature vectors;

[0084] Step S720, determining a dimensionality reduction neural network according to the current feature dimension and a preset target feature dimension;

[0085] Step S730: Dimensionally compressing the multiple input feature vectors based on the dimensionality reduction neural network to obtain multiple compressed vectors.

[0086] In some embodiments, dimensionality compression can effectively reduce the dimension of the feature vector, thereby reducing the memory usage of the model during training and inference. The compressed feature vector has a lower dimension, which reduces the amount of calculation and speeds up the training and inference of the model, especially when processing large-scale data sets. In addition, high-dimensional feature vectors are prone to overfitting of the model. Dimension compression can reduce redundant information of the features and reduce the complexity of the model, thereby reducing the risk of overfitting. The compressed feature vector retains the main features of the input data and removes noise and irrelevant information, so that the model has better generalization ability when processing new data.

[0087] In some embodiments, the process of obtaining current feature dimensions corresponding to multiple input feature vectors, determining a dimensionality reduction neural network, and performing dimensionality compression may include obtaining current feature dimensions corresponding to multiple input feature vectors. Assuming that multiple input feature vectors are obtained after encoding processing, the current feature dimensions are obtained by checking the shapes of the input feature vectors. This can be achieved through array or tensor operations in the programming language to ensure that the dimensional information of the input feature vectors is accurate, providing a basis for subsequent dimensional compression operations. In addition, by obtaining the current feature dimensions, data consistency checks can be performed to ensure that the dimensions of all input feature vectors are consistent, thereby avoiding errors caused by inconsistent dimensions.

[0088] In some embodiments, when determining a dimensionality reduction neural network based on a current feature dimension and a preset target feature dimension, a target feature dimension can be set. The selection of the target feature dimension can be determined based on the requirements of a specific task and the limitations of computing resources. A suitable dimensionality reduction method is selected based on the current feature dimension and the target feature dimension. For example, the dimensionality reduction method may include: Fully Connected Layer: dimensionality reduction through one or more fully connected layers; Convolutional Neural Network (CNN): dimensionality reduction through convolutional layers and pooling layers; Autoencoder: dimensionality reduction and feature reconstruction through encoders and decoders.

[0089] In some embodiments, a dimensionality reduction neural network is constructed according to a selected dimensionality reduction method. For example, when using a fully connected layer, a simple multi-layer perceptron can be constructed, and then a suitable dimensionality reduction method can be selected according to the input feature dimension and the target feature dimension, so that the model can flexibly adapt to different data and task requirements. By selecting a suitable dimensionality reduction method, the feature dimension can be efficiently reduced and the computational efficiency of the model can be improved. By dimensional compression of multiple input feature vectors, the consumption of computing resources can be effectively reduced, the model efficiency can be improved, the model structure can be optimized, and the feature table can be enhanced. In addition, an incremental learning method can be used to gradually update the historical feature vectors in the memory matrix, and only part of the memory units are updated each time, instead of updating the entire memory matrix at one time. After each update, the previous update results are accumulated, and the historical feature vectors in the memory matrix are gradually optimized. The amount of computation for each update can be reduced through incremental memory updates, thereby improving the efficiency of the model.

[0090] In some embodiments, in each training step or inference step of the target deep learning unit, the memory matrix can be dynamically updated according to the current input data and the output of the model. For example, the gradient descent method or other optimization algorithm can be used to update the historical feature vectors in the memory matrix, or certain memory cells in the memory matrix can be selectively updated according to the importance of the current input data and the performance of the model. The memory cells in the memory matrix can be regularly pruned to maintain the efficiency and manageability of the memory matrix and reduce redundant information in the memory matrix.

[0091] In some embodiments, it is understood that the input data of the present application can be multimodal data (such as text, images, time series, etc.), so the memory matrix in the present application can be a memory matrix that supports multimodal data (such as text, images, time series, etc.). Each memory unit can contain different types of data features. When calculating the global feature vector, the present application can perform cross-modal fusion of memory units of different modalities. Attention mechanisms or other fusion methods can be used to integrate information of different modalities, and memory units of different modalities can be specifically processed, such as using different neural network structures or feature processing methods. Through cross-modal fusion, the model can better understand and process multimodal data and improve the performance of the model.

[0092] refer to Figure 8 , Figure 8A schematic diagram of a device corresponding to the feature processing method based on attention and memory mechanism provided in an embodiment of the present application; in some embodiments, icon 8 corresponds to the overall process of a high-performance deep learning platform or device, and the input data can be from an external data source as the original input of the entire system; the acquisition and compression device input data can be to collect input data and perform preliminary compression processing on it to reduce the demand for data transmission and storage; the memory and attention hybrid neural network is mainly responsible for the core feature extraction and analysis work. This module includes attention mechanism and memory mechanism for efficient processing and understanding of complex data patterns; the decoding and restoration device converts the processed data back into a readable form so that subsequent applications or users can understand and use it; the reconstructed data is the output result, which is usually optimized and enhanced information, and can be directly applied to various application scenarios.

[0093] Understandably, Figure 8 The medium and high performance deep learning platform or device corresponds to the target deep learning model in the embodiment of the present application; input data: corresponding to step S110, the to-be-input data corresponding to the target deep learning unit at the current moment is obtained; the acquisition and compression device: corresponding to step S120, the to-be-input data is encoded to obtain multiple input feature vectors, and the multiple input feature vectors are dimensionally compressed to obtain multiple compressed vectors; the memory and attention hybrid neural network: includes the attention mechanism and the memory mechanism, corresponding to each sub-step of steps S130 to S160, specifically, corresponding to step S130, the attention matrix is ​​reconstructed and trained based on multiple compressed vectors to obtain the query projection matrix, the key projection matrix and the value projection matrix; the memory module: corresponding to step S150, the memory matrix is ​​generated according to the historical input data corresponding to the target deep learning unit at the historical moment, the multiple global feature vectors corresponding to the local feature vector are calculated based on the memory matrix, and the multiple global feature vectors are weighted to obtain multiple memory feature vectors; the decoding and restoration device: corresponding to step S160, that is, the data is reconstructed based on the fused feature vector: corresponding to the output of step S160, the reconstructed data is obtained, and the reconstructed data is used as the current input data at the current moment to train the target deep learning unit.

[0094] It is worth noting that the present application scheme can be applied to Figure 8 In the high-performance deep learning platform or device shown, through the above embodiments, Figure 8 The target deep learning model deployed on the high-performance deep learning platform or device shown provides feature extraction processing so that it can effectively capture long-term dependencies in long-term input data, thereby improving its situational awareness performance, and then effectively perform corresponding training and reasoning tasks. For example, in network anomaly detection tasks, it can effectively identify potential security threats and cope with complex and changeable network environments and attack methods.

[0095] refer to Fig. 9 , Fig. 9 Schematic diagram of modules corresponding to the feature processing method based on attention and memory mechanism provided in one embodiment of the present application; in some embodiments, the encoder corresponds to the first half of step S120, that is, encoding the input data to obtain multiple input feature vectors; the attention module corresponds to steps S130 and S140, that is, performing attention matrix reconstruction training based on multiple compression vectors, and calculating the attention weight to obtain local feature vectors; memory module: corresponds to step S150, that is, using the memory matrix to calculate the global feature vector; weighted fusion corresponds to the first half of step S160, that is, feature fusion of multiple memory feature vectors; the decoder corresponds to the second half of step S160, that is, dimensionality restoration to obtain reconstructed data; the learning module corresponds to the target deep learning model in the embodiment of the present application. It can be understood that through Fig. 9 The internal structure of the attention and memory hybrid neural network in this application is reflected, and the specific functional modules and their mutual relationships are displayed. The three features issued by the attention module are, from left to right, the query vector, the key vector and the value vector. The query vector and the key vector are normalized and probabilized by matrix multiplication to obtain the global feature vector; further, the global feature vector and the value vector are obtained by matrix multiplication to obtain the local feature vector; further, in the memory module, the feature represents the local feature vector, and the memory represents the memory matrix, and then the global feature vector can be generated in turn according to the memory feature vector and the memory matrix, and the weighted fusion (weighted calculation) can be performed to obtain multiple memory feature vectors, and the memory points 1 to n correspond to multiple memory units in the memory matrix; further, after the decoder, it can be restored to the reconstructed data of the same dimension as the input data, so that the target deep learning unit can perform the corresponding deep learning task based on the reconstructed data, so that the deep learning model can effectively capture the long-term dependencies in the long-term input data, thereby improving its model performance.

[0096] It can be understood that the memory mechanism designed by the embodiment of the present application can be used to store and update global historical information, so that the model can capture long-term dependencies in the data by continuously updating these memory units. The memory module calculates the correlation between the input features and the memory units through dot product operations, and generates a weighted global feature representation. Then, the global features of the memory module and the local features of the attention mechanism can be fused to form a comprehensive feature representation that combines global and local information. This fused feature not only retains the global historical information, but also has the spatiotemporal dependency of local features. Furthermore, the fused features are restored to the same dimension as the input data through a fully connected network or a deconvolution network for subsequent reconstruction or classification tasks, so that the model can generate feature representations similar to the original input data, thereby completing efficient reconstruction of the data and effectively improving the accuracy of feature extraction.

[0097] In some embodiments, specifically, when the target deep learning unit is used to perform network anomaly detection, and the data to be input includes network traffic data for performing network anomaly detection, the specific implementation of the following embodiments is as follows: Step S110: Obtain the data to be input corresponding to the target deep learning unit at the current moment, where the data to be input is network traffic data, including but not limited to IP address, port number, data packet size, protocol type, etc., and ensure the format and quality of the data by preprocessing the network traffic data, such as normalization, standardization, missing value filling, etc.; Step S120: Encode the input data to obtain multiple input feature vectors, and perform dimension compression on the multiple input feature vectors to obtain multiple compressed vectors, for example, use a feature processing method (such as TF-IDF, word embedding, etc.) to convert the network traffic data into feature vectors, and use a dimensionality reduction neural network to perform dimension compression on the feature vectors; Step S140, calculate the attention weights corresponding to the multiple compressed vectors based on the query projection matrix, the key projection matrix and the value projection matrix, and obtain the local feature vector according to the multiple attention weights; Step S150, according to the target deep learning unit at the historical moment A memory matrix is ​​generated according to historical input data, multiple global feature vectors corresponding to the local feature vector are calculated based on the memory matrix, and multiple global feature vectors are weighted to obtain multiple memory feature vectors; step S160, feature fusion and dimensionality reduction are performed on multiple memory feature vectors to obtain reconstructed data. Through the above steps, the present application can effectively extract features from network traffic data, and generate global feature vectors and memory feature vectors by using the attention mechanism and the memory mechanism, and generate reconstructed data by feature fusion and dimensionality reduction; wherein, the memory mechanism helps the system capture long-term dependencies in the network by storing and updating historical information, especially those attack behaviors that are operated covertly for a long time, and the attention mechanism dynamically identifies the current important local features in the network and ignores noise data, thereby improving the system's response speed and accuracy to complex attacks. These two means are closely combined, the memory mechanism ensures that the system can remember events and anomalies that have occurred in the network, and the attention mechanism enables the system to focus on key parts in complex data, so that the system can dynamically adapt to the changing network environment, meeting the business needs of real-time and high-precision anomaly detection.

[0098] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application provides a feature processing device based on an attention and memory mechanism, including: a data encoding module, used to obtain the to-be-input data corresponding to the target deep learning unit at the current moment; encoding the to-be-input data to obtain multiple input feature vectors, and dimensionally compressing the multiple input feature vectors to obtain multiple compressed vectors; an attention module, used to perform attention matrix reconstruction training based on the multiple compressed vectors to obtain a query projection matrix, a key projection matrix, and a value projection matrix; calculating the attention weights corresponding to the multiple compressed vectors based on the query projection matrix, the key projection matrix, and the value projection matrix, and obtaining local feature vectors based on the multiple attention weights; a memory module, used to generate a memory matrix based on the historical input data corresponding to the target deep learning unit at the historical moment, A plurality of global feature vectors corresponding to the local feature vector are calculated in the memory matrix, and the plurality of global feature vectors are weightedly calculated to obtain a plurality of memory feature vectors; feature fusion and dimension restoration are performed on the plurality of memory feature vectors to obtain reconstructed data, so that the target deep learning unit performs the corresponding deep learning task based on the reconstructed data. The present application effectively captures the global historical information through the memory module, and performs fine modeling of the local features in combination with the attention mechanism. The present invention effectively improves the accuracy of feature extraction, and the memory mechanism can effectively handle the long-term dependency problem in the data, and is particularly suitable for multivariate time series and anomaly detection scenarios. In addition, the method and system of the present application do not require a large amount of labeled data, and can extract high-quality features in an unsupervised learning environment, thereby reducing the cost and complexity of data annotation.

[0099] It can be understood that the feature processing device based on attention and memory mechanism in the present application extracts features from the original data to perform local feature vector extraction based on the attention mechanism, and uses the memory mechanism on the basis of the attention mechanism to extract global feature vectors, and reconstructs features to obtain reconstructed data, so that the corresponding target deep learning unit can perform various downstream tasks such as classification, regression, and anomaly detection based on the reconstructed data. It can be understood that since the memory mechanism in the embodiment of the present application can effectively handle long-term dependency problems in the data and can be applied to multivariate time series and anomaly detection scenarios, the target deep learning unit can effectively capture the long-term dependency relationship in long-term input data such as network traffic behavior based on the reconstructed data when performing long-term dependent training and reasoning tasks such as network anomaly detection, improve the modeling ability of long-term dependency, and improve the potential perception performance of the corresponding deep learning network model state, so as to effectively perform the corresponding training and reasoning tasks, for example, effectively identify potential security threats in network anomaly detection tasks, and be able to cope with complex and changeable network environments and attack methods.

[0100] In some embodiments, the attention module is also used to calculate the query vector corresponding to multiple compressed vectors according to the query projection matrix; calculate the key vector corresponding to multiple compressed vectors according to the key projection matrix; calculate the value vector corresponding to multiple compressed vectors according to the value projection matrix; for each compressed vector, perform a dot product operation according to the transposed matrix of the query vector and the key vector to obtain a query-key similarity index; normalize the multiple query-key similarity indices to obtain the attention weight corresponding to each compressed vector.

[0101] In some embodiments, the attention module is further used to perform a dot product operation on each compression vector according to the corresponding value vector and the attention weight to obtain an attention feature; and obtain a local feature vector according to the attention feature corresponding to each compression vector.

[0102] In some embodiments, the memory module is further used to perform a dot product operation on each memory unit in the memory matrix according to the transposed matrix of the local eigenvector and the historical eigenvector to obtain a global eigenvector.

[0103] In some embodiments, the memory module is further used to perform a dot product operation on each memory unit in the memory matrix according to the global feature vector and the historical feature vector to obtain a memory feature vector.

[0104] In some embodiments, the data encoding module is also used to perform feature fusion according to the memory feature vectors corresponding to multiple memory units to obtain a fused feature vector; and perform dimension reduction on the fused feature vector based on the data dimension corresponding to the input data to obtain reconstructed data.

[0105] In some embodiments, the data encoding module is also used to preprocess the input data to obtain preprocessed data; encode the preprocessed data to obtain multiple input feature vectors; wherein the preprocessing includes normalization or standardization. In some embodiments, the memory module is also used to obtain the current feature dimensions corresponding to the multiple input feature vectors; determine the dimension reduction neural network according to the current feature dimension and the preset target feature dimension; and compress the dimensions of the multiple input feature vectors based on the dimension reduction neural network to obtain multiple compressed vectors.

[0106] In summary, this application can effectively improve the accuracy of feature extraction and the performance of the model by combining the attention mechanism and the memory mechanism. These improvements enable the model to better capture key information and improve the accuracy of detection and prediction when processing tasks such as network anomaly detection and time series prediction.

[0107] In some embodiments, the feature processing method in the embodiment of the present application can be completed using data processing devices including but not limited to large-scale deep learning equipment, edge computing equipment, mobile phones or personal computers; in order to verify the effectiveness of the method, the embodiment of the present application carried out comparative experiments on the open source data sets MSL, SMD, SWaT, SMAP, and PSM respectively, and used out-of-distribution perception tasks to evaluate feature quality. In order to eliminate randomness, the average results reported after all algorithms were run five times. The experimental results show that regardless of the feature processing method based on attention and memory mechanisms in the present application, it can effectively improve the perception task performance of the model.

[0108] Some embodiments of the present application provide a computer device, Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application, referring to Fig.10 The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the feature processing method based on the attention and memory mechanism of any of the above embodiments is implemented, for example, the above described Figure 1 Steps S110 to S160 of the method, Figure 2 Steps S210 to S250 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 In method step S410, Figure 5 In method step S510, Figure 6 Steps S610 to S620 of the method, Figure 7 The method comprises steps S710 to S730.

[0109] Among them, it can be understood that the computer device in the present application can extract features from the original data by executing the above method to perform local feature vector extraction based on the attention mechanism, and use the memory mechanism on the basis of the attention mechanism to extract the global feature vector, and reconstruct the input data to obtain reconstructed data, so that the corresponding target deep learning unit can perform various downstream tasks such as classification, regression, and anomaly detection based on the reconstructed data. It can be understood that since the memory mechanism in the embodiment of the present application can effectively handle the long-term dependency problem in the data and can be applicable to multivariate time series and anomaly detection scenarios, the target deep learning unit can effectively capture the long-term dependency relationship in the long-term input data such as network traffic behavior based on the reconstructed data when performing long-term dependent training and reasoning tasks such as network anomaly detection, improve the modeling ability of long-term dependency, and improve the potential perception performance of the corresponding deep learning network model state, so as to effectively perform the corresponding training and reasoning tasks, for example, effectively identify potential security threats in network anomaly detection tasks, and be able to cope with complex and changeable network environments and attack methods.

[0110] The computer device 1000 of the embodiment of the present application includes one or more processors 1010 and a memory 1020. Fig.10 In the figure, a processor 1010 and a memory 1020 are taken as an example.

[0111] The processor 1010 and the memory 1020 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0112] The memory 1020, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1020 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1020 may optionally include a memory 1020 remotely disposed relative to the processor 1010, and these remote memories may be connected to the computer device 1000 via a network. At the same time, examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0113] In some embodiments, when the processor executes the computer program, the feature processing method based on the attention and memory mechanism of any of the above embodiments is performed at preset intervals.

[0114] Those skilled in the art will understand that Fig.10The device structure shown in the figure does not constitute a limitation on the computer device 1000, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0115] exist Fig.10 In the computer device 1000 shown, the processor 1010 can be used to call the feature processing method based on the attention and memory mechanism stored in the memory 1020, so as to implement the feature processing method based on the attention and memory mechanism.

[0116] Based on the hardware structure of the above-mentioned computer device 1000, various embodiments of the feature processing device based on attention and memory mechanism of the present application are proposed. At the same time, the non-transient software program and instructions required to implement the feature processing method based on attention and memory mechanism of the above-mentioned embodiment are stored in the memory. When executed by the processor, the feature processing method based on attention and memory mechanism of the above-mentioned embodiment is executed.

[0117] The present application also provides a computer-readable storage medium storing computer-executable instructions for executing the above-mentioned feature processing method based on attention and memory mechanism, which can enable the above-mentioned one or more processors to execute the feature processing method based on attention and memory mechanism of any of the above-mentioned embodiments, for example, to execute the above-mentioned Figure 1 Steps S110 to S160 of the method, Figure 2 Steps S210 to S250 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 In method step S410, Figure 5 In method step S510, Figure 6 Steps S610 to S620 of the method, Figure 7 The method steps S710 to S730 in the above-described device embodiment are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network nodes. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer-readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD to ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery media. The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A feature processing method based on attention and memory mechanism, characterized in that: include: Obtain the input data corresponding to the current moment of the target deep learning unit; Encoding the data to be input to obtain a plurality of input feature vectors, and dimensionally compressing the plurality of input feature vectors to obtain a plurality of compressed vectors; Performing attention matrix reconstruction training based on the multiple compressed vectors to obtain a query projection matrix, a key projection matrix, and a value projection matrix; Calculating attention weights corresponding to a plurality of the compressed vectors based on the query projection matrix, the key projection matrix, and the value projection matrix, and obtaining a local feature vector according to the plurality of the attention weights; A memory matrix is ​​generated according to the historical input data corresponding to the target deep learning unit at the historical moment, a plurality of global feature vectors corresponding to the local feature vector are calculated based on the memory matrix, and a plurality of global feature vectors are weightedly calculated to obtain a plurality of memory feature vectors; The multiple memory feature vectors are subjected to feature fusion and dimensionality reduction to obtain reconstructed data, and the reconstructed data is used as current input data at a current moment to train the target deep learning unit.

2. The feature processing method based on attention and memory mechanism according to claim 1, characterized in that: The calculating the attention weights corresponding to the plurality of compressed vectors based on the query projection matrix, the key projection matrix and the value projection matrix comprises: Calculate query vectors corresponding to the plurality of compressed vectors according to the query projection matrix; Calculate key vectors corresponding to a plurality of the compression vectors according to the key projection matrix; Calculate the value vectors corresponding to the plurality of compressed vectors according to the value projection matrix; For each of the compressed vectors, perform a dot product operation according to a transposed matrix of the query vector and the key vector to obtain a query-key similarity index; The plurality of query key similarity indexes are normalized to obtain the attention weight corresponding to each of the compression vectors.

3. The feature processing method based on attention and memory mechanism according to claim 2, characterized in that: The obtaining of a local feature vector according to the plurality of attention weights comprises: For each of the compressed vectors, perform a dot product operation according to the corresponding value vector and the attention weight to obtain an attention feature; A local feature vector is obtained according to the attention feature corresponding to each of the compression vectors.

4. The feature processing method based on attention and memory mechanism according to claim 1, characterized in that: The memory matrix includes multiple memory units, each of which contains a historical feature vector generated based on the historical input data. The number of the memory units and the feature dimensions of the historical feature vector are determined based on the input data scale of the target deep learning unit.

5. The feature processing method based on attention and memory mechanism according to claim 4, characterized in that: The calculating, based on the memory matrix, a plurality of global eigenvectors corresponding to the local eigenvectors comprises: For each of the memory units in the memory matrix, a dot product operation is performed according to the transposed matrix of the local feature vector and the historical feature vector to obtain a global feature vector.

6. The feature processing method based on attention and memory mechanism according to claim 5, characterized in that: The weighted calculation of the multiple global feature vectors to obtain multiple memory feature vectors includes: For each of the memory units in the memory matrix, a dot product operation is performed according to the global feature vector and the historical feature vector to obtain a memory feature vector.

7. The feature processing method based on attention and memory mechanism according to claim 6, characterized in that: The step of performing feature fusion and dimensionality restoration on the plurality of memory feature vectors to obtain reconstructed data includes: Perform feature fusion according to the memory feature vectors corresponding to the plurality of memory units to obtain a fused feature vector; The fusion feature vector is dimensionally restored based on the data dimension corresponding to the data to be input to obtain reconstructed data.

8. The feature processing method based on attention and memory mechanism according to claim 1, characterized in that: The encoding process of the input data to obtain a plurality of input feature vectors includes: Preprocessing the data to be input to obtain preprocessed data; Encoding the preprocessed data to obtain a plurality of input feature vectors; The preprocessing includes normalization processing or standardization processing.

9. The feature processing method based on attention and memory mechanism according to claim 1, characterized in that: The step of performing dimension compression on the plurality of input feature vectors to obtain a plurality of compressed vectors includes: Obtaining current feature dimensions corresponding to a plurality of the input feature vectors; Determine a dimensionality reduction neural network according to the current feature dimension and a preset target feature dimension; Based on the dimension reduction neural network, the dimensions of the multiple input feature vectors are compressed respectively to obtain multiple compressed vectors.

10. The feature processing method based on attention and memory mechanism according to any one of claims 1 to 9, characterized in that: The target deep learning unit is used to perform network anomaly detection, and the data to be input includes network traffic data used to perform the network anomaly detection.

11. A feature processing device based on attention and memory mechanism, characterized in that: include: A data encoding module is used to obtain the input data corresponding to the target deep learning unit at the current moment; Encoding the data to be input to obtain a plurality of input feature vectors, and dimensionally compressing the plurality of input feature vectors to obtain a plurality of compressed vectors; An attention module is used to perform attention matrix reconstruction training based on the multiple compressed vectors to obtain a query projection matrix, a key projection matrix and a value projection matrix; calculate attention weights corresponding to the multiple compressed vectors based on the query projection matrix, the key projection matrix and the value projection matrix, and obtain a local feature vector according to the multiple attention weights; A memory module is used to generate a memory matrix according to historical input data corresponding to the target deep learning unit at historical moments, calculate multiple global feature vectors corresponding to the local feature vector based on the memory matrix, and perform weighted calculation on the multiple global feature vectors to obtain multiple memory feature vectors; perform feature fusion and dimensionality restoration on the multiple memory feature vectors to obtain reconstructed data, and the reconstructed data is used to train the target deep learning unit as current input data at the current moment.

12. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the feature processing method based on the attention and memory mechanism as described in any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the feature processing method based on attention and memory mechanism as described in any one of claims 1 to 10.

Images