Intelligent management method for multi-source data based on deep learning

Abstract

The application discloses a multi-source data intelligent management method based on deep learning, and belongs to the technical field of data intelligent management, and comprises the following steps: acquiring a multi-source data stream and cutting the multi-source data stream into data blocks according to time windows, extracting semantic feature vectors of the data blocks by using a first deep neural network; dividing data fragments according to local density peaks of the semantic feature vectors, and generating metadata labels containing source identifiers and fragment serial numbers; fusing the metadata labels and data distribution of the data fragments by using a second deep neural network, and outputting a protection level, an access control list and a storage node mapping; sending the data fragments to storage nodes for encrypted storage according to the mapping, and writing the access control list into an access control manager; and providing decrypted data after authenticating an access request according to the access control list. The method improves the adaptability and security of multi-source data management through semantic-driven fragmentation and dynamic security strategy mapping.

Description

Technical Field

[0001] This invention relates to the fields of deep learning technology and data management technology, specifically to a multi-source data intelligent management method based on deep learning. Background Technology

[0002] Multi-source data intelligent management faces massive heterogeneous data streams from different sensors, business systems, and log platforms, requiring a balance between efficient storage, granular access control, and data security. Existing technologies typically segment data streams using fixed time windows or preset data volume thresholds. These segments are then simply labeled with timestamps or sequence numbers, and security management is implemented using statically configured protection levels and centralized access control lists. Storage node selection often relies on round-robin or load-based scheduling algorithms, without being linked to the characteristics of the data itself or security requirements.

[0003] This approach has significant drawbacks. Fixed partitioning rules ignore the semantic relationships between data content. Semantically similar or logically coherent data blocks are split into different shards, requiring upper-level analysis tasks to reassemble the context across shards, increasing processing overhead and weakening the contextual consistency of security controls. Static protection strategies are manually preset based on general data sources and cannot adaptively adjust according to dynamic attributes such as the internal structure of data blocks, information entropy, and the degree of source mixing. This can easily lead to overprotection or underprotection when dealing with multi-source heterogeneous data. Metadata tags only record simple sources and sequence numbers, failing to reflect the data distribution characteristics within shards, thus making it difficult to support fine-grained differentiated storage and access decisions.

[0004] In a multi-source data environment, how to align the formation of data fragments with the semantic distribution of the data itself to avoid information fragmentation; and how to adaptively generate protection levels, access control lists, and storage node mappings based on the source identifiers and internal statistical distributions of data fragments are urgent problems to be solved. Summary of the Invention

[0005] This paper presents a deep learning-based intelligent management method for multi-source data to address the issues of semantic gaps and static security policies in existing sharding methods, thereby enabling semantically driven adaptive data sharding and dynamic security mapping and control.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a multi-source data intelligent management method based on deep learning, comprising:

[0007] Acquire multi-source data streams, divide them into data blocks according to time windows, and use a first deep neural network to extract semantic feature vectors of the data blocks;

[0008] Based on the local density peaks of the semantic feature vectors, the data block is divided into multiple data fragments, and metadata tags containing source identifiers and fragment sequence numbers are generated for each data fragment.

[0009] The second deep neural network is used to fuse the metadata tags and the data distribution of the corresponding data shards, and outputs the protection level, access control list and storage node mapping of the data shards;

[0010] Data fragments are sent to storage nodes according to storage node mapping, data fragments are encrypted on storage nodes according to protection level, and access control lists are written to the access control manager of storage nodes.

[0011] Upon receiving a data fragment access request from a target user, the access control manager authenticates the target user according to the access control list. Once authentication is successful, the decrypted data fragment is provided to the target user.

[0012] As a preferred embodiment of the present invention, the process of extracting semantic feature vectors from a data block using a first deep neural network includes: encoding the numerical fields, text fields, and time-series fields in the data block into a first sub-vector, a second sub-vector, and a third sub-vector, respectively; concatenating the first sub-vector, the second sub-vector, and the third sub-vector in the field order to form an original input vector; inputting the original input vector into a first deep neural network composed of three cascaded gated residual networks; each gated residual network first performs a linear transformation on the current input to obtain a gate value, then multiplies the gate value element-wise with the current input and adds it to the current input to obtain the output of that layer; and performing vector normalization on the output of the last gated residual network to obtain the semantic feature vector. By combining multi-field differential encoding with a gated residual structure, heterogeneous field information is effectively integrated and semantic expression is enhanced, making the extracted semantic feature vectors better reflect the inherent semantics of the data block and providing a reliable foundation for subsequent density clustering.

[0013] As a preferred technical solution of the present invention, the process of dividing the data block into multiple data fragments based on the local density peaks of the semantic feature vector includes: performing principal component analysis on the semantic feature vector, retaining the first few principal components to obtain dimensionality-reduced feature points; taking each feature point as the center, counting the number of neighbors of the feature point within a preset radius as the local density, and calculating the minimum distance between the feature point and feature points with a local density higher than the feature point; marking feature points with a local density higher than a preset density threshold and a minimum distance greater than a preset distance threshold as density peaks; using each density peak as the starting point, dividing the data blocks closest to the density peak and unassigned into the same temporary cluster; sorting the data blocks in the temporary cluster according to the source timestamp order of the data blocks, and marking the sorted temporary clusters as the data fragments. The density peak-driven fragmentation method can discover natural clustering structures in multi-source data in an unsupervised manner, making the fragmentation boundary consistent with the distribution characteristics of the data itself, which reduces the cost of manual partitioning and maintains the temporal continuity between data blocks.

[0014] As a preferred technical solution of the present invention, the process of generating metadata tags containing source identifiers and fragment sequence numbers for data fragments includes: extracting the source data source numbers of all data blocks within each data fragment, and using the source data source number with the highest frequency as the source identifier; calculating the sorting position of each data fragment among all data fragments, and using the sorting position as the fragment sequence number; concatenating the source identifier and the fragment sequence number into a string, and using a hash function to calculate the string and the timestamp of the first data block of the data fragment to obtain a checksum; and writing the source identifier, the fragment sequence number, the checksum, and the timestamp together into the metadata tag. The metadata tag generated in this way not only identifies the source attribution and sequential position of the fragment, but also provides tamper-proof protection through the checksum, facilitating rapid retrieval and integrity verification during subsequent storage and access.

[0015] As a preferred technical solution of the present invention, the process of fusing the metadata tags and the data distribution of corresponding data shards using a second deep neural network includes: one-hot encoding the source identifier and the shard number in the metadata tags, and forming a statistical vector from the length, type, and entropy value of each data block within the corresponding data shard; concatenating the one-hot encoded source identifier, the one-hot encoded shard number, and the statistical vector along the feature dimension to obtain the fusion input; feeding the fusion input into a second deep neural network composed of a bidirectional long short-term memory network and an attention layer, wherein the bidirectional long short-term memory network outputs a bidirectional hidden state sequence; and the attention layer performs a weighted summation of the bidirectional hidden state sequence according to the attention weight at each position in the bidirectional hidden state sequence to obtain a fused feature vector. This technical solution deeply fuses the discrete attributes of the metadata tags with the statistical features of the data distribution within the shards, captures the sequential dependencies between data blocks through the bidirectional long short-term memory network, and adaptively highlights key information through an attention mechanism, enabling the fused feature vector to finely characterize the security requirements and access patterns of the shards, providing high-quality feature representation for intelligent decision-making.

[0016] As a preferred technical solution of the present invention, the process of outputting the protection level, access control list, and storage node mapping of data sharding includes: inputting the fused feature vector into a first classification head, a second classification head, and a third classification head respectively; the first classification head uses a softmax function to map the fused feature vector, and takes the protection level corresponding to the maximum probability in the mapping result as the protection level; the second classification head calculates the cosine similarity between the fused feature vector and the pre-configured permission candidate templates, selects the three permission candidate templates with the highest cosine similarity, and merges the three selected permission candidate templates into the access control list, which includes user identifier, operation type, and effective time period; the third classification head performs linear mapping on the fused feature vector to obtain the score of each candidate storage node, takes the candidate storage node with the highest score as the target storage node in the storage node mapping, and records the node number of the target storage node in the metadata tag. Through parallel processing by multi-task classification heads, the required protection strength, specific access control rules, and optimal storage location for sharding can be output simultaneously, realizing the coordinated optimization of security policies and resource scheduling, shortening the decision-making chain, and improving response efficiency.

[0017] As a preferred embodiment of the present invention, the process of sending data fragments to storage nodes according to storage node mapping includes: querying the global service directory to obtain the network address of the target storage node based on the node number of the target storage node in the storage node mapping; encapsulating the data fragments into transmission messages, wherein the header of the transmission message carries the metadata tag and the payload of the transmission message carries the data fragments; sending the transmission message to the network address using a transport layer security protocol; recording the sending time, destination node, and message length of each data fragment in real time and generating a sending log; and updating the sending log to the acknowledged status after the target storage node returns a receipt confirmation. This process ensures the integrity and confidentiality of the fragments on the transmission link through metadata tag carrying and encrypted transmission, while the real-time recording and confirmation mechanism of the sending log enhances the auditability and reliable delivery capability of the transmission process.

[0018] As a preferred technical solution of the present invention, the process of encrypting data fragments according to protection levels on storage nodes includes: when the protection level is level 1, encrypting the data fragments on the target storage node using a symmetric encryption algorithm; when the protection level is level 2, encrypting the data fragments on the target storage node using an attribute-based encryption algorithm, and binding the attribute key generated by the attribute-based encryption algorithm to the user identifier in the access control list; when the protection level is level 3, encrypting the data fragments on the target storage node using a homomorphic encryption algorithm, and storing the homomorphic public key generated by the homomorphic encryption algorithm in the public key certificate of the target storage node; dividing the encrypted data fragments into multiple redundant blocks, distributing the multiple redundant blocks to different disk partitions within the target storage node according to an erasure coding strategy, and recording the correspondence between each redundant block and the disk partition. Dynamically selecting symmetric encryption, attribute-based encryption, or homomorphic encryption according to the protection level can achieve a balance between security strength and computational overhead; combined with redundant block division and erasure coding strategy, even if some disk partitions fail, data fragments can be recovered through the remaining redundant blocks, significantly improving data durability and availability.

[0019] As a preferred embodiment of the present invention, the process of writing the access control list (ACL) to the access control manager of the storage node includes: serializing the ACL into a JSON structure, and adding the protection level and metadata tag of the data shard to the JSON structure; calculating the hash value of the JSON structure, and using the hash value as the version identifier of the ACL; writing the ACL and the version identifier into the access control manager of the target storage node in key-value pairs; the access control manager synchronizing the version identifier to a backup node connected to the target storage node, and the backup node updating the ACL copy in its local cache according to the version identifier. By attaching the protection level and metadata tag and calculating the version identifier, the binding relationship between the ACL and the specific data shard is ensured, and the consistency of the access control policy between the storage node and the backup node is maintained through the version synchronization mechanism, preventing unauthorized access caused by configuration inconsistencies.

[0020] As a preferred embodiment of the present invention, the process by which the access control manager authenticates a target user according to an access control list includes: the access control manager receiving an access token sent by the target user, the access token containing the target user's user identifier and request operation type; searching the access control list for a first permission entry matching the user identifier; determining whether the set of operation types in the first permission entry contains the request operation type; if the set of operation types contains the request operation type, further checking whether the current time is within the valid time period in the access control list, and checking whether the target user's trust value is greater than a preset trust threshold; if the current time is within the valid time period and the trust value is greater than the preset trust threshold, then outputting the decryption key of the data shard to the target user, and writing the target user's access behavior into the audit log. Through multi-condition joint verification of user identifier, operation type, valid time period, and trust value, a fine-grained dynamic authentication mechanism is formed. Only users who simultaneously meet the requirements of legitimate identity, authorized operation, valid time, and trustworthy behavior can obtain the decryption key, thus balancing data security and compliance audit requirements.

[0021] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0022] The semantic feature vectors of data blocks are extracted using a first deep neural network, and the data blocks are divided into data shards based on the local density peaks of the semantic feature vectors. Specifically, numerical fields, text fields, and time-series fields within the data block are encoded into sub-vectors and concatenated, then input into a three-layer gated residual network, outputting normalized semantic feature vectors. Principal component analysis is performed on the semantic feature vectors for dimensionality reduction, calculating the local density of each feature point and the minimum distance to higher density points. Density peak points are marked, and temporary clusters are formed starting from the density peaks. Data shards are then generated by sorting by source timestamp. This scheme ensures that the shard boundaries are consistent with the inherent semantic clustering structure of the data. Semantically similar and closely related data blocks are aggregated within the same shard, eliminating information fragmentation caused by conventional fixed-window segmentation and improving the contextual integrity of data within shards. In subsequent encrypted storage and access control, the homogeneity of data within the same shard makes protection operations more targeted and reduces the overhead of cross-shard relational queries.

[0023] A second deep neural network is introduced to fuse metadata tags and data distribution across data shards, outputting protection levels, access control lists, and storage node mappings. The source identifier and shard number from the metadata tags are one-hot encoded and concatenated with a statistical vector composed of the length, type, and entropy value of data blocks within the shard to obtain the fused input. A bidirectional long short-term memory network and an attention layer are used to serialize and weight the fused input, generating a fused feature vector. This fused feature vector is fed into the first, second, and third classification heads. The first classification head outputs the protection level through softmax mapping; the second classification head calculates the cosine similarity with pre-configured permission candidate templates and merges the first three items to generate an access control list; and the third classification head scores candidate storage nodes to determine the target storage node. This approach directly links the source composition and internal statistical characteristics of data shards with security requirements, enabling end-to-end autonomous decision-making from data characteristics to security policies and storage locations. The protection level dynamically changes with the heterogeneity and sensitive characteristics of the fragmented data. Access control lists automatically generate fine-grained user identifiers, operation types, and valid time periods. Storage node mapping takes into account both data protection requirements and storage resource status, replacing manually preset fixed strategies. The resulting dynamic security boundary can adapt to the continuous changes in multi-source data streams, maintaining fine-grained protection and reasonable storage distribution for data fragments without manual intervention. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0025] Figure 1 This is a flowchart of a multi-source data intelligent management method;

[0026] Figure 2 This is a flowchart of data partitioning based on density peak clustering;

[0027] Figure 3 This is a flowchart of parallel processing of multi-classification heads with fused feature vectors and secure data transmission.

[0028] Figure 4 This is the loss curve during the training process of the first deep neural network. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] See Figure 1 This invention provides a deep learning-based intelligent management method for multi-source data. The method includes: acquiring multi-source data streams; dividing the multi-source data streams into data blocks according to time windows; extracting semantic feature vectors for each data block using a first deep neural network; clustering the data blocks into multiple data fragments based on the local density peaks of each semantic feature vector; generating metadata tags for each data fragment containing a source identifier and fragment sequence number; fusing the metadata tags and the data distribution of the corresponding data fragments using a second deep neural network; outputting the protection level, access control list, and storage node mapping of the data fragments; sending the data fragments to the corresponding target storage nodes according to the storage node mapping; encrypting the data fragments on the target storage nodes according to the protection level; and writing the access control list to the access control manager of the target storage node; when receiving an access request from a target user for a data fragment, the access control manager authenticates the target user according to the access control list, and after successful authentication, provides the decrypted data fragment to the target user.

[0031] In practice, after acquiring the multi-source data stream, the multi-source data stream is divided into data blocks according to a fixed time window. The length of the time window is set based on the average arrival rate of data in the multi-source data stream and the downstream processing latency requirements, for example, set to 5 seconds. All data arriving within each time window constitutes a data block.

[0032] For each data block, the semantic feature vector is extracted as follows: Numerical fields, text fields, and time-series fields are identified within the data block. Numerical fields are normalized, linearly mapping each value to the [0,1] interval; the mapped numerical sequence serves as the first sub-vector. Text fields are encoded using a pre-trained word embedding model, converting them into fixed-length floating-point vectors, which serve as the second sub-vector. For time-series fields, the interval sequence between timestamps is extracted, and the mean, variance, and the first three peaks of the autocorrelation function of the interval sequence are calculated as periodic features. The statistical moments of the interval sequence and the periodic features are concatenated into a fixed-length vector, which serves as the third sub-vector. Following the original order of appearance of numerical, text, and time-series fields in the data block, the first, second, and third sub-vectors are concatenated end-to-end to form the original input vector.

[0033] The first deep neural network consists of three cascaded gated residual networks. These three gated residual networks have the same structure, with the input vector dimension and output vector dimension being the same for each layer. For the... Layer-gated residual network, receiving the first Output vector of layer-gated residual network As input, when hour, This is the original input vector. (The...) The computation process of the layer-gated residual network is represented as follows:

[0034]

[0035] in, Indicates the first The vector output by the layer-gated residual network. Indicates the input number Vectors of layer-gated residual networks Indicates the first The weight matrix of the layer-gated residual network has the same number of rows and columns. Dimensions Indicates the first The bias vector of a layer-gated residual network, the dimension of the bias vector and The same dimensions This represents the sigmoid function. This represents element-wise multiplication. The sigmoid function maps the result of a linear transformation to the (0,1) interval, forming gated weights; the gated weights and... After element-wise multiplication, the pair Selectively scale each component in the data, and then combine it with... The residual connections are achieved through addition. The input to the first-layer gated residual network is the original input vector, the output of the first-layer gated residual network is used as the input to the second-layer gated residual network, and the output of the second-layer gated residual network is used as the input to the third-layer gated residual network.

[0036] The vector output by the third-layer gated residual network needs further vector normalization to obtain the semantic feature vector. During normalization, the L2 norm of the output vector of the third-layer gated residual network is calculated, and each element in the output vector is divided by the L2 norm. The resulting vector is the semantic feature vector of the data block.

[0037] In practice, the first deep neural network needs to be trained before extracting semantic feature vectors. During training, a large number of multi-source data block samples are collected, and each multi-source data block sample is encoded separately to obtain the original input vector as described above. The randomly obscured fields in the reconstructed data block are used as the training task. The original input vector is input into the first deep neural network to obtain the semantic feature vector, and then the semantic feature vector is input into a linear decoder to reconstruct the content of the obscured fields. The mean squared error between the reconstructed result and the true field is calculated as the loss function. The Adam optimizer is used to update the parameters of the first deep neural network and the linear decoder with a learning rate of 0.001. Training stops when the loss function value no longer decreases over several consecutive training epochs. The weight matrix and bias vector of the first deep neural network are then fixed for extracting semantic feature vectors.

[0038] See Figure 4 In the graph, the horizontal axis represents the training period, ranging from 0 to 512, and the vertical axis represents the mean squared error loss value, ranging from 0.0 to 3.0. The blue solid line represents the training loss, and the red dashed line represents the validation loss. As the training period increases, both the training loss and validation loss show a significant decreasing trend. In the initial stage (0-100 periods), the loss values ​​rapidly decrease from approximately 2.5 and 2.9 to around 0.7 and 0.9, respectively. In the intermediate stage (100-300 periods), they continue to decrease steadily, approaching the 0.1-0.3 range. In the later stage (300-512 periods), the loss values ​​tend to stabilize, with the training loss eventually stabilizing at approximately 0.05 and the validation loss at approximately 0.1. The training loss curve is consistently lower than the validation loss curve, and their trends are consistent, indicating that the first deep neural network effectively learned the data block reconstruction task during the semantic feature vector extraction training process. The loss function value gradually converged, and no significant overfitting occurred. The stability of the validation loss indicates that the model has good generalization ability. The figure illustrates the training process of the first deep neural network in Example 1 using the masked field reconstruction task, verifying that the network structure and training method can effectively extract semantic feature vectors from multi-source data blocks.

[0039] In specific implementation, please refer to Figure 2 After obtaining the semantic feature vectors of all data blocks, principal component analysis is performed on all semantic feature vectors. First, all semantic feature vectors are arranged row-wise to construct a sample matrix, with the dimensions of the semantic feature vectors constituting the number of columns in the sample matrix. The covariance matrix of the sample matrix is ​​calculated, and eigenvalue decomposition is performed on the covariance matrix to obtain each eigenvalue and its corresponding eigenvector. The eigenvalues ​​are sorted from largest to smallest, and the contribution rate of each eigenvalue is calculated, i.e., the eigenvalue divided by the sum of all eigenvalues. The contribution rates of multiple eigenvalues ​​are accumulated from largest to smallest to obtain the cumulative variance contribution rate. A cumulative variance contribution rate threshold of 0.95 is set, and the first few principal components whose cumulative variance contribution rates just exceed 0.95 are selected. The eigenvectors corresponding to these principal components are retained as the projection matrix. The original semantic feature vectors are multiplied by the projection matrix to obtain the dimensionality-reduced feature points, each feature point corresponding to a data block.

[0040] For each feature point, calculate its local density. For example, statistics at feature points preset radius The number of other feature points contained within the range is used as the feature point count. The local density, using The formula for calculating local density is as follows:

[0041]

[0042] in, Representing feature points Local density, Indicates except Any other feature point besides, Representing feature points With feature points The Euclidean distance between them Indicates the preset radius. Represents a step function, when the independent variable... hour When the independent variable hour Preset radius The value is taken as the 20th percentile of the set of Euclidean distances between all pairwise feature points. This value is obtained by calculating the distances between all feature point pairs, obtaining a distance set, sorting this distance set in ascending order, and taking the value at the 20th percentile position after sorting. This value allows feature points to have a high local density in high-density regions, while avoiding the influence of extreme values ​​on the preset radius.

[0043] For each feature point, calculate the minimum distance to that feature point. In all local densities higher than Among other feature points, find the feature point Record the feature point that is closest to it using Euclidean distance as the feature point. The minimum distance, using Indicate. If feature points If the local density is already the maximum value among all feature points, then the feature points will be... minimum distance Set to the maximum value of the Euclidean distance between all feature points.

[0044] When determining the density peak, a preset density threshold and a preset distance threshold are set. The preset density threshold is the median of the local densities of all feature points, and the preset distance threshold is the average of the minimum distances of all feature points. The local density of each feature point is then calculated. Compare with a preset density threshold, and simultaneously minimize its distance. Compare with a preset distance threshold. When the local density of a feature point... The density is higher than the preset density threshold, and the minimum distance of the feature point is... When the distance exceeds a preset threshold, the feature point is marked as a density peak. The density peak corresponds to the initial cluster center of the data shard.

[0045] When constructing temporary clusters, each density peak is used as the starting point of a temporary cluster, and each density peak constitutes a separate temporary cluster. For each unassigned feature point that does not belong to any density peak, the Euclidean distance between the feature point and each density peak is calculated, and the feature point is assigned to the temporary cluster containing the nearest density peak. After this assignment process, all data blocks are divided into a temporary cluster.

[0046] After the temporary clusters are partitioned, the data blocks within each temporary cluster are sorted in ascending order according to their source timestamps. The source timestamp of a data block is recorded in the data block's metadata and serves as a timestamp indicating when the data block was acquired. The sequence of data blocks within the temporary cluster is then rearranged according to their timestamps, and the sorted temporary clusters are output as a single data shard. Ultimately, all data blocks are organized into multiple data shards, each consisting of several temporally consecutive and semantically similar data blocks.

[0047] In practice, when generating metadata tags for data shards, all data blocks within the data shard are traversed. For each data block, the source data source ID is extracted. The source data source ID is a globally unique numerical identifier of the data source system that generated the data block. The frequency of each source data source ID within the current data shard is counted, and the source data source ID with the highest frequency is used as the source identifier for that data shard. If there are multiple source data source IDs with the highest frequency, the one with the smallest value is selected as the source identifier for the data shard. Simultaneously, after all data shards are partitioned, their sorting position within the entire dataset is determined according to their chronological or identifier order, and the integer value of this sorting position is used as the shard sequence number. The starting value for the sorting position is 1.

[0048] The source identifier and the fragment number are converted into strings, and then concatenated end-to-end to obtain a concatenated string. The timestamp of the first data block in the data fragment is extracted; this timestamp is the UNIX timestamp used when the data block was collected, accurate to milliseconds. A preset hash function is used to calculate the checksum from the concatenated string and the timestamp. The preset hash function uses the SHA-256 algorithm. The calculation method is as follows: the concatenated string and the timestamp string are connected by a hyphen "-", which serves as the input message for the SHA-256 algorithm. The first eight hexadecimal characters of the hash output digest are used as the checksum. The calculated source identifier, fragment number, checksum, and timestamp of the first data block are written as key-value pairs into the metadata tag corresponding to the data fragment.

[0049] When integrating metadata tags with the corresponding data shards, the source identifier in the metadata tag is first one-hot encoded. The total number of pre-configured data sources, N_s, is obtained, and a vector of length N_s containing all zeros is constructed. The data source index position corresponding to the source identifier is set to 1, resulting in the one-hot vector of the source identifier. When one-hot encoding the shard sequence number, the maximum sequence number M_f of the data shard is obtained, and a vector of length M_f+1 containing all zeros is constructed. The corresponding position of the shard sequence number is set to 1, resulting in the one-hot vector of the shard sequence number. If the shard sequence number exceeds the preset length, the vector length is dynamically expanded.

[0050] For the data distribution of data shards, a statistical vector for each data shard is constructed. All data blocks within a data shard are traversed, and for each block, its length, type, and entropy value are extracted. The length of a data block refers to the total number of bytes it contains. The type of a data block refers to the combination identifier of field types within the block; this combination is mapped to a predefined type encoding table to obtain a type integer value. The entropy value of a data block is calculated by determining the information entropy of the distribution of all bytes within the block, using the formula of summing the negative logarithms of the probabilities of each byte value appearing within the block. For the current data shard, the length, type, and entropy values ​​of all data blocks are organized into a sequence according to their timestamps. The feature corresponding to each time step consists of the length, type, and entropy value of that data block, and the entire sequence constitutes the statistical vector of the data shard.

[0051] When concatenating along the feature dimension, first obtain the number K of data blocks within the data slice. Copy the one-hot vector of the source identifier K times to form a tensor of shape (K, N_s); copy the one-hot vector of the slice index K times to form a tensor of shape (K, M_f+1); the statistical vector of the data slice has a shape of (K, 3). Concatenate these three tensors along the last dimension to obtain a fused input sequence of shape (K, N_s+M_f+1+3), where each row of the fused input sequence corresponds to the fused feature of one data block.

[0052] The second deep neural network consists of a bidirectional long short-term memory (LSTM) network and a cascaded attention layer. The bidirectional LSTM network comprises a forward LSTM layer and a backward LSTM layer, with each LSTM layer having a hidden state dimension of 128. The forward LSTM network processes the fused input sequence in ascending time order, outputting a forward hidden state sequence; the backward LSTM network processes the fused input sequence in reverse time order, outputting a backward hidden state sequence. The forward and backward hidden states at the same time step are concatenated to obtain the bidirectional hidden state sequence, the length of which is equal to the number of data blocks K, and the vector dimension corresponding to each time step is 256.

[0053] The attention layer weights the hidden states at each time step in the bidirectional hidden state sequence. The computation of the attention layer uses an additive attention mechanism based on a trainable attention parameter vector. A trainable weight matrix and bias vector For the first hidden state in the bidirectional hidden state sequence... Hidden state Calculate attention score The formula is:

[0054]

[0055] in, Represents the first hidden state in the bidirectional hidden state sequence. The hidden state vector at each time step Values ​​range from 1 to K; The weight matrix is ​​trainable and has a shape of (64, 256). The bias vector is a trainable vector with a dimension of 64; This is a trainable attention parameter vector with a dimension of 64; It is the hyperbolic tangent function. This represents the transpose operation. The scores for all hidden states are calculated. Then, the softmax function is used to convert the scores into attention weights. : The attention layer outputs the weighted sum as a fused feature vector. The dimension of the fused feature vector is 256.

[0056] In practice, the parameters of the second deep neural network need to be determined through training. During training, a large number of data shard samples are collected. Each sample contains a data shard and its corresponding metadata label, and each sample has been manually labeled with a corresponding protection level label, access control list template number set, and target storage node number. When constructing the training batch, a fused input sequence is generated for each data shard according to the above process, fed into the second deep neural network to obtain a fused feature vector, and then connected to the subsequent first, second, and third classification heads for joint training. The first classification head outputs the probability distribution of the protection level and calculates the cross-entropy loss with the protection level label; the second classification head outputs the cosine similarity with each permission candidate template and calculates the multi-label marginal loss with the access control list template number set; the third classification head outputs the score of the candidate storage node and calculates the cross-entropy loss with the target storage node number. The three loss functions are added with equal weights to obtain the total loss. The Adam optimizer is used for parameter updates, with an initial learning rate set to 0.0001 and a batch size set to 32. When the total loss no longer decreases for 10 consecutive training epochs on the validation set, training is stopped, and all trainable parameters of the second deep neural network and attention layer are fixed for extracting the fused feature vector.

[0057] In specific implementation, please refer to Figure 3 After the second deep neural network outputs the fused feature vector, the fused feature vector is input in parallel into the first classification head, the second classification head, and the third classification head.

[0058] The first classification head consists of a fully connected layer and a softmax function. The input dimension of the fully connected layer is the same as the dimension of the fused feature vector, and the output dimension is equal to the predefined total number of protection levels. Total number of protection levels The value is 3, representing the three protection levels: Level 1, Level 2, and Level 3. The fully connected layer contains a trainable weight matrix. and a trainable bias vector Weight matrix The shape is ,in The dimension of the fused feature vector is... Bias vector The length is For fused feature vectors The first classification head calculates the score vector. fractional vectors The first in The elements are denoted as The softmax function is used to transform the score vector into a probability distribution for each candidate protection level. The probability calculation formula is as follows:

[0059]

[0060] in, Indicates that the data fragment belongs to the first The probability value of each level of protection. For protection level index, , Represents the natural exponential function, with the denominator pair From 1 to Summation, This is a variable that iterates through all protection level indices. The calculation yields all... Then, the protection level corresponding to the maximum probability is selected as the protection level output for that data segment.

[0061] The second classification head performs template matching based on cosine similarity. Within the second classification head, pre-configured... One permission candidate template The value is set to 20. Each permission candidate template is encoded as a fixed-dimensional template vector, the dimension of which is the same as the fused feature vector. The dimensions are consistent, all being 256. Each permission candidate template carries a set of predefined permission entries, which include a set of user identifiers, a set of operation types, and a valid time period. The second classification head receives the fused feature vector. Calculate the fused feature vector The cosine similarity between the vector and each template vector. For the th... One permission candidate template, cosine similarity The calculation method is as follows: ,in Indicates the first template vectors, Represents the vector dot product. This represents the L2 norm of a vector. (Get all) After calculating the cosine similarity scores, the three permission candidate templates with the highest cosine similarity are selected. The permission entries contained in each of these three candidate templates are then extracted and merged. The merged set of permission entries forms an access control list. Each permission entry in the access control list contains a user identifier, a set of authorized operation types for that user identifier, and a valid time period. The operation types in the operation type set include read, write, copy, and delete.

[0062] The third classification head consists of a linear mapping layer. This linear mapping layer contains a trainable weight matrix. and a trainable bias vector Weight matrix The shape is ,in The bias vector represents the total number of candidate storage nodes. The length is Fusion of feature vectors After inputting the third classification header, the score vector is obtained. Rating vector The first in The value of the nth element is the nth element. The candidate storage nodes are scored. The candidate storage node with the highest score in the score vector is selected as the target storage node. The node number of the target storage node is written into the metadata tag of the data shard, thus completing the storage node mapping.

[0063] In practical implementation, the weight matrix of the first classification head and bias vector The template vector set in the second classification head and the weight matrix of the third classification head. and bias vector It is obtained through joint training with a second deep neural network. During training, a training dataset is constructed. After extracting fused feature vectors from each data slice, the cross-entropy loss between the protection level output by the first classification head and the manually labeled protection level, the multi-label marginal loss between the cosine similarity output by the second classification head and the manually labeled template number set, and the cross-entropy loss between the score output by the third classification head and the manually labeled target storage node number are calculated. After summing the three loss functions, the trainable parameters of the second deep neural network, the first classification head, the second classification head, and the third classification head are updated through backpropagation until the loss converges. The parameters after training are fixed and used for inference.

[0064] During the data fragment transmission process according to the storage node mapping, a query request is initiated to the global service directory based on the node number of the target storage node recorded in the storage node mapping. The global service directory is a centralized key-value storage mapping table that maintains the mapping relationship between the node number of each storage node and its network IP address and port number. After obtaining the network address of the target storage node, the data fragment is encapsulated into a transmission message. The header field of the transmission message contains the complete key-value pair of the metadata tag, including the source identifier, fragment sequence number, checksum, and timestamp; the payload field contains the binary content of the data fragment. The transmission message is sent to the network address of the target storage node using the Transport Layer Security (TLS) protocol, which provides encrypted transmission and authentication. During the data fragment transmission process, the sending time, destination node number, and message length of each data fragment are recorded in real time on the management node, generating a transmission log. The initial status of the transmission log is marked as "pending confirmation". When the target storage node successfully receives the data fragment and returns a reception confirmation message to the management node, the management node updates the status of the corresponding data fragment's transmission log to "confirmed", completing the secure transmission of the data fragment.

[0065] In practice, the target storage node receives transmission messages through the network interface, extracts metadata tags from the message header, and extracts the original content of the data fragments from the message payload. The target storage node reads the protection level of the data fragments from the metadata tags and selects the appropriate encryption method based on the specific value of the protection level.

[0066] When the protection level is Level 1, a symmetric encryption algorithm is used on the target storage node to encrypt the data fragments. The symmetric encryption algorithm uses AES-256-GCM. A 256-bit random symmetric key is generated, and the plaintext data fragments are encrypted using AES-256-GCM using the symmetric key to obtain ciphertext and authentication tags. The symmetric key is securely stored by the key management module of the target storage node. When the protection level is Level 2, an attribute-based encryption algorithm is used on the target storage node to encrypt the data fragments. The attribute-based encryption algorithm uses a ciphertext policy attribute-based encryption algorithm. During the ciphertext policy attribute-based encryption process, an access policy tree is generated based on the user attribute set corresponding to the user identifier of each permission entry in the access control list. The access policy tree is embedded in the encryption process to generate attribute-based ciphertext. The attribute-based encryption algorithm also generates a set of attribute keys, which correspond one-to-one with the attributes of the leaf nodes in the access policy tree. The generated attribute keys are bound one by one to the corresponding user identifiers in the access control list, and a mapping table from user identifiers to attribute keys is maintained internally on the target storage node. When the protection level is Level 3, a homomorphic encryption algorithm is invoked on the target storage node to encrypt the data fragments. The homomorphic encryption algorithm used is the Paillier homomorphic encryption algorithm. A Paillier key pair is generated, containing a Paillier public key and a Paillier private key. The plaintext data fragments are encrypted block by block using the Paillier public key to generate homomorphic ciphertext. The Paillier public key is encoded in X.509 certificate format and stored in the target storage node's local public key certificate store. The certificate body identifier is the checksum in the metadata tag of the corresponding data fragment. The Paillier private key is protected by the target storage node's hardware security module.

[0067] For each level of protection, the target storage node performs redundancy splitting and distributed storage operations on the encrypted data fragments. The encrypted data fragments are split into B original data blocks according to a preset block size of 4 megabytes. The target storage node employs an erasure coding strategy, using RS(10,4) erasure coding configuration, where 10 represents the number of data blocks and 4 represents the number of check blocks. The B original data blocks are encoded into a total of 14 redundant blocks, where B=10. These 14 redundant blocks are stored evenly across 14 different disk partitions within the target storage node; if the target storage node has fewer than 14 disk partitions, they are stored cyclically across partitions. For each redundant block written, the storage path, redundant block number, and disk partition identifier of the redundant block are recorded in the target storage node's storage mapping table, along with the hash value of the redundant block. The storage mapping table is implemented using a LevelDB key-value database, where the key is a string combining the data fragment's checksum and the redundant block number, and the value is a JSON structure containing the storage path and partition identifier.

[0068] In some embodiments, the process of writing the access control list (ACL) is as follows: The access control manager on the target storage node extracts the ACL from the header of the transmission message and then serializes it. The serialization process organizes all permission entries in the ACL into a JSON array structure. Each permission entry is a JSON object within the JSON array, containing three fields: "User Identifier," "Operation Type Set," and "Valid Time Period." Two top-level fields are added to the generated JSON structure: "Protection Level" and "Metadata Tag." The "Protection Level" field contains an integer value representing the protection level of the data shard, and the "Metadata Tag" field contains the complete JSON representation of the metadata tag. After constructing the JSON structure, a JSON serialization library is called to convert the JSON structure into an unformatted, compact string, denoted as [string 1]. .

[0069] Access Control Manager calculates strings The hash value. The hash function uses the SHA-256 algorithm. Hash value The calculation formula is:

[0070]

[0071] in, The version identifier of the access control list is a hexadecimal string of 64 characters. This indicates the Secure Hash Algorithm SHA-256; This represents the serialized JSON string, encoded in UTF-8. The calculated result... With access control lists The strings form key-value pairs, where the key is the checksum of the data fragment appended with the "_acl" suffix, and the value contains... and The JSON structure is used to write key-value pairs to the access control manager data store running locally on the target storage node. The access control manager data store is based on RocksDB.

[0072] The access control manager initiates a version identifier synchronization process between one or more backup nodes connected to the target storage node. The target storage node sends a synchronization message to each backup node via the gRPC protocol. This message includes a checksum for the data fragment and a latest version identifier. And the "PUT" command, representing the operation type. After receiving the synchronization message, the access control manager on the backup node extracts the checksum and version identifier of the data fragment. The version identifier of the same data shard is compared with that in the local cache of the backup node. If the local cache does not contain a record of the data shard, or if the version identifier in the local cache does not match... If the versions are different, the backup node sends a pull request to the target storage node to obtain the complete JSON structure of the access control list, and updates the local cache with the access control list copy and version identifier. If the version identifiers are the same, no update operation is performed.

[0073] When a target user initiates an access request to a data shard, the access control manager on the target storage node extracts the access token submitted by the target user from the network request. The access token is in JSONWebToken format and includes a payload portion, which carries the target user's user identifier and the request operation type. The user identifier is a globally unique identifier for the target user in the system, and the request operation type is a single operation, such as "read," "write," "copy," or "delete."

[0074] The access control manager reads the access control list for the data shard from the local data store. It iterates through all permission entries in the access control list, searching for the first permission entry whose "User ID" field exactly matches the target user's user ID; this is designated as the first permission entry. If no matching first permission entry is found, the access control manager directly denies the access request and logs it in the audit log. If a first permission entry is found, the access control manager retrieves the "Operation Type Set" field from this entry. This field is an array containing strings representing authorized operation types. The access control manager determines whether the requested operation type is included in the operation type set. If the requested operation type is not included in the operation type set, the access request is denied.

[0075] If the requested operation type belongs to the operation type set, the access control manager continues to retrieve the "Valid Time Period" field from the first permission entry. The valid time period includes a start time and an end time, formatted as an ISO8601 time string. The access control manager retrieves the current system time and determines whether the current system time is within the valid time period. If the current system time is not within the valid time period, the access request is denied.

[0076] Simultaneously, the access control manager obtains the target user's trust value. This trust value is stored in a centralized trust value service. The access control manager sends the target user's identifier to the trust value service via an HTTPS request. The trust value service returns an integer value between 0 and 100, which is used as the target user's trust value. The access control manager reads a preset trust threshold from its local configuration, which is set to 60. This preset trust threshold is based on the following: In the early stages of system risk assessment, 60 was used as a baseline to distinguish between trusted and low-trust users. The trust value service calculates a score based on the user's historical behavior records using a Beta trust model, with 60 corresponding to a medium or higher trust level. The access control manager determines whether the target user's trust value is greater than the preset trust threshold of 60. If the trust value is less than or equal to 60, the access request is denied.

[0077] If the current time is within a valid time period and the trust value is greater than the preset trust threshold, the access control manager outputs the decryption key for the corresponding data fragment to the target user. The method for obtaining the decryption key varies depending on the encryption method: for the first level, the symmetric key is retrieved from the key management module; for the second level, the attribute key bound to the user identifier is obtained by looking up the attribute key mapping table based on the target user's user identifier; for the third level, the Paillier private key is retrieved from the hardware security module. The access control manager sends the decryption key to the target user's client via a TLS encrypted channel. The access control manager writes detailed information about the target user's access behavior to the audit log. The audit log records include: the target user's user identifier, request operation type, access time, data fragment checksum, data fragment protection level, access result status code, and the operation record for the decryption key.

[0078] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A multi-source data intelligent management method based on deep learning, characterized in that, include: Acquire multi-source data streams, divide them into data blocks according to time windows, and use a first deep neural network to extract semantic feature vectors of the data blocks; Based on the local density peaks of the semantic feature vectors, the data block is divided into multiple data fragments, and metadata tags containing source identifiers and fragment sequence numbers are generated for each data fragment. The second deep neural network is used to fuse the metadata tags and the data distribution of the corresponding data shards, and outputs the protection level, access control list and storage node mapping of the data shards; Data fragments are sent to storage nodes according to storage node mapping, data fragments are encrypted on storage nodes according to protection level, and access control lists are written to the access control manager of storage nodes. Upon receiving a data fragment access request from a target user, the access control manager authenticates the target user according to the access control list. Once authentication is successful, the decrypted data fragment is provided to the target user.

2. The multi-source data intelligent management method based on deep learning according to claim 1, characterized in that, The step of extracting semantic feature vectors from data blocks using a first deep neural network includes: The numeric fields, text fields, and time-series fields in the data block are respectively encoded into a first sub-vector, a second sub-vector, and a third sub-vector; Concatenate the first sub-vector, the second sub-vector, and the third sub-vector according to the field order to form the original input vector; The original input vector is input into the first deep neural network, which is composed of three cascaded gated residual networks. Each gated residual network first performs a linear transformation on the current input to obtain a gate value, and then multiplies the gate value with the current input element by element and adds it to the current input to obtain the output of that layer. The output of the last gated residual network is vector normalized to obtain the semantic feature vector.

3. The multi-source data intelligent management method based on deep learning according to claim 1, characterized in that, The step of dividing the data block into multiple data fragments based on the local density peaks of the semantic feature vector includes: Principal component analysis is performed on the semantic feature vector, and the first few principal components are retained to obtain the feature points after dimensionality reduction. For each feature point, the number of its neighbors within a preset radius is counted as the local density, and the minimum distance between the feature point and feature points with a higher local density is calculated. Feature points whose local density is higher than a preset density threshold and whose minimum distance is greater than a preset distance threshold are marked as density peaks; Starting from each density peak, the unassigned data blocks closest to that density peak are grouped into the same temporary cluster; The data blocks within the temporary cluster are sorted according to their source timestamps, and the sorted temporary clusters are marked as the data fragments.

4. The multi-source data intelligent management method based on deep learning according to claim 1, characterized in that, The process of generating metadata tags containing source identifiers and fragment sequence numbers for data fragments includes: Extract the source data source number of all data blocks within each data shard, and use the source data source number that appears most frequently as the source identifier; Calculate the sorting position of each data shard among all data shards, and use the sorting position as the shard number; The source identifier and the fragment sequence number are concatenated into a string, and a hash function is used to calculate the verification code by combining the string with the timestamp of the first data block of the data fragment. The source identifier, the fragment sequence number, the checksum, and the timestamp are written together into the metadata tag.

5. The multi-source data intelligent management method based on deep learning according to claim 1, characterized in that, The data distribution that utilizes a second deep neural network to fuse the metadata tags and corresponding data fragments includes: The source identifier and the fragment sequence number in the metadata tag are one-hot encoded respectively, and the length, type and entropy value of each data block in the corresponding data fragment are combined into a statistical vector; The source identifier (one-hot encoded), the fragment number (one-hot encoded), and the statistical vector are concatenated along the feature dimension to obtain the fused input. The fused input is fed into a second deep neural network consisting of a bidirectional long short-term memory network and an attention layer, wherein the bidirectional long short-term memory network outputs a bidirectional hidden state sequence. The attention layer performs a weighted summation of the bidirectional hidden state sequence based on the attention weight at each position in the sequence to obtain a fused feature vector.

6. The multi-source data intelligent management method based on deep learning according to claim 5, characterized in that, The protection level, access control list, and storage node mapping of the output data fragments include: The fused feature vector is input into the first classification head, the second classification head, and the third classification head, respectively; The first classification head uses the softmax function to map the fused feature vector, and the protection level corresponding to the maximum probability in the mapping result is taken as the protection level; The second classification head calculates the cosine similarity between the fused feature vector and the pre-configured permission candidate templates, selects the three permission candidate templates with the highest cosine similarity, and merges the three selected permission candidate templates into the access control list, which includes user identifier, operation type, and valid time period; The third classification head performs a linear mapping on the fused feature vector to obtain a score for each candidate storage node. The candidate storage node with the highest score is taken as the target storage node in the storage node mapping, and the node number of the target storage node is recorded in the metadata tag.

7. The multi-source data intelligent management method based on deep learning according to claim 6, characterized in that, The step of sending data fragments to storage nodes according to storage node mapping includes: Based on the node number of the target storage node in the storage node mapping, query the global service directory to obtain the network address of the target storage node; The data fragments are encapsulated into a transmission message, the header of the transmission message carries the metadata tag, and the payload of the transmission message carries the data fragments. The transmission message is sent to the network address using a transport layer security protocol; Record the sending time, destination node, and message length of each data shard in real time, and generate a sending log; Once the target storage node returns a receipt confirmation, the sending log will be updated to a confirmed status.

8. The multi-source data intelligent management method based on deep learning according to claim 7, characterized in that, The process of encrypting data fragments according to protection levels on storage nodes includes: When the protection level is the first level, the data fragments are encrypted using a symmetric encryption algorithm on the target storage node; When the protection level is the second level, the data fragments are encrypted using an attribute-based encryption algorithm on the target storage node, and the attribute key generated by the attribute-based encryption algorithm is bound to the user identifier in the access control list. When the protection level is level three, the data fragments are encrypted using a homomorphic encryption algorithm on the target storage node, and the homomorphic public key generated by the homomorphic encryption algorithm is stored in the public key certificate of the target storage node. The encrypted data is fragmented into multiple redundant blocks, and the multiple redundant blocks are distributed to different disk partitions in the target storage node according to the erasure coding strategy. The correspondence between each redundant block and the disk partition is recorded.

9. The multi-source data intelligent management method based on deep learning according to claim 8, characterized in that, The step of writing the access control list to the access control manager of the storage node includes: The access control list is serialized into a JSON structure, and the protection level and metadata tag of the data fragment are added to the JSON structure. Calculate the hash value of the JSON structure and use the hash value as the version identifier of the access control list; Write the access control list and the version identifier into the access control manager of the target storage node in key-value pairs; The access control manager synchronizes the version identifier to the backup node connected to the target storage node, and the backup node updates the copy of the access control list in its local cache according to the version identifier.

10. The multi-source data intelligent management method based on deep learning according to claim 9, characterized in that, The access control manager authenticates target users according to the access control list, including: The access control manager receives an access token sent by the target user, the access token containing the target user's user identifier and the requested operation type; Search the access control list for the first permission entry that matches the user identifier; Determine whether the set of operation types in the first permission entry contains the requested operation type; If the set of operation types includes the request operation type, then it is further checked whether the current time is within the valid time period in the access control list, and whether the trust value of the target user is greater than the preset trust threshold. If the current time is within the valid time period and the trust value is greater than the preset trust threshold, then the decryption key of the data shard is output to the target user, and the target user's access behavior is written into the audit log.

Images