Erasure code compatible read-write method and system based on bidirectional data access proxy

Abstract

The invention provides an erasure code compatible read-write method and system based on a bidirectional data access agency, and relates to the technical field of distributed storage, and the method comprises the steps: employing multi-agent reinforcement learning to construct a heterogeneous resource portrait and optimize task allocation, combining a space-time graph convolutional network and a federated learning framework to construct a data format conversion model, and constructing an erasure code compatible read-write model; data features are analyzed through an encoder based on an attention mechanism, a fragmentation strategy is generated, self-adaptive encoding is achieved through a capsule neural network, data integrity verification and repairing are conducted through a double-layer space attention network and a graph attention network, the data access efficiency of a heterogeneous storage system is improved, data format compatibility is enhanced, and the data access efficiency is improved. And the data integrity is ensured.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to an erasure code compatible reading and writing method and system based on a bidirectional data access agent. Background Art

[0002] With the advent of the big data era, data access and management in heterogeneous storage systems have become increasingly complex. Modern storage systems need to handle multiple data formats and storage architectures at the same time and ensure data reliability and consistency. In a distributed storage environment, data read and write operations face multiple challenges such as format compatibility, performance optimization, and fault-tolerant recovery. Traditional data access methods usually adopt a single proxy mode or a fixed encoding scheme, which lacks the ability to intelligently perceive and dynamically schedule heterogeneous storage resources, lacks the ability to deeply understand and adaptively learn data features, and lacks the ability to dynamically perceive data features and storage environments. Therefore, a solution is urgently needed to solve the problems existing in the prior art. Summary of the invention

[0003] The embodiments of the present invention provide an erasure code compatible reading and writing method and system based on a bidirectional data access agent, which can at least solve some of the problems existing in the prior art.

[0004] A first aspect of an embodiment of the present invention provides an erasure code compatible reading and writing method based on a bidirectional data access agent, comprising: The computing power and storage status of each storage node in the system are collected, and the heterogeneous resource portrait is constructed by encoding with a multi-layer perceptron. Based on the heterogeneous resource portrait, the node state sequence is input into the multi-agent reinforcement learning model, and the optimal task allocation strategy is obtained through interactive iterative calculation of the policy network and the value network. According to the optimal task allocation strategy, the read agent module and the write agent module are deployed in the system, and the data format conversion model is constructed by using a space-time graph convolutional network. The model parameters are distributed to the storage nodes based on the federated learning framework, and the data format conversion model is optimized through local training and global aggregation iterations to generate a data feature mapping rule base. The data feature mapping rule base and the erasure code configuration parameters are used to establish a multi-level index structure through a hierarchical Bloom filter, and the query performance is optimized in combination with the skip table mechanism to obtain the system initial configuration table; After receiving a data write request, a multi-head attention analysis is performed on the data to be written through an encoder based on an attention mechanism, and the temporal features and spatial features of the data to be written are extracted to generate a high-dimensional data feature vector, and the high-dimensional data feature vector is added to the capsule neural network of dynamic routing, and the optimal sharding strategy and adaptive coding parameters are calculated through iterative negotiation between capsules. The data to be written is layered and segmented based on the optimal sharding strategy, and multi-level data shards and layered verification information with redundancy marks are generated in combination with the adaptive coding parameters. The data distribution plan is calculated through a consistent hashing algorithm and a load-aware node selection strategy, and the multi-level data shards and the layered verification information are sent to the storage node based on the data distribution plan and the system initial configuration table is updated; After receiving a data read request, the target data storage location and encoding parameters are retrieved in the updated system initial configuration table, multi-channel parallel reading is started based on the storage location, the multi-level data segmentation and layered verification information are obtained in combination with the pipeline mechanism, the read data is added to the data format conversion model, the data features are extracted in combination with the graph convolution layer, and the optimal conversion path is calculated according to the data feature mapping rule library, and a conversion execution plan is generated and the format conversion is performed to obtain an initial conversion result, and the integrity of the initial conversion result is verified through a two-layer spatial attention network to generate an integrity score. If the integrity score is lower than a preset integrity threshold, a data dependency graph is constructed through a graph attention network, and the defect location is located and repaired in combination with the layered verification information. The reconstruction is repeated until the preset maximum number of reconstructions is reached to obtain the target format data.

[0005] In an optional embodiment, The computing power and storage status of each storage node in the collection system are combined with a multi-layer perceptron for encoding to construct a heterogeneous resource portrait. Based on the heterogeneous resource portrait, the node state sequence is input into the multi-agent reinforcement learning model, and the optimal task allocation strategy is obtained through interactive iterative calculation of the policy network and the value network. According to the optimal task allocation strategy, the read agent module and the write agent module are deployed in the system, and the space-time graph convolution network is used to build a data format conversion model. The model parameters are distributed to the storage nodes based on the federated learning framework. The data format conversion model is optimized through local training and global aggregation iterations to generate a data feature mapping rule base. The data feature mapping rule base and the erasure code configuration parameters are used to establish a multi-level index structure through a hierarchical Bloom filter. The query performance is optimized in combination with the skip table mechanism. The system initial configuration table includes: Deploy distributed monitoring probes on each storage node, collect computing power and storage status of the storage node through the distributed monitoring probes to obtain raw monitoring data, pre-process the raw monitoring data, eliminate short-term fluctuations in processor and memory usage data through a sliding window averaging method, convert storage capacity data into a ratio of used space to total space, and normalize input and output throughput data based on historical peak values ​​to obtain pre-processed data; A multilayer perceptron with a residual connection structure is used to extract features from the preprocessed data. The first hidden layer of the multilayer perceptron extracts independent features of various indicators. The second hidden layer learns the combined features between indicators through nonlinear transformation. The output layer generates feature vectors to construct heterogeneous resource portraits. Organizing the feature vector into a node state sequence by means of a sliding window, inputting the node state sequence into a multi-agent reinforcement learning model, each agent maintaining a corresponding node feature sequence, task queue status and resource reservation status, processing the node state sequence by means of a hierarchical state encoder, wherein the hierarchical state encoder includes a bottom encoder for processing continuous features, a middle encoder for extracting a temporal pattern and a top encoder for integrating information, and outputting a state code; The strategy network of the multi-agent reinforcement learning model includes an action network and an evaluation network. The action network outputs an action probability distribution based on the state encoding, and the evaluation network evaluates the state value. A dynamic communication topology between agents is constructed based on the task dependency relationship and resource sharing degree between nodes. Neighbor information sequences are processed through gated recurrent units, and hierarchical reinforcement learning is used for decision decomposition to obtain an optimal task allocation strategy. Deploy the reading proxy module and the writing proxy module according to the optimal task allocation strategy, and use the space-time graph convolution network with a dual-stream architecture to build a data format conversion model, wherein the space-time graph convolution layer of the space-time graph convolution network aggregates neighbor features through a message passing mechanism to update the node representation, and the temporal convolution layer uses a hole convolution to expand the receptive field to capture data dependencies, and introduces a hierarchical attention mechanism of structural attention and temporal attention to process features, thereby obtaining a data format conversion model; Based on the federated learning framework, the model parameters of the data format conversion model are distributed, a hierarchical training strategy is adopted to locally train the basic feature extractor, and high-level features are globally optimized. The pre-trained model is used to guide the training process through progressive knowledge distillation to generate a data feature mapping rule base; Construct a sharded hierarchical Bloom filter, divide the bitmap of the hierarchical Bloom filter into multiple independently maintained shards, establish a multi-level index structure for the data feature mapping rule base and erasure code configuration parameters through the sharded hierarchical Bloom filter, determine the index level through a random layer number generation algorithm based on a dynamically balanced skip table mechanism and the multi-level index structure, maintain a data version chain in the index node to record the version evolution relationship, adopt a hierarchical filtering strategy to perform query optimization, and obtain the system initial configuration table.

[0006] In an optional embodiment, The strategy network of the multi-agent reinforcement learning model includes an action network and an evaluation network. The action network outputs the action probability distribution based on the state encoding, and the evaluation network evaluates the state value. The dynamic communication topology between agents is constructed based on the task dependency relationship and resource sharing degree between nodes. The neighbor information sequence is processed by the gated recurrent unit, and the decision decomposition is performed by hierarchical reinforcement learning. The optimal task allocation strategy includes: Constructing a policy network of a multi-agent reinforcement learning model, the policy network includes an action network and an evaluation network, receiving state encoding data, the state encoding data includes a computing resource dimension, a storage resource dimension, and a network resource dimension, wherein the computing resource dimension records processor usage information, memory occupancy information, and process quantity information, the storage resource dimension records disk read and write information, storage space usage information, and storage queue length information, and the network resource dimension records bandwidth usage information, network transmission delay information, and data packet loss information; Adding the state encoding data to the action network, performing data normalization transformation and reducing the dimension through the first layer to obtain standardized features, extracting the correlation patterns in the standardized features and constructing combined features through the second layer, and generating action probability distribution according to the combined features; Based on the evaluation network, the state encoding data is processed in parallel, the mutual influence relationship between different dimensional indicators is analyzed by the feature interaction processing unit to obtain the interaction feature, and the interaction feature is mapped through the neural network to obtain the state value; Statistically record the task flow information between nodes during the operation of the system, calculate the task dependency between nodes according to the task flow frequency, obtain the resource sharing degree according to the storage resource sharing state between nodes, and construct the dynamic communication topology between intelligent agents according to the task dependency and the resource sharing degree; Determine a neighbor node set of each intelligent agent based on the dynamic communication topology, process a neighbor information sequence in the neighbor node set through a gated recurrent unit, calculate an information update weight based on the change amplitude between the current state and the historical state using an information update control unit, calculate a historical information retention weight based on the time correlation of the state information using a historical information control unit, and update the node state information according to the information update weight and the historical information retention weight; Hierarchical reinforcement learning is used to perform decision decomposition. Task feature information is extracted in the first-level decision to obtain task computing load features and task data access features. The task computing load features include processor time consumption information, memory usage information, and storage access information. The task data access features include data storage location information, data transmission scale information, and data access regularity information. In the second-level decision, the task feature information, the node state information, the action probability distribution and the state value are used as input to determine the target execution node; Based on the dynamic communication topology and the target execution nodes, the policy network is trained in a distributed manner. Each target execution node is independently trained to obtain network parameters based on local task execution records. The network parameters are exchanged and integrated between target execution nodes with communication connections to generate an optimal task allocation strategy, wherein the optimal task allocation strategy includes a correspondence between task types and target nodes.

[0007] In an optional embodiment, After receiving a data write request, a multi-head attention analysis is performed on the data to be written through an encoder based on an attention mechanism, and the temporal features and spatial features of the data to be written are extracted to generate a high-dimensional data feature vector, and the high-dimensional data feature vector is added to the capsule neural network of dynamic routing, and the optimal sharding strategy and adaptive coding parameters are calculated through iterative negotiation between capsules, and the data to be written is hierarchically segmented based on the optimal sharding strategy, and multi-level data shards and hierarchical verification information with redundancy marks are generated in combination with the adaptive coding parameters, and a data distribution scheme is calculated through a consistent hashing algorithm and a load-aware node selection strategy, and the multi-level data shards and the hierarchical verification information are sent to the storage node based on the data distribution scheme and the system initial configuration table is updated, including: Receive a data write request, and perform multi-head attention analysis on the data to be written through an encoder based on the attention mechanism, wherein the encoder includes multiple parallel attention processing heads, wherein the first group of attention processing heads is responsible for analyzing temporal features, and the second group of attention processing heads is responsible for analyzing spatial features. Each data block generates a query vector, a key vector, and a value vector through a feature extraction layer, and the combination obtains a data block sequence; The data block sequence is processed by a sliding window mechanism, the temporal features of the data to be written are extracted, the similarity between the query vector of the current data block and the key vectors of other data blocks in the window is calculated within the sliding window, and the temporal attention weight is obtained through normalization processing, the data block sequence is processed by a grid partitioning method, the spatial features of the data to be written are extracted, the adjacent data blocks are organized into a two-dimensional grid structure, and the spatial correlation between the target data block and the surrounding data blocks is calculated to obtain the spatial attention weight; Recursively perform feature fusion on the time series features, perform weighted combination of features of adjacent time windows to obtain a time series feature vector, perform hierarchical feature fusion on the spatial features, perform feature aggregation in a local grid and then fuse layer by layer to obtain a spatial feature vector, concatenate the time series feature vector and the spatial feature vector and generate the high-dimensional data feature vector through full connection layer mapping; Adding the high-dimensional data feature vector to a capsule neural network of dynamic routing, the capsule neural network comprises a primary capsule layer, an intermediate capsule layer and an advanced capsule layer, detecting local feature patterns through the primary capsule layer, reorganizing and abstracting the features of the primary capsule layer through the intermediate capsule layer, and representing data sharding strategies of different granularities through the advanced capsule layer; The primary capsule layer, the intermediate capsule layer and the high-level capsule layer are connected through iterative negotiation between capsules, the feature similarity between the low-level capsules and the high-level capsules is calculated and the connection weight is updated in each round of the iterative negotiation, and the optimal fragmentation strategy and adaptive coding parameters are extracted from the high-level capsule layer; Based on the optimal sharding strategy, the data to be written is divided into layers, a multi-level data shard including a basic block, a time-series shard, a space shard and an aggregate shard is constructed, the redundancy mark of the multi-level data shard is calculated according to the adaptive coding parameter, the multi-level data shard with the redundancy mark is generated, and the hierarchical check information is generated by calculating the parity check code of the basic block, the erasure code corresponding to different shards and the checksum of the aggregate shard; A consistent hash ring of virtual nodes is constructed through a consistent hash algorithm and multiple virtual nodes are evenly distributed on the consistent hash ring for each storage node. A load-aware node selection strategy is adopted to collect the load status of the storage node and calculate the load score of the storage node. A data distribution plan is calculated according to the load score. Based on the data distribution plan, the multi-level data shards and the hierarchical verification information are sent to the storage node and the system initial configuration table is updated.

[0008] In an optional embodiment, A consistent hash ring of virtual nodes is constructed by a consistent hash algorithm, and a plurality of virtual nodes are evenly distributed on the consistent hash ring for each storage node. A load-aware node selection strategy is adopted to collect the load status of the storage node and calculate the load score of the storage node. The data distribution scheme is calculated according to the load score, including: A consistent hash ring is constructed by a consistent hash algorithm, a first hash operation is performed on the storage node identifier to obtain a reference hash value and determine the reference position of the virtual node, a second hash operation is performed on the reference hash value to obtain a modified hash value and determine the final position of the virtual node on the consistent hash ring, and a plurality of virtual nodes including node attribute information, load monitoring parameters and routing tags are generated for each storage node; The node distribution discreteness of the local area on the consistent hash ring is calculated by a sliding window method, and the area where the node distribution discreteness exceeds a preset threshold is marked as an area to be optimized, and the distribution position of the virtual node is adjusted in the area to be optimized, and the distribution of the virtual node is optimized by a decreasing adjustment step until the virtual nodes are evenly distributed on the consistent hash ring; Adopting a load-aware node selection strategy, obtaining the system resource usage status and business load data of the storage node through a load acquisition agent, hierarchically storing the system resource usage status and the business load data according to the time span, performing data smoothing and abnormal filtering on the stored load data, and analyzing the processed load data to obtain the load status of the storage node; Calculating a system resource load score and a business load score of the storage node based on the load status of the storage node, determining a weight coefficient of the system resource load score and the business load score in combination with the system operation status, and combining the weighted system resource load score and the business load score to obtain a load score of the storage node; Calculate the hash value of the data shard and determine the target position on the consistent hash ring, select multiple virtual nodes within a fixed distance from the target position as candidate nodes, sort the storage nodes corresponding to the candidate nodes based on the load scores and select the storage node with the lowest score as the target storage node, check the load status of the target storage node, and if the load status exceeds a preset load limit, select the storage node with the second lowest score as the target storage node; Calculate the difference in load scores between the storage nodes, screen the storage node pairs whose load score difference exceeds a preset threshold, mark the storage nodes with high load scores in the screened storage node pairs as source nodes, and mark the storage nodes with low load scores as target nodes, calculate the number of data shards that need to be migrated in the source nodes, and generate a data distribution plan that includes the source nodes, the target nodes, and the number of data shards.

[0009] In an optional embodiment, After receiving the data read request, the target data storage location and encoding parameters are retrieved in the updated system initial configuration table, multi-channel parallel reading is started based on the storage location, the multi-level data sharding and layered verification information are obtained in combination with the pipeline mechanism, the read data is added to the data format conversion model, the data features are extracted in combination with the graph convolution layer, and the optimal conversion path is calculated according to the data feature mapping rule library, and a conversion execution plan is generated and the format conversion is performed to obtain the initial conversion result, and the integrity of the initial conversion result is verified by a two-layer spatial attention network to generate an integrity score. If the integrity score is lower than a preset integrity threshold, a data dependency graph is constructed through a graph attention network, and the defect position is located and repaired in combination with the layered verification information, and reconstruction is repeated until the preset maximum number of reconstructions is reached, and the target format data is obtained, including: Receive a data read request, retrieve the storage location and encoding parameters of the target data in the updated system initial configuration table, start multi-channel parallel reading based on the storage location, and obtain multi-level data sharding and layered verification information in combination with a pipeline mechanism. The pipeline mechanism divides data processing into four processing stages: reading, decoding, verification, and assembly. Each processing stage maintains an independent task queue, and data is transferred between the processing stages through a producer-consumer model; Add the read data to the data format conversion model, build a data structure graph, set the data fields as graph nodes, set the associations between the fields as graph edges, assign weight values ​​to the graph edges according to the business relevance between the fields, extract data features in combination with the graph convolution layer, extract local structural features in the first layer of graph convolution processing, aggregate information of directly connected nodes for each graph node, fuse adjacent node information in the second layer of graph convolution processing, combine the first layer node features with the features of second-order adjacent nodes, obtain global structural features in the third layer of graph convolution processing, fuse the information of all nodes in the data structure graph, and generate a data feature vector; Retrieving conversion rules in a data feature mapping rule library according to the data feature vector, calculating feature similarity between the feature vector and a preset conversion template, and selecting the conversion template with the highest similarity to calculate an optimal conversion path; Generate a conversion execution plan based on the optimal conversion path and perform format conversion, adjust the hierarchical structure of the source data format to the hierarchical structure of the target format in the structural reorganization phase, establish a corresponding relationship between the source format field and the target format field in the field mapping phase, and convert the field value into the data type required by the target format in the data conversion phase to obtain an initial conversion result; Verify the integrity of the initial conversion result through a two-layer spatial attention network, check the existence, value range and type matching of the data field at the local attention layer, calculate the field-level integrity score, verify the primary and foreign key relationships, uniqueness constraints and business rules between data records at the global attention layer, calculate the record-level integrity score, and combine the field-level integrity score and the record-level integrity score to generate an integrity score; If the integrity score is lower than a preset integrity threshold, a data dependency graph is constructed through a graph attention network, data fields are set as graph nodes, dependencies between fields are set as graph edges, weight values ​​are assigned to the graph edges based on the degree of business impact, the defect location is located in combination with the hierarchical verification information, the repair priority is determined based on the weight of the graph edge, the abnormal fields are sorted in descending order according to the impact range, and the abnormal fields are repaired according to the sorting results, and data reconstruction is performed in an iterative manner; The integrity score is recalculated after each round of reconstruction, and the reconstruction is repeated until the integrity score reaches the integrity threshold or reaches a preset maximum number of reconstruction times, thereby obtaining target format data.

[0010] In an optional embodiment, A data dependency graph is constructed through a graph attention network, data fields are set as graph nodes, dependencies between fields are set as graph edges, weight values ​​are assigned to the graph edges based on the degree of business impact, the defect location is located in combination with the hierarchical verification information, the repair priority is determined based on the weight of the graph edge, the abnormal fields are sorted in descending order according to the impact range, and the abnormal fields are repaired according to the sorting results. Data reconstruction in combination with an iterative method includes: Construct a data dependency graph through a graph attention network, set data fields as graph nodes, parse database constraint information to extract dependencies between fields to construct reference dependency edges, parse triggers and stored procedures to construct calculation dependency edges, parse field update rules to construct timing dependency edges, analyze data content to calculate the mutual information value of field value distribution to construct association dependency edges, analyze the synchronous change pattern of field values ​​to construct collaborative dependency edges, detect the hierarchical inclusion relationship of field values ​​to construct nested dependency edges, and set the reference dependency edges, the calculation dependency edges, the timing dependency edges, the association dependency edges, the collaborative dependency edges, and the nested dependency edges as graph edges of the data dependency graph; Based on the business impact, weight values ​​are assigned to the graph edges of the data dependency graph. At the dependency type layer, the base weight of the reference dependency edge is set to a first weight value, the base weight of the calculation dependency edge is set to a second weight value, and the base weight of the timing dependency edge is set to a third weight value. At the data feature layer, the weight values ​​of the graph edges are adjusted based on the reference data volume ratio, calculation complexity, and time correlation strength. At the business importance layer, the weight values ​​of the graph edges of the dependency relationships involved in the core business process are increased. A three-level verification rule tree is constructed based on hierarchical verification information. The first-level rules check the non-emptiness, type matching, and length constraints of the field, and mark the fields that violate the basic constraints as the first-level abnormal nodes. The second-level rules check the value range, enumeration value set, and field combination constraints of the field, and mark the fields that violate the business rules as the second-level abnormal nodes. The third-level rules check the consistency, timeliness, and accuracy standards of the data, and mark the fields that violate the quality requirements as the third-level abnormal nodes. In the data dependency graph, starting from each abnormal node, traverse all reachable nodes downstream along the graph edge, calculate the impact attenuation coefficient between nodes based on the cumulative graph edge weight of the dependency path, sum the impact attenuation coefficients of the abnormal node on all reachable nodes to obtain the impact range score, and sort the abnormal fields in descending order according to the impact range score to obtain a repair priority sequence; The abnormal fields are repaired in sequence according to the repair priority sequence, the values ​​of the numerical abnormal fields are randomly generated based on the historical data of the time series, the categories of the categorical abnormal fields are inferred based on the association rules between the fields, and the complete information of the text abnormal fields is supplemented based on the string similarity matching, and the repaired field values ​​are updated to the original data; The data is reconstructed in an iterative manner, node features and impact range scores are recalculated on the data dependency graph, the abnormal node set is updated and the data quality score is calculated, and the quality scores of two adjacent rounds are compared. When the score improvement is greater than a preset score improvement threshold or reaches a preset maximum number of iterations, the repair result is output.

[0011] A second aspect of an embodiment of the present invention provides an erasure code compatible read / write system based on a bidirectional data access agent, comprising: The first unit is used to collect the computing power and storage status of each storage node in the system, and encode and construct a heterogeneous resource portrait in combination with a multi-layer perceptron. Based on the heterogeneous resource portrait, the node state sequence is input into the multi-agent reinforcement learning model, and the optimal task allocation strategy is obtained through interactive iterative calculation of the policy network and the value network. According to the optimal task allocation strategy, a read agent module and a write agent module are deployed in the system, and a data format conversion model is constructed using a space-time graph convolutional network. The model parameters are distributed to the storage nodes based on a federated learning framework, and the data format conversion model is optimized through local training and global aggregation iterations to generate a data feature mapping rule base. The data feature mapping rule base and the erasure code configuration parameters are used to establish a multi-level index structure through a hierarchical Bloom filter, and the query performance is optimized in combination with a skip table mechanism to obtain a system initial configuration table; The second unit is used for, after receiving a data write request, performing multi-head attention analysis on the data to be written through an encoder based on the attention mechanism, extracting the temporal features and spatial features of the data to be written to generate a high-dimensional data feature vector, adding the high-dimensional data feature vector to the capsule neural network of dynamic routing, calculating the optimal sharding strategy and adaptive coding parameters through iterative negotiation between capsules, performing hierarchical segmentation on the data to be written based on the optimal sharding strategy, generating multi-level data sharding and hierarchical verification information with redundancy marks in combination with the adaptive coding parameters, calculating the data distribution plan through a consistent hashing algorithm and a load-aware node selection strategy, sending the multi-level data sharding and the hierarchical verification information to the storage node based on the data distribution plan, and updating the system initial configuration table; The third unit is used to retrieve the target data storage location and encoding parameters in the updated system initial configuration table after receiving a data read request, start multi-channel parallel reading based on the storage location, obtain the multi-level data segmentation and layered verification information in combination with the pipeline mechanism, add the read data to the data format conversion model, extract data features in combination with the graph convolution layer and calculate the optimal conversion path according to the data feature mapping rule library, generate a conversion execution plan and perform format conversion to obtain an initial conversion result, verify the integrity of the initial conversion result through a two-layer spatial attention network, generate an integrity score, and if the integrity score is lower than a preset integrity threshold, construct a data dependency graph through the graph attention network, locate the defect location in combination with the layered verification information and repair it, repeat the reconstruction until the preset maximum number of reconstructions is reached, and obtain the target format data.

[0012] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0013] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0014] In the present invention, a heterogeneous resource portrait and data format conversion model is constructed through multi-agent reinforcement learning and federated learning framework, which realizes the dynamic perception of storage node computing power and storage status, improves the system resource utilization efficiency and data processing adaptability, and makes the system more scalable and compatible. An encoder and capsule neural network based on the attention mechanism are used for data analysis and sharding strategy optimization. Consistent hashing and load perception are combined to achieve balanced distribution of data, which significantly improves the reliability and efficiency of the data writing process and reduces the risk of load imbalance in the storage system. Data feature extraction and integrity verification are performed based on graph convolutional networks and double-layer spatial attention networks. Defect repair is performed by constructing data dependency relationships through graph attention networks, which effectively ensures the accuracy of data reading and format conversion, and improves the system's fault tolerance and data availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of a flow chart of an erasure code compatible reading and writing method based on a bidirectional data access agent according to an embodiment of the present invention; Figure 2 It is a schematic diagram of the structure of an erasure code compatible read-write system based on a bidirectional data access agent according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0018] Figure 1 FIG. 1 is a flow chart of an erasure code compatible reading and writing method based on a bidirectional data access agent according to an embodiment of the present invention. Figure 1 As shown, the method includes: S1. Collect the computing power and storage status of each storage node in the system, combine the multi-layer perceptron to encode and construct a heterogeneous resource portrait, based on the heterogeneous resource portrait, input the node state sequence into the multi-agent reinforcement learning model, and obtain the optimal task allocation strategy through interactive iterative calculation of the policy network and the value network. According to the optimal task allocation strategy, deploy the read agent module and the write agent module in the system, use the space-time graph convolution network to build a data format conversion model, distribute the model parameters to the storage nodes based on the federated learning framework, optimize the data format conversion model through local training and global aggregation iteration, generate a data feature mapping rule base, establish a multi-level index structure with the data feature mapping rule base and the erasure code configuration parameters through a hierarchical Bloom filter, and optimize the query performance in combination with the skip table mechanism to obtain the system initial configuration table; The heterogeneous resource portrait is a technology that models and represents the characteristics and relationships of different types of resources (such as computing, storage, and network), and is used to optimize resource allocation and scheduling. The space-time graph convolutional network is a neural network model that combines spatial and temporal information, and is used to process graph structured data with spatiotemporal dependencies. The data feature mapping rule base is a knowledge base containing data feature and target model mapping rules, which is used to guide feature selection and feature engineering. The hierarchical Bloom filter is a hierarchical and efficient data storage and query method, which is used to quickly determine whether an element exists in a set. The skip list mechanism is a data structure that improves data retrieval speed through a multi-level linked list, which is used to implement fast search and sorting operations.

[0019] In an optional embodiment, The computing power and storage status of each storage node in the collection system are combined with a multi-layer perceptron for encoding to construct a heterogeneous resource portrait. Based on the heterogeneous resource portrait, the node state sequence is input into the multi-agent reinforcement learning model, and the optimal task allocation strategy is obtained through interactive iterative calculation of the policy network and the value network. According to the optimal task allocation strategy, the read agent module and the write agent module are deployed in the system, and the space-time graph convolution network is used to build a data format conversion model. The model parameters are distributed to the storage nodes based on the federated learning framework. The data format conversion model is optimized through local training and global aggregation iterations to generate a data feature mapping rule base. The data feature mapping rule base and the erasure code configuration parameters are used to establish a multi-level index structure through a hierarchical Bloom filter. The query performance is optimized in combination with the skip table mechanism. The system initial configuration table includes: Deploy distributed monitoring probes on each storage node, collect computing power and storage status of the storage node through the distributed monitoring probes to obtain raw monitoring data, pre-process the raw monitoring data, eliminate short-term fluctuations in processor and memory usage data through a sliding window averaging method, convert storage capacity data into a ratio of used space to total space, and normalize input and output throughput data based on historical peak values ​​to obtain pre-processed data; A multilayer perceptron with a residual connection structure is used to extract features from the preprocessed data. The first hidden layer of the multilayer perceptron extracts independent features of various indicators. The second hidden layer learns the combined features between indicators through nonlinear transformation. The output layer generates feature vectors to construct heterogeneous resource portraits. Organizing the feature vector into a node state sequence by means of a sliding window, inputting the node state sequence into a multi-agent reinforcement learning model, each agent maintaining a corresponding node feature sequence, task queue status and resource reservation status, processing the node state sequence by means of a hierarchical state encoder, wherein the hierarchical state encoder includes a bottom encoder for processing continuous features, a middle encoder for extracting a temporal pattern and a top encoder for integrating information, and outputting a state code; The strategy network of the multi-agent reinforcement learning model includes an action network and an evaluation network. The action network outputs an action probability distribution based on the state encoding, and the evaluation network evaluates the state value. A dynamic communication topology between agents is constructed based on the task dependency relationship and resource sharing degree between nodes. Neighbor information sequences are processed through gated recurrent units, and hierarchical reinforcement learning is used for decision decomposition to obtain an optimal task allocation strategy. Deploy the reading proxy module and the writing proxy module according to the optimal task allocation strategy, and use the space-time graph convolution network with a dual-stream architecture to build a data format conversion model, wherein the space-time graph convolution layer of the space-time graph convolution network aggregates neighbor features through a message passing mechanism to update the node representation, and the temporal convolution layer uses a hole convolution to expand the receptive field to capture data dependencies, and introduces a hierarchical attention mechanism of structural attention and temporal attention to process features, thereby obtaining a data format conversion model; Based on the federated learning framework, the model parameters of the data format conversion model are distributed, a hierarchical training strategy is adopted to locally train the basic feature extractor, and high-level features are globally optimized. The pre-trained model is used to guide the training process through progressive knowledge distillation to generate a data feature mapping rule base; Construct a sharded hierarchical Bloom filter, divide the bitmap of the hierarchical Bloom filter into multiple independently maintained shards, establish a multi-level index structure for the data feature mapping rule base and erasure code configuration parameters through the sharded hierarchical Bloom filter, determine the index level through a random layer number generation algorithm based on a dynamically balanced skip table mechanism and the multi-level index structure, maintain a data version chain in the index node to record the version evolution relationship, adopt a hierarchical filtering strategy to perform query optimization, and obtain the system initial configuration table.

[0020] The sliding window averaging method is a method for calculating the average value using a fixed-size time window, which is used to smooth time series data or reduce noise. The hierarchical training strategy is a strategy for dividing model training into multiple levels and gradually optimizing them, which is used to improve training efficiency and model performance. The progressive knowledge distillation is a technology for gradually transferring the knowledge of complex models to simple models, which is used to accelerate model deployment and reasoning. The index hierarchy refers to the distribution and organization of indexes at different levels in a multi-level index structure, which is used to improve data retrieval efficiency. The hierarchical filtering strategy is a method for filtering data in stages according to priority and conditions, which is used to reduce the overhead of invalid data processing.

[0021] In heterogeneous storage systems, distributed monitoring probes are deployed to collect status information of each node. Monitoring probes collect data every 10 seconds, including CPU usage, memory occupancy, storage space usage, network throughput and other indicators. For CPU and memory usage data, a 60-second sliding window is used to average and eliminate instantaneous fluctuations. Storage space data is converted into usage rate. For example, if the total capacity of a node is 1TB and 800GB has been used, the usage rate is 0.8. Network throughput is normalized based on the historical peak value of 10Gbps.

[0022] A four-layer perceptron network is used to extract features. The first hidden layer contains 256 neurons, which process various original indicators respectively; the second hidden layer contains 128 neurons, which learn the association between indicators; the third hidden layer contains 64 neurons, which perform feature dimension reduction; the output layer 32-dimensional feature vector represents the node resource portrait. Residual connections are added between each layer to alleviate the gradient disappearance problem.

[0023] The feature vector sequence of the last 30 minutes is sampled with a sliding step of 5 minutes and input into the multi-agent reinforcement learning model. Each agent corresponds to a storage node, which maintains the feature sequence of the last 6 time slices of the node, the length of the current task queue, resource reservation status and other state information. The hierarchical state encoder first uses a one-dimensional convolutional network to process continuous features, then uses a long short-term memory network to extract the temporal pattern, and finally uses a fully connected layer to integrate various types of information.

[0024] The policy network consists of two parts: the action generation network and the state evaluation network. The action network outputs the probability of task assignment, and the evaluation network predicts the state value. The agent communication topology is constructed based on the amount of data transmission between nodes and the degree of resource competition, and the information sequence from neighboring nodes is processed through the gated recurrent unit. A two-layer decision-making structure is adopted: the high-level determines the task type assignment, and the low-level refines the specific execution plan.

[0025] Deploy the read-write proxy module to execute the task allocation strategy. The data format conversion model adopts a dual-stream graph convolutional network architecture. The spatial branch uses three-layer graph convolution for feature aggregation, and the temporal branch uses dilated convolution to expand the receptive field to 128 time steps. Combined with the structural attention mechanism, it focuses on important node connections, and the temporal attention mechanism captures long-term dependencies.

[0026] The conversion model is trained using a federated learning framework. Each node retains the original data and only exchanges model parameters. The underlying feature extractor is trained locally, and the high-level feature representation is optimized through model aggregation. The pre-trained model is used to guide the training process, and the model performance is improved through three-stage progressive knowledge distillation. Finally, a feature mapping rule base is generated.

[0027] A four-layer Bloom filter index structure is constructed, with each layer divided into 8 independently maintained bitmap slices. A dynamically balanced skip table mechanism is used to optimize queries. The number of node layers is determined by a random algorithm and maintained at an average of 4 layers. The index node records the data version chain and supports querying historical versions by timestamp. A top-down hierarchical filtering strategy is used during query to narrow the search range layer by layer to obtain the system initial configuration table.

[0028] In this embodiment, accurate profiling of heterogeneous resources is achieved through distributed monitoring and multi-layer perceptron feature extraction, which can effectively improve resource utilization efficiency and system throughput. The task scheduling strategy based on multi-agent reinforcement learning can adaptively balance the load and optimize data locality, which helps to reduce task response time and system energy consumption. Federated learning and multi-level indexing are used to optimize data access, protect data privacy and improve query performance, which helps to improve storage space utilization while reducing query latency.

[0029] In an optional embodiment, The strategy network of the multi-agent reinforcement learning model includes an action network and an evaluation network. The action network outputs the action probability distribution based on the state encoding, and the evaluation network evaluates the state value. The dynamic communication topology between agents is constructed based on the task dependency relationship and resource sharing degree between nodes. The neighbor information sequence is processed by the gated recurrent unit, and the decision decomposition is performed by hierarchical reinforcement learning. The optimal task allocation strategy includes: Constructing a policy network of a multi-agent reinforcement learning model, the policy network includes an action network and an evaluation network, receiving state encoding data, the state encoding data includes a computing resource dimension, a storage resource dimension, and a network resource dimension, wherein the computing resource dimension records processor usage information, memory occupancy information, and process quantity information, the storage resource dimension records disk read and write information, storage space usage information, and storage queue length information, and the network resource dimension records bandwidth usage information, network transmission delay information, and data packet loss information; Adding the state encoding data to the action network, performing data normalization transformation and reducing the dimension through the first layer to obtain standardized features, extracting the correlation patterns in the standardized features and constructing combined features through the second layer, and generating action probability distribution according to the combined features; Based on the evaluation network, the state encoding data is processed in parallel, the mutual influence relationship between different dimensional indicators is analyzed by the feature interaction processing unit to obtain the interaction feature, and the interaction feature is mapped through the neural network to obtain the state value; Statistically record the task flow information between nodes during the operation of the system, calculate the task dependency between nodes according to the task flow frequency, obtain the resource sharing degree according to the storage resource sharing state between nodes, and construct the dynamic communication topology between intelligent agents according to the task dependency and the resource sharing degree; Determine a neighbor node set of each intelligent agent based on the dynamic communication topology, process a neighbor information sequence in the neighbor node set through a gated recurrent unit, calculate an information update weight based on the change amplitude between the current state and the historical state using an information update control unit, calculate a historical information retention weight based on the time correlation of the state information using a historical information control unit, and update the node state information according to the information update weight and the historical information retention weight; Hierarchical reinforcement learning is used to perform decision decomposition. Task feature information is extracted in the first-level decision to obtain task computing load features and task data access features. The task computing load features include processor time consumption information, memory usage information, and storage access information. The task data access features include data storage location information, data transmission scale information, and data access regularity information. In the second-level decision, the task feature information, the node state information, the action probability distribution and the state value are used as input to determine the target execution node; Based on the dynamic communication topology and the target execution nodes, the policy network is trained in a distributed manner. Each target execution node is independently trained to obtain network parameters based on local task execution records. The network parameters are exchanged and integrated between target execution nodes with communication connections to generate an optimal task allocation strategy, wherein the optimal task allocation strategy includes a correspondence between task types and target nodes.

[0030] The task flow frequency refers to the frequency at which tasks are scheduled and executed in the system, which is used to measure the efficiency of task flow. The resource sharing degree is an indicator that describes the level of sharing and utilization of the same resource by multiple tasks or users, which is used to evaluate the resource allocation effect. The dynamic communication topology refers to the communication network structure that changes dynamically over time, which is used to adapt to real-time needs in distributed systems.

[0031] Obtain status code data, which includes three main aspects: computing resource dimension, storage resource dimension, and network resource dimension. The computing resource dimension records information such as processor utilization, memory usage percentage, and the number of currently running processes, for example, the processor utilization is 75%, the memory usage is 60%, and the number of running processes is 12. The storage resource dimension includes the disk read and write speed per second, storage space usage, and the number of waiting tasks in the storage queue, for example, the disk read and write speed is 120MB / s, the storage space utilization is 45%, and the storage queue length is 8. The network resource dimension records indicators such as bandwidth utilization, network transmission delay time, and packet loss rate, for example, bandwidth utilization is 65%, transmission delay is 25ms, and packet loss rate is 0.1%.

[0032] After the action network receives the state encoding data, it performs data normalization through the first layer. The maximum and minimum normalization method is used to convert the indicators of different dimensions into a unified interval, and the original features are reduced in dimension to obtain standardized features through the principal component analysis method. The second layer uses a deep neural network to extract the correlation pattern in the standardized features, and the combined features are constructed through a multi-layer perceptron. The last layer uses the Softmax function to map the combined features to the action probability distribution.

[0033] The evaluation network uses a parallel architecture to process state encoding data. The feature interaction processing unit uses the attention mechanism to analyze the mutual influence between indicators of different dimensions, such as the correlation between computing resource utilization and storage access latency. The interactive features are mapped through a multi-layer neural network to obtain the state value evaluation result.

[0034] During the operation of the system, the task flow information between nodes is statistically recorded, including task submission time, execution node, completion time, etc. The strength of task dependency between nodes is calculated based on the task flow frequency within a certain time window. At the same time, the storage resource sharing status between nodes is monitored, including information such as shared storage capacity and data transmission frequency, and the degree of resource sharing is obtained based on this. The dynamic communication topology between agents is constructed based on task dependency and resource sharing degree.

[0035] For each agent, its neighbor node set is determined according to the communication topology. The gated recurrent unit is used to process the state information sequence from the neighbor nodes. The information update control unit calculates the change range between the current state and the historical state to obtain the information update weight. The historical information control unit calculates the historical information retention weight based on the time correlation analysis of the state information. The node state information is updated according to the weight.

[0036] In the first-level decision of hierarchical reinforcement learning, task feature information is extracted. Task computation load features include information such as estimated processor time consumption, estimated memory usage, storage access frequency, etc. Task data access features include information such as data file storage location, data transmission size, and data access time interval.

[0037] The second-level decision takes task feature information, node status information, action probability distribution, and state value as input, and determines the target node that is most suitable for executing the task through the policy network. Based on the determined target execution node, the policy network is trained in a distributed manner. Each target execution node independently trains network parameters based on local task execution records. Nodes with communication connections regularly exchange parameters and integrate them to ultimately generate the optimal task allocation strategy.

[0038] In this embodiment, by constructing a policy network including an action network and an evaluation network, accurate perception of the system state and effective evaluation of action decisions are achieved, the accuracy and efficiency of task allocation are improved, a dynamic communication topology is constructed based on the task dependency relationship and resource sharing degree between nodes, and a gated recurrent unit is used to process the neighbor information sequence, which enhances the coordination ability between nodes and improves the overall resource utilization of the system. Hierarchical reinforcement learning is used for decision decomposition, and the complex task allocation problem is decomposed into two sub-problems of feature extraction and target node selection, which reduces the complexity of the decision space and improves the training efficiency and convergence speed of the model.

[0039] S2. After receiving a data write request, perform multi-head attention analysis on the data to be written through an encoder based on the attention mechanism, extract the temporal features and spatial features of the data to be written to generate a high-dimensional data feature vector, add the high-dimensional data feature vector to the capsule neural network of dynamic routing, calculate the optimal sharding strategy and adaptive coding parameters through iterative negotiation between capsules, divide the data to be written into layers based on the optimal sharding strategy, generate multi-level data shards and layered verification information with redundancy marks in combination with the adaptive coding parameters, calculate the data distribution plan through the consistent hashing algorithm and the load-aware node selection strategy, send the multi-level data shards and the layered verification information to the storage node based on the data distribution plan, and update the system initial configuration table; The capsule neural network with dynamic routing is a neural network model that combines capsule networks and dynamic routing mechanisms. It can achieve more accurate feature transfer and spatial relationship modeling by dynamically adjusting connection weights. It is widely used to process input data with complex structures. The iterative negotiation is a process in which multiple entities reach a consensus or optimize a solution by repeatedly exchanging information and adjusting strategies. It is often used for decision-making and collaboration in multi-agent systems. The optimal sharding strategy refers to a strategy for efficiently dividing data into multiple fragments in data distributed storage or computing, aiming to optimize resource utilization and reduce access delays. The adaptive encoding parameters refer to automatically adjusting parameter settings in the encoding process according to input data or system status to improve compression efficiency or data transmission quality. The redundancy marking is a technology for marking or managing data redundancy, aiming to improve data fault tolerance and recovery capabilities, and is usually used in storage and transmission systems. The consistent hashing algorithm is an algorithm that ensures the balance of data distribution on nodes by hashing data and nodes, and minimizes data redistribution when nodes are increased or decreased. The load-aware node selection strategy is a strategy for dynamically selecting task processing nodes according to node load conditions, which is used to optimize system performance and improve task execution efficiency.

[0040] In an optional embodiment, After receiving a data write request, a multi-head attention analysis is performed on the data to be written through an encoder based on an attention mechanism, and the temporal features and spatial features of the data to be written are extracted to generate a high-dimensional data feature vector, and the high-dimensional data feature vector is added to the capsule neural network of dynamic routing, and the optimal sharding strategy and adaptive coding parameters are calculated through iterative negotiation between capsules, and the data to be written is hierarchically segmented based on the optimal sharding strategy, and multi-level data shards and hierarchical verification information with redundancy marks are generated in combination with the adaptive coding parameters, and a data distribution scheme is calculated through a consistent hashing algorithm and a load-aware node selection strategy, and the multi-level data shards and the hierarchical verification information are sent to the storage node based on the data distribution scheme and the system initial configuration table is updated, including: Receive a data write request, and perform multi-head attention analysis on the data to be written through an encoder based on the attention mechanism, wherein the encoder includes multiple parallel attention processing heads, wherein the first group of attention processing heads is responsible for analyzing temporal features, and the second group of attention processing heads is responsible for analyzing spatial features. Each data block generates a query vector, a key vector, and a value vector through a feature extraction layer, and the combination obtains a data block sequence; The data block sequence is processed by a sliding window mechanism, the temporal features of the data to be written are extracted, the similarity between the query vector of the current data block and the key vectors of other data blocks in the window is calculated within the sliding window, and the temporal attention weight is obtained through normalization processing, the data block sequence is processed by a grid partitioning method, the spatial features of the data to be written are extracted, the adjacent data blocks are organized into a two-dimensional grid structure, and the spatial correlation between the target data block and the surrounding data blocks is calculated to obtain the spatial attention weight; Recursively perform feature fusion on the time series features, perform weighted combination of features of adjacent time windows to obtain a time series feature vector, perform hierarchical feature fusion on the spatial features, perform feature aggregation in a local grid and then fuse layer by layer to obtain a spatial feature vector, concatenate the time series feature vector and the spatial feature vector and generate the high-dimensional data feature vector through full connection layer mapping; Adding the high-dimensional data feature vector to a capsule neural network of dynamic routing, the capsule neural network comprises a primary capsule layer, an intermediate capsule layer and an advanced capsule layer, detecting local feature patterns through the primary capsule layer, reorganizing and abstracting the features of the primary capsule layer through the intermediate capsule layer, and representing data sharding strategies of different granularities through the advanced capsule layer; The primary capsule layer, the intermediate capsule layer and the high-level capsule layer are connected through iterative negotiation between capsules, the feature similarity between the low-level capsules and the high-level capsules is calculated and the connection weight is updated in each round of the iterative negotiation, and the optimal fragmentation strategy and adaptive coding parameters are extracted from the high-level capsule layer; Based on the optimal sharding strategy, the data to be written is divided into layers, a multi-level data shard including a basic block, a time-series shard, a space shard and an aggregate shard is constructed, the redundancy mark of the multi-level data shard is calculated according to the adaptive coding parameter, the multi-level data shard with the redundancy mark is generated, and the hierarchical check information is generated by calculating the parity check code of the basic block, the erasure code corresponding to different shards and the checksum of the aggregate shard; A consistent hash ring of virtual nodes is constructed through a consistent hash algorithm and multiple virtual nodes are evenly distributed on the consistent hash ring for each storage node. A load-aware node selection strategy is adopted to collect the load status of the storage node and calculate the load score of the storage node. A data distribution plan is calculated according to the load score. Based on the data distribution plan, the multi-level data shards and the hierarchical verification information are sent to the storage node and the system initial configuration table is updated.

[0041] The grid partitioning is a technology that divides the data space or computing area into multiple regular grids, which is usually used to balance the load and improve data access efficiency. The local characteristic pattern refers to a representative pattern or feature extracted from data or signals in a local range, which is used for efficient data analysis and processing. The parity code is a coding method for detecting data transmission errors, and determines whether data errors occur by adding parity bits. The erasure code is a coding method that can tolerate a certain amount of data loss or errors, and recovers lost information through redundant data. It is widely used in storage and network systems. The consistent hash ring is a consistent hash structure that manages data and nodes by mapping them to a ring space, which can effectively handle the dynamic increase and decrease of nodes.

[0042] After receiving a data write request, the data to be written needs to be preprocessed and feature analyzed. The original data is divided into basic data blocks according to a preset size (such as 4KB), and features are extracted for each data block to obtain its basic feature representation. During the feature extraction process, a feature extraction network is used to generate three vectors for each data block: a query vector (used to calculate correlation with other data blocks), a key vector (used to be queried by other data blocks), and a value vector (carrying the actual feature information of the data block). These vectors together constitute the feature representation sequence of the data block.

[0043] When performing multi-head attention analysis, two parallel processing branches, temporal feature analysis and spatial feature analysis, are started simultaneously. For temporal feature analysis, a sliding window mechanism is used, and the window size can be adjusted according to the data characteristics (such as 10 data blocks). In each sliding window, the similarity between the query vector of the current data block and the key vectors of other data blocks in the window is calculated, and the original similarity score is obtained by dot product operation, and then the Softmax function is used for normalization to obtain the temporal attention weight. This process continues as the window slides until all data blocks are processed.

[0044] At the same time, in the spatial feature analysis branch, the data block sequence is reorganized into a two-dimensional grid structure. The grid size can be dynamically adjusted according to the data scale (such as 4×4 or 8×8). In the grid structure, the set of neighboring data blocks is determined for each data block, and the raw score reflecting the spatial correlation is obtained by calculating the correlation between the feature vectors of the target data block and its surrounding data blocks, and the spatial attention weight is obtained through normalization.

[0045] The feature fusion stage is divided into two steps: temporal feature fusion and spatial feature fusion. In temporal feature fusion, a recursive feature fusion method is used to weightedly combine the features of adjacent time windows. Specifically, for each pair of adjacent time windows, the weight coefficient is calculated according to the temporal attention weight, and the features of the two windows are weighted summed to obtain the fused features. This process is recursively performed until the final temporal feature vector is obtained.

[0046] In spatial feature fusion, a hierarchical feature fusion strategy is adopted. Feature aggregation is performed within a local grid range (such as a 2×2 subgrid), and the weighted average of the data block features within the subgrid is calculated. The local aggregated features are used as the input for the next layer of fusion, and feature fusion is performed layer by layer until a vector representing the spatial features of the entire data set is obtained. The time series feature vector and the spatial feature vector are concatenated together, and dimensionality reduction is performed through the fully connected layer to generate the final high-dimensional data feature vector.

[0047] The high-dimensional data feature vector is input into the capsule neural network with dynamic routing for processing. The network contains three layers of capsule structure: the primary capsule layer is responsible for detecting the local feature patterns of the data; the intermediate capsule layer reorganizes and abstracts these local features to form a higher-level feature representation; the advanced capsule layer generates data sharding strategies of different granularities based on these feature representations.

[0048] In the dynamic routing process of the capsule network, connections between capsules at each layer are established through iterative negotiation. Specifically, in each iteration, the similarity between the output of the lower-level capsule and the prediction of the higher-level capsule is calculated, and the connection weights between capsules are updated according to the similarity. After multiple rounds of iterations (usually 3-5 rounds), the optimal data sharding strategy and corresponding adaptive coding parameters are extracted from the higher-level capsule layer.

[0049] Based on the obtained sharding strategy, the data to be written is sharded at multiple levels. First, the data is divided into basic blocks, then temporal shards are constructed based on temporal correlation, spatial shards are constructed based on spatial correlation, and finally, the related shards are combined to form aggregated shards. For each level of sharding, the required redundancy is calculated based on the adaptive coding parameters, and the corresponding redundancy mark is added to the sharding. At the same time, multi-level check information is generated: parity check codes are calculated for basic blocks, erasure codes are generated for shards at different levels, and checksums are calculated for aggregated shards.

[0050] A consistent hash ring is constructed, and multiple virtual nodes (usually 100-200 times the number of actual nodes) are created for each actual storage node on the hash ring. These virtual nodes are evenly distributed on the hash ring through hash functions, and the real-time status information of each storage node is collected through the load perception mechanism, including CPU usage, memory usage, disk I / O and other indicators, and the load score of each node is calculated comprehensively.

[0051] The optimal data distribution scheme is calculated based on the node load score and the distribution of the consistent hash ring. According to the scheme, the multi-level data shards and the corresponding verification information are distributed to the selected storage nodes. After the data distribution is completed, the configuration table is updated to record the location information, redundancy information and verification information of the data shards for subsequent data access.

[0052] In this embodiment, through the combination of multi-head attention mechanism and capsule network, deep mining of data temporal characteristics and spatial characteristics is achieved, the adaptability and accuracy of data sharding strategy are improved, and data storage is made more efficient and reliable. A multi-level data sharding structure and an adaptive redundant coding mechanism are adopted to optimize the utilization of storage space while ensuring data reliability, improve the system's fault tolerance and recovery efficiency, and realize dynamic load balancing of storage nodes based on load-aware consistent hashing algorithm, thereby improving the scalability and stability of the system and reducing the impact of node failures on the overall performance of the system.

[0053] In an optional embodiment, A consistent hash ring of virtual nodes is constructed by a consistent hash algorithm, and a plurality of virtual nodes are evenly distributed on the consistent hash ring for each storage node. A load-aware node selection strategy is adopted to collect the load status of the storage node and calculate the load score of the storage node. The data distribution scheme is calculated according to the load score, including: A consistent hash ring is constructed by a consistent hash algorithm, a first hash operation is performed on the storage node identifier to obtain a reference hash value and determine the reference position of the virtual node, a second hash operation is performed on the reference hash value to obtain a modified hash value and determine the final position of the virtual node on the consistent hash ring, and a plurality of virtual nodes including node attribute information, load monitoring parameters and routing tags are generated for each storage node; The node distribution discreteness of the local area on the consistent hash ring is calculated by a sliding window method, and the area where the node distribution discreteness exceeds a preset threshold is marked as an area to be optimized, and the distribution position of the virtual node is adjusted in the area to be optimized, and the distribution of the virtual node is optimized by a decreasing adjustment step until the virtual nodes are evenly distributed on the consistent hash ring; Adopting a load-aware node selection strategy, obtaining the system resource usage status and business load data of the storage node through a load acquisition agent, hierarchically storing the system resource usage status and the business load data according to the time span, performing data smoothing and abnormal filtering on the stored load data, and analyzing the processed load data to obtain the load status of the storage node; Calculating a system resource load score and a business load score of the storage node based on the load status of the storage node, determining a weight coefficient of the system resource load score and the business load score in combination with the system operation status, and combining the weighted system resource load score and the business load score to obtain a load score of the storage node; Calculate the hash value of the data shard and determine the target position on the consistent hash ring, select multiple virtual nodes within a fixed distance from the target position as candidate nodes, sort the storage nodes corresponding to the candidate nodes based on the load scores and select the storage node with the lowest score as the target storage node, check the load status of the target storage node, and if the load status exceeds a preset load limit, select the storage node with the second lowest score as the target storage node; Calculate the difference in load scores between the storage nodes, screen the storage node pairs whose load score difference exceeds a preset threshold, mark the storage nodes with high load scores in the screened storage node pairs as source nodes, and mark the storage nodes with low load scores as target nodes, calculate the number of data shards that need to be migrated in the source nodes, and generate a data distribution plan that includes the source nodes, the target nodes, and the number of data shards.

[0054] The benchmark hash value refers to a hash value used as a standard or reference in hash calculations, and is used for data comparison, retrieval or verification. The node distribution dispersion refers to the degree of distribution of nodes in the network in space or topology, which is used to measure the distance difference between nodes and affects network performance and load balancing.

[0055] Process the identification information of the storage node. The identification information of the storage node includes the node's IP address, port number and other information. After combining this information into a string, use the SHA-256 hash algorithm to perform the first hash operation to obtain a reference hash value. For example, the IP address of a storage node is 192.168.1.100, and the port number is 8080. The combined identification string is "192.168.1.100:8080". After the SHA-256 hash operation, the reference hash value is obtained.

[0056] After determining the reference position of the virtual node based on the reference hash value, a second hash operation is required to obtain a modified hash value. The second hash operation uses a different hash seed to generate hash values ​​of multiple virtual nodes by appending an increasing serial number after the reference hash value. For example, if 100 virtual nodes are generated for each storage node, a second hash operation is performed after appending serial numbers from 0 to 99 after the reference hash value. This allows the virtual nodes to be more evenly distributed on the hash ring.

[0057] In order to achieve uniform distribution of virtual nodes, it is necessary to evaluate the distribution of nodes on the hash ring. Using the sliding window method, a fixed-size window slides on the hash ring to count the number of virtual nodes in the window. The window size can be set to one-tenth of the hash ring space, and the sliding step is one-quarter of the window size. For each window position, calculate the deviation of the number of nodes in the window from the theoretical average, and mark the area with a deviation of more than 20% as the area to be optimized.

[0058] When adjusting the distribution position of virtual nodes in the area to be optimized, a decreasing adjustment step is used. The initial adjustment step can be set to half the size of the area to be optimized, and the step is reduced to half after each adjustment. During each adjustment, the virtual nodes in the area to be optimized are moved to the left and right blank areas until the node distribution dispersion in the area is lower than the preset threshold or the maximum number of adjustments is reached.

[0059] In the load-aware node selection strategy, the load collection agent collects the system resource usage of the storage node every 10 seconds, including CPU usage, memory usage, disk IO and other indicators. For business load data, such as the number of requests and response time, it is collected once every minute. The collected data is stored in three levels according to the time span: real-time data, hourly data and daily data.

[0060] When processing the collected load data, the moving average method is used to smooth the data, and the window size is 5 sampling points. The outliers are identified by the box plot method, and the data that exceeds 1.5 times the upper and lower quartile range is marked as abnormal and filtered out. The processed load data can more accurately reflect the actual load status of the storage node.

[0061] When calculating the load score of a storage node, the system resource load score is composed of the weighted CPU usage, memory usage, and disk IO. The business load score is composed of the weighted number of requests and average response time. When the system load is light, the weight of the business load score is larger; when the system load is heavy, the weight of the system resource load score increases accordingly.

[0062] When routing data shards, assume that the key value of a data shard is "user_profile_1001", and obtain its position on the hash ring through hash operation. Select virtual nodes within a fixed distance (such as 5% of the hash ring space) clockwise from the position as candidate nodes. If 5 candidate nodes are selected, corresponding to 3 actual storage nodes, they are sorted according to the load scores of these 3 storage nodes.

[0063] During the generation of the data migration plan, when the load score difference between two storage nodes exceeds the preset threshold (such as 30%), the node with higher load is marked as the source node, and the node with lower load is marked as the target node. When calculating the number of data shards that need to be migrated from the source node, the goal is to balance the load of the two nodes, and half of the difference can be used as the proportion that needs to be migrated.

[0064] In this embodiment, the virtual node distribution is constructed by double hashing operation and decreasing step optimization method, which significantly improves the uniformity of node distribution on the consistent hash ring, reduces the uneven distribution of data among storage nodes, and improves the stability and reliability of the storage system. A multi-level load data collection and processing mechanism is adopted, combined with system resource usage and business load data, to achieve accurate evaluation of the load status of storage nodes, provide a reliable basis for load balancing decisions, and effectively improve the efficiency of system resource utilization. Based on load-aware node selection strategy and adaptive data migration scheme, dynamic balance of storage system load is achieved, avoiding the situation where a single node is overloaded, and improving the overall performance and service quality of the system.

[0065] S3. After receiving a data read request, the target data storage location and encoding parameters are retrieved from the updated system initial configuration table, multi-channel parallel reading is started based on the storage location, the multi-level data segmentation and layered verification information are obtained in combination with the pipeline mechanism, the read data is added to the data format conversion model, the data features are extracted in combination with the graph convolution layer, and the optimal conversion path is calculated according to the data feature mapping rule library, and a conversion execution plan is generated and the format conversion is performed to obtain an initial conversion result, and the integrity of the initial conversion result is verified through a two-layer spatial attention network to generate an integrity score. If the integrity score is lower than a preset integrity threshold, a data dependency graph is constructed through a graph attention network, and the defect location is located and repaired in combination with the layered verification information. The reconstruction is repeated until the preset maximum number of reconstructions is reached to obtain the target format data.

[0066] The multi-channel parallel reading is a technology that reads data from multiple data sources or channels in parallel at the same time, aiming to improve data processing efficiency and system throughput. The pipeline mechanism is a mechanism that accelerates the computing process and improves system performance by decomposing the task into multiple consecutive stages and executing each stage in parallel. The optimal conversion path refers to selecting a path that maximizes efficiency or minimizes cost from multiple possible paths, which is usually used in data conversion, network routing or optimization problems. The data dependency graph is a graph structure that describes the dependency relationship between data elements, which is used to indicate which data must be processed first and which can be processed in parallel to optimize the computing process. The hierarchical verification information refers to checking the data through a multi-level verification mechanism to ensure the integrity and correctness of the data at different levels, and is often used in fault tolerance and data verification.

[0067] In an optional embodiment, After receiving the data read request, the target data storage location and encoding parameters are retrieved in the updated system initial configuration table, multi-channel parallel reading is started based on the storage location, the multi-level data sharding and layered verification information are obtained in combination with the pipeline mechanism, the read data is added to the data format conversion model, the data features are extracted in combination with the graph convolution layer, and the optimal conversion path is calculated according to the data feature mapping rule library, and a conversion execution plan is generated and the format conversion is performed to obtain the initial conversion result, and the integrity of the initial conversion result is verified by a two-layer spatial attention network to generate an integrity score. If the integrity score is lower than a preset integrity threshold, a data dependency graph is constructed through a graph attention network, and the defect position is located and repaired in combination with the layered verification information, and reconstruction is repeated until the preset maximum number of reconstructions is reached, and the target format data is obtained, including: Receive a data read request, retrieve the storage location and encoding parameters of the target data in the updated system initial configuration table, start multi-channel parallel reading based on the storage location, and obtain multi-level data sharding and layered verification information in combination with a pipeline mechanism. The pipeline mechanism divides data processing into four processing stages: reading, decoding, verification, and assembly. Each processing stage maintains an independent task queue, and data is transferred between the processing stages through a producer-consumer model; Add the read data to the data format conversion model, build a data structure graph, set the data fields as graph nodes, set the associations between the fields as graph edges, assign weight values ​​to the graph edges according to the business relevance between the fields, extract data features in combination with the graph convolution layer, extract local structural features in the first layer of graph convolution processing, aggregate information of directly connected nodes for each graph node, fuse adjacent node information in the second layer of graph convolution processing, combine the first layer node features with the features of second-order adjacent nodes, obtain global structural features in the third layer of graph convolution processing, fuse the information of all nodes in the data structure graph, and generate a data feature vector; Retrieving conversion rules in a data feature mapping rule library according to the data feature vector, calculating feature similarity between the feature vector and a preset conversion template, and selecting the conversion template with the highest similarity to calculate an optimal conversion path; Generate a conversion execution plan based on the optimal conversion path and perform format conversion, adjust the hierarchical structure of the source data format to the hierarchical structure of the target format in the structural reorganization phase, establish a corresponding relationship between the source format field and the target format field in the field mapping phase, and convert the field value into the data type required by the target format in the data conversion phase to obtain an initial conversion result; Verify the integrity of the initial conversion result through a two-layer spatial attention network, check the existence, value range and type matching of the data field at the local attention layer, calculate the field-level integrity score, verify the primary and foreign key relationships, uniqueness constraints and business rules between data records at the global attention layer, calculate the record-level integrity score, and combine the field-level integrity score and the record-level integrity score to generate an integrity score; If the integrity score is lower than a preset integrity threshold, a data dependency graph is constructed through a graph attention network, data fields are set as graph nodes, dependencies between fields are set as graph edges, weight values ​​are assigned to the graph edges based on the degree of business impact, the defect location is located in combination with the hierarchical verification information, the repair priority is determined based on the weight of the graph edge, the abnormal fields are sorted in descending order according to the impact range, and the abnormal fields are repaired according to the sorting results, and data reconstruction is performed in an iterative manner; The integrity score is recalculated after each round of reconstruction, and the reconstruction is repeated until the integrity score reaches the integrity threshold or reaches a preset maximum number of reconstruction times, thereby obtaining target format data.

[0068] The producer-consumer pattern is a design pattern commonly used in multi-threaded or multi-process systems. Producers generate data and consumers process data, and they collaborate through buffers to improve system efficiency. The conversion template is a template used to define data conversion rules or steps, which helps to standardize and automate the conversion process between different data formats. The field mapping stage is a step in the data conversion process. By matching the source data field with the target data field, it ensures that the data remains accurate during the conversion process. The business impact level refers to the impact of a business operation or decision on the system, user or the entire business process, and is used to evaluate and optimize business strategies.

[0069] The data reading phase first receives a data reading request initiated by the user, which contains the identification information of the target data. The system retrieves the storage location and encoding parameters corresponding to the identification information in the updated initial configuration table. The storage location may be distributed on different storage nodes, such as different data nodes in a distributed file system. Taking medical data as an example, the basic information of the patient may be stored in node A, the test data is stored in node B, and the imaging data is stored in node C.

[0070] Based on the retrieved storage location information, the system starts a multi-channel parallel reading mechanism. Each storage location is assigned an independent reading thread, which executes data reading tasks in parallel. The pipeline mechanism is used to divide data processing into multiple stages: reading raw data, decoding data, data verification, and data assembly. An independent task queue is set up for each processing stage, and data is transferred between stages through the producer-consumer model.

[0071] The data format conversion phase inputs the read data into the conversion model. First, a data structure graph is constructed, with each data field as a graph node and the relationship between fields as graph edges. For example, in medical data, there is a relationship between the patient ID and other information, and the system will assign weight values ​​to the graph edges based on the business relevance of these fields.

[0072] Then, data features are extracted through a multi-layer graph convolution network. The first layer of graph convolution processing focuses on local structural features and aggregates the information of directly connected nodes for each node. The second layer of graph convolution fuses the information of adjacent nodes and combines the first-order node features with the second-order adjacent node features. The third layer of graph convolution obtains global structural features, fuses the information of all nodes in the entire data structure graph, and finally generates a feature vector.

[0073] The system retrieves the conversion rules in the rule library based on the feature vector, calculates the similarity between the feature vector and the preset conversion template, selects the most similar conversion template, and generates the optimal conversion path. For example, when converting XML format to JSON format, the system will select the most suitable conversion template.

[0074] The conversion execution phase includes three steps: structural reorganization, field mapping, and data conversion. Structural reorganization adjusts the hierarchical structure of the source data, field mapping establishes the field correspondence between the source format and the target format, and data conversion completes the specific data type conversion.

[0075] The integrity verification phase uses a two-layer spatial attention network for verification. The local attention layer checks the integrity of the data field, including whether the field exists, whether the value is within the valid range, whether the data type matches, etc. The global attention layer verifies the integrity of the relationships between data records, such as primary and foreign key relationships, unique constraints, etc. The two-layer verification results are combined to generate the final integrity score.

[0076] When the integrity score is lower than the threshold, the system constructs a data dependency graph through the graph attention network to locate the defect. Dependencies are weighted according to the business impact and repair priorities are determined. The system will prioritize repairing abnormal fields with a larger impact range and reconstruct data in an iterative manner until the integrity score reaches the threshold or the maximum number of reconstructions is reached.

[0077] In this embodiment, through multi-channel parallel reading and pipeline mechanism, the data processing efficiency is significantly improved, the time overhead of data reading and conversion is reduced, and the overall performance of the system is improved. The graph convolutional network and attention mechanism are used to extract data features, and more accurate data format conversion is achieved. The information loss during the conversion process is reduced, and the data quality is guaranteed. A two-layer spatial attention network is introduced for integrity verification, and the graph attention network is combined for defect repair, which improves the reliability and accuracy of the data conversion results and ensures the integrity and consistency of the converted data.

[0078] In an optional embodiment, A data dependency graph is constructed through a graph attention network, data fields are set as graph nodes, dependencies between fields are set as graph edges, weight values ​​are assigned to the graph edges based on the degree of business impact, the defect location is located in combination with the hierarchical verification information, the repair priority is determined based on the weight of the graph edge, the abnormal fields are sorted in descending order according to the impact range, and the abnormal fields are repaired according to the sorting results. Data reconstruction in combination with an iterative method includes: Construct a data dependency graph through a graph attention network, set data fields as graph nodes, parse database constraint information to extract dependencies between fields to construct reference dependency edges, parse triggers and stored procedures to construct calculation dependency edges, parse field update rules to construct timing dependency edges, analyze data content to calculate the mutual information value of field value distribution to construct association dependency edges, analyze the synchronous change pattern of field values ​​to construct collaborative dependency edges, detect the hierarchical inclusion relationship of field values ​​to construct nested dependency edges, and set the reference dependency edges, the calculation dependency edges, the timing dependency edges, the association dependency edges, the collaborative dependency edges, and the nested dependency edges as graph edges of the data dependency graph; Based on the business impact, weight values ​​are assigned to the graph edges of the data dependency graph. At the dependency type layer, the base weight of the reference dependency edge is set to a first weight value, the base weight of the calculation dependency edge is set to a second weight value, and the base weight of the timing dependency edge is set to a third weight value. At the data feature layer, the weight values ​​of the graph edges are adjusted based on the reference data volume ratio, calculation complexity, and time correlation strength. At the business importance layer, the weight values ​​of the graph edges of the dependency relationships involved in the core business process are increased. A three-level verification rule tree is constructed based on hierarchical verification information. The first-level rules check the non-emptiness, type matching, and length constraints of the field, and mark the fields that violate the basic constraints as the first-level abnormal nodes. The second-level rules check the value range, enumeration value set, and field combination constraints of the field, and mark the fields that violate the business rules as the second-level abnormal nodes. The third-level rules check the consistency, timeliness, and accuracy standards of the data, and mark the fields that violate the quality requirements as the third-level abnormal nodes. In the data dependency graph, starting from each abnormal node, traverse all reachable nodes downstream along the graph edge, calculate the impact attenuation coefficient between nodes based on the cumulative graph edge weight of the dependency path, sum the impact attenuation coefficients of the abnormal node on all reachable nodes to obtain the impact range score, and sort the abnormal fields in descending order according to the impact range score to obtain a repair priority sequence; The abnormal fields are repaired in sequence according to the repair priority sequence, the values ​​of the numerical abnormal fields are randomly generated based on the historical data of the time series, the categories of the categorical abnormal fields are inferred based on the association rules between the fields, and the complete information of the text abnormal fields is supplemented based on the string similarity matching, and the repaired field values ​​are updated to the original data; The data is reconstructed in an iterative manner, node features and impact range scores are recalculated on the data dependency graph, the abnormal node set is updated and the data quality score is calculated, and the quality scores of two adjacent rounds are compared. When the score improvement is greater than a preset score improvement threshold or reaches a preset maximum number of iterations, the repair result is output.

[0079] The time series dependency edge refers to an edge in time series data that indicates a dependency relationship between different time points, and is used to model and analyze causal relationships in time series data. The impact attenuation coefficient is a coefficient used to describe the gradual weakening of the impact over time or space, and is often used in models to control the scope and intensity of the impact. The impact range score is a quantitative description of the impact range of a factor or event, and is used to analyze and measure the influence or diffusion degree of different factors on the system.

[0080] The data dependency graph is constructed through the graph attention network, and the fields in the data table are used as nodes of the graph. Different types of dependency edges are constructed through multi-dimensional analysis. In the specific implementation, the foreign key constraint information of the database is first parsed to identify the reference relationship between the fields. For example, the user ID field of the order table references the primary key ID of the user table, and such relationships are constructed as reference dependency edges. Then the triggers and stored procedures in the database are analyzed to extract the field calculation logic. For example, the total order price is calculated based on the order details, and such relationships are constructed as calculation dependency edges. Then the data update rules are parsed to identify the update order of the field values, such as the order status from "pending payment" to "paid" and then to "shipped" status flow, and such relationships are constructed as time-series dependency edges.

[0081] For the correlation analysis of data content, calculate the mutual information value of the value distribution between fields. Taking user age and consumption amount as an example, by analyzing the value distribution of these two fields, we can find the correlation pattern between age group and consumption level, and build correlation dependency edges between pairs of fields with significant correlation. At the same time, observe the synchronous change rules of field values, such as the inventory quantity and sales quantity of goods often show a trend of increase and decrease, and build collaborative dependency edges between such synchronously changing field pairs. In addition, it is necessary to detect the hierarchical inclusion relationship of field values, such as the geographical hierarchical relationship between provinces, cities, and counties, and build such inclusion relationships as nested dependency edges.

[0082] After completing the construction of the graph, you need to assign reasonable weight values ​​to the graph edges. At the dependency type level, set the baseline weight of the reference dependency edge to 0.8, the calculation dependency edge to 0.6, and the timing dependency edge to 0.4. Adjust the weights at the data feature level. For example, for reference dependency edges, if the amount of referenced data accounts for a large proportion, increase the weight appropriately; for calculation dependency edges, adjust the weight according to the complexity of the calculation logic; for timing dependency edges, adjust the weight according to the closeness of the time association. At the business importance level, for dependencies involving core business processes such as order processing and payment settlement, increase their weights by 20%.

[0083] Construct a three-level validation rule tree to check data quality. The first-level rules mainly check basic constraints, such as whether the user name is empty, whether the mobile phone number is 11 digits, etc. The second-level rules check business constraints, such as whether the age is within a reasonable range, whether the order status belongs to a predefined status set, etc. The third-level rules check advanced quality requirements, such as whether the order information is consistent with the payment information, whether the product price is updated in a timely manner, etc.

[0084] After locating the abnormal node based on the inspection results, perform an impact range analysis in the dependency graph. Starting from each abnormal node, find all reachable nodes through the graph traversal algorithm. During the traversal process, calculate the impact attenuation based on the edge weights on the path. For example, if an abnormal node is connected to a downstream node through an edge with a weight of 0.8, the impact on the downstream node is the original impact value multiplied by 0.8. Accumulate the impact values ​​of the abnormal node on all reachable nodes to obtain the impact range score of the abnormal node.

[0085] Prioritize and repair abnormal fields based on the impact range score. For numerical anomalies, such as abnormal product prices, a reasonable value can be randomly generated based on the fluctuation pattern of the product's historical prices. For category anomalies, such as abnormal product classification, the most likely classification can be inferred through association rules based on the values ​​of related fields such as product name and description. For text anomalies, such as incomplete product description information, complete information can be supplemented from the descriptions of similar products through text similarity matching.

[0086] The repair effect is continuously optimized in an iterative manner. After each round of iteration, the characteristic value and influence range of the node are recalculated, the abnormal node set is updated, and the data quality score is calculated. When the quality score of two consecutive rounds is improved by more than 5% or reaches 10 iterations, the iteration process is terminated and the final repair result is output.

[0087] In this embodiment, by constructing a data dependency graph containing multiple dependencies, the correlation between data fields is comprehensively characterized, the accuracy and completeness of data quality analysis are improved, and based on the graph attention network and multi-level weight distribution mechanism, the impact range of abnormal data is accurately quantified, the accurate positioning and priority sorting of abnormal repair are achieved, and the efficiency of data repair is improved. An iterative optimization strategy is adopted to dynamically adjust the repair plan to ensure continuous improvement of data quality. At the same time, through the constraints of multi-level verification rules, the accuracy and reliability of the repair results are guaranteed.

[0088] Figure 2 FIG. 1 is a schematic diagram of the structure of an erasure code compatible read-write system based on a bidirectional data access agent according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to collect the computing power and storage status of each storage node in the system, and encode and construct a heterogeneous resource portrait in combination with a multi-layer perceptron. Based on the heterogeneous resource portrait, the node state sequence is input into the multi-agent reinforcement learning model, and the optimal task allocation strategy is obtained through interactive iterative calculation of the policy network and the value network. According to the optimal task allocation strategy, a read agent module and a write agent module are deployed in the system, and a data format conversion model is constructed using a space-time graph convolutional network. The model parameters are distributed to the storage nodes based on a federated learning framework, and the data format conversion model is optimized through local training and global aggregation iterations to generate a data feature mapping rule base. The data feature mapping rule base and the erasure code configuration parameters are used to establish a multi-level index structure through a hierarchical Bloom filter, and the query performance is optimized in combination with a skip table mechanism to obtain a system initial configuration table; The second unit is used for, after receiving a data write request, performing multi-head attention analysis on the data to be written through an encoder based on the attention mechanism, extracting the temporal features and spatial features of the data to be written to generate a high-dimensional data feature vector, adding the high-dimensional data feature vector to the capsule neural network of dynamic routing, calculating the optimal sharding strategy and adaptive coding parameters through iterative negotiation between capsules, performing hierarchical segmentation on the data to be written based on the optimal sharding strategy, generating multi-level data sharding and hierarchical verification information with redundancy marks in combination with the adaptive coding parameters, calculating the data distribution plan through a consistent hashing algorithm and a load-aware node selection strategy, sending the multi-level data sharding and the hierarchical verification information to the storage node based on the data distribution plan, and updating the system initial configuration table; The third unit is used to retrieve the target data storage location and encoding parameters in the updated system initial configuration table after receiving a data read request, start multi-channel parallel reading based on the storage location, obtain the multi-level data segmentation and layered verification information in combination with the pipeline mechanism, add the read data to the data format conversion model, extract data features in combination with the graph convolution layer and calculate the optimal conversion path according to the data feature mapping rule library, generate a conversion execution plan and perform format conversion to obtain an initial conversion result, verify the integrity of the initial conversion result through a two-layer spatial attention network, generate an integrity score, and if the integrity score is lower than a preset integrity threshold, construct a data dependency graph through the graph attention network, locate the defect location in combination with the layered verification information and repair it, repeat the reconstruction until the preset maximum number of reconstructions is reached, and obtain the target format data.

[0089] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0090] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0091] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An erasure code compatible reading and writing method based on a bidirectional data access agent, characterized in that: include: The computing power and storage status of each storage node in the system are collected, and the heterogeneous resource portrait is constructed by encoding with a multi-layer perceptron. Based on the heterogeneous resource portrait, the node state sequence is input into the multi-agent reinforcement learning model, and the optimal task allocation strategy is obtained through interactive iterative calculation of the policy network and the value network. According to the optimal task allocation strategy, the read agent module and the write agent module are deployed in the system, and the data format conversion model is constructed by using a space-time graph convolutional network. The model parameters are distributed to the storage nodes based on the federated learning framework, and the data format conversion model is optimized through local training and global aggregation iterations to generate a data feature mapping rule base. The data feature mapping rule base and the erasure code configuration parameters are used to establish a multi-level index structure through a hierarchical Bloom filter, and the query performance is optimized in combination with the skip table mechanism to obtain the system initial configuration table; After receiving a data write request, a multi-head attention analysis is performed on the data to be written through an encoder based on an attention mechanism, and the temporal features and spatial features of the data to be written are extracted to generate a high-dimensional data feature vector, and the high-dimensional data feature vector is added to the capsule neural network of dynamic routing, and the optimal sharding strategy and adaptive coding parameters are calculated through iterative negotiation between capsules. The data to be written is layered and segmented based on the optimal sharding strategy, and multi-level data shards and layered verification information with redundancy marks are generated in combination with the adaptive coding parameters. The data distribution plan is calculated through a consistent hashing algorithm and a load-aware node selection strategy, and the multi-level data shards and the layered verification information are sent to the storage node based on the data distribution plan and the system initial configuration table is updated; After receiving a data read request, the target data storage location and encoding parameters are retrieved in the updated system initial configuration table, multi-channel parallel reading is started based on the storage location, the multi-level data segmentation and layered verification information are obtained in combination with the pipeline mechanism, the read data is added to the data format conversion model, the data features are extracted in combination with the graph convolution layer, and the optimal conversion path is calculated according to the data feature mapping rule library, and a conversion execution plan is generated and the format conversion is performed to obtain an initial conversion result, and the integrity of the initial conversion result is verified through a two-layer spatial attention network to generate an integrity score. If the integrity score is lower than a preset integrity threshold, a data dependency graph is constructed through a graph attention network, and the defect location is located and repaired in combination with the layered verification information. The reconstruction is repeated until the preset maximum number of reconstructions is reached to obtain the target format data.

2. The method according to claim 1, characterized in that The computing power and storage status of each storage node in the collection system are combined with a multi-layer perceptron for encoding to construct a heterogeneous resource portrait. Based on the heterogeneous resource portrait, the node state sequence is input into the multi-agent reinforcement learning model, and the optimal task allocation strategy is obtained through interactive iterative calculation of the policy network and the value network. According to the optimal task allocation strategy, the read agent module and the write agent module are deployed in the system, and the space-time graph convolution network is used to build a data format conversion model. The model parameters are distributed to the storage nodes based on the federated learning framework. The data format conversion model is optimized through local training and global aggregation iterations to generate a data feature mapping rule base. The data feature mapping rule base and the erasure code configuration parameters are used to establish a multi-level index structure through a hierarchical Bloom filter. The query performance is optimized in combination with the skip table mechanism. The system initial configuration table includes: Deploy distributed monitoring probes on each storage node, collect computing power and storage status of the storage node through the distributed monitoring probes to obtain raw monitoring data, pre-process the raw monitoring data, eliminate short-term fluctuations in processor and memory usage data through a sliding window averaging method, convert storage capacity data into a ratio of used space to total space, and normalize input and output throughput data based on historical peak values ​​to obtain pre-processed data; A multilayer perceptron with a residual connection structure is used to extract features from the preprocessed data. The first hidden layer of the multilayer perceptron extracts independent features of various indicators. The second hidden layer learns the combined features between indicators through nonlinear transformation. The output layer generates feature vectors to construct heterogeneous resource portraits. Organizing the feature vector into a node state sequence by means of a sliding window, inputting the node state sequence into a multi-agent reinforcement learning model, each agent maintaining a corresponding node feature sequence, task queue status and resource reservation status, processing the node state sequence by means of a hierarchical state encoder, wherein the hierarchical state encoder includes a bottom encoder for processing continuous features, a middle encoder for extracting a temporal pattern and a top encoder for integrating information, and outputting a state code; The strategy network of the multi-agent reinforcement learning model includes an action network and an evaluation network. The action network outputs an action probability distribution based on the state encoding, and the evaluation network evaluates the state value. A dynamic communication topology between agents is constructed based on the task dependency relationship and resource sharing degree between nodes. Neighbor information sequences are processed through gated recurrent units, and hierarchical reinforcement learning is used for decision decomposition to obtain an optimal task allocation strategy. Deploy the reading proxy module and the writing proxy module according to the optimal task allocation strategy, and use the space-time graph convolution network with a dual-stream architecture to build a data format conversion model, wherein the space-time graph convolution layer of the space-time graph convolution network aggregates neighbor features through a message passing mechanism to update the node representation, and the temporal convolution layer uses a hole convolution to expand the receptive field to capture data dependencies, and introduces a hierarchical attention mechanism of structural attention and temporal attention to process features, thereby obtaining a data format conversion model; Based on the federated learning framework, the model parameters of the data format conversion model are distributed, a hierarchical training strategy is adopted to locally train the basic feature extractor, and high-level features are globally optimized. The pre-trained model is used to guide the training process through progressive knowledge distillation to generate a data feature mapping rule base; Construct a sharded hierarchical Bloom filter, divide the bitmap of the hierarchical Bloom filter into multiple independently maintained shards, establish a multi-level index structure for the data feature mapping rule base and erasure code configuration parameters through the sharded hierarchical Bloom filter, determine the index level through a random layer number generation algorithm based on a dynamically balanced skip table mechanism and the multi-level index structure, maintain a data version chain in the index node to record the version evolution relationship, adopt a hierarchical filtering strategy to perform query optimization, and obtain the system initial configuration table.

3. The method according to claim 2, characterized in that The strategy network of the multi-agent reinforcement learning model includes an action network and an evaluation network. The action network outputs the action probability distribution based on the state encoding, and the evaluation network evaluates the state value. The dynamic communication topology between agents is constructed based on the task dependency relationship and resource sharing degree between nodes. The neighbor information sequence is processed by the gated recurrent unit, and the decision decomposition is performed by hierarchical reinforcement learning. The optimal task allocation strategy includes: Constructing a policy network of a multi-agent reinforcement learning model, the policy network includes an action network and an evaluation network, receiving state encoding data, the state encoding data includes a computing resource dimension, a storage resource dimension, and a network resource dimension, wherein the computing resource dimension records processor usage information, memory occupancy information, and process quantity information, the storage resource dimension records disk read and write information, storage space usage information, and storage queue length information, and the network resource dimension records bandwidth usage information, network transmission delay information, and data packet loss information; Adding the state encoding data to the action network, performing data normalization transformation and reducing the dimension through the first layer to obtain standardized features, extracting the correlation patterns in the standardized features and constructing combined features through the second layer, and generating action probability distribution according to the combined features; Based on the evaluation network, the state encoding data is processed in parallel, the mutual influence relationship between different dimensional indicators is analyzed by the feature interaction processing unit to obtain the interaction feature, and the interaction feature is mapped through the neural network to obtain the state value; Statistically record the task flow information between nodes during the operation of the system, calculate the task dependency between nodes according to the task flow frequency, obtain the resource sharing degree according to the storage resource sharing state between nodes, and construct the dynamic communication topology between intelligent agents according to the task dependency and the resource sharing degree; Determine a neighbor node set of each intelligent agent based on the dynamic communication topology, process a neighbor information sequence in the neighbor node set through a gated recurrent unit, calculate an information update weight based on the change amplitude between the current state and the historical state using an information update control unit, calculate a historical information retention weight based on the time correlation of the state information using a historical information control unit, and update the node state information according to the information update weight and the historical information retention weight; Hierarchical reinforcement learning is used to perform decision decomposition. Task feature information is extracted in the first-level decision to obtain task computing load features and task data access features. The task computing load features include processor time consumption information, memory usage information, and storage access information. The task data access features include data storage location information, data transmission scale information, and data access regularity information. In the second-level decision, the task feature information, the node state information, the action probability distribution and the state value are used as input to determine the target execution node; Based on the dynamic communication topology and the target execution nodes, the policy network is trained in a distributed manner. Each target execution node is independently trained to obtain network parameters based on local task execution records. The network parameters are exchanged and integrated between target execution nodes with communication connections to generate an optimal task allocation strategy, wherein the optimal task allocation strategy includes a correspondence between task types and target nodes.

4. The method according to claim 1, characterized in that: After receiving a data write request, a multi-head attention analysis is performed on the data to be written through an encoder based on an attention mechanism, and the temporal features and spatial features of the data to be written are extracted to generate a high-dimensional data feature vector, and the high-dimensional data feature vector is added to the capsule neural network of dynamic routing, and the optimal sharding strategy and adaptive coding parameters are calculated through iterative negotiation between capsules, and the data to be written is hierarchically segmented based on the optimal sharding strategy, and multi-level data shards and hierarchical verification information with redundancy marks are generated in combination with the adaptive coding parameters, and a data distribution scheme is calculated through a consistent hashing algorithm and a load-aware node selection strategy, and the multi-level data shards and the hierarchical verification information are sent to the storage node based on the data distribution scheme and the system initial configuration table is updated, including: Receive a data write request, and perform multi-head attention analysis on the data to be written through an encoder based on the attention mechanism, wherein the encoder includes multiple parallel attention processing heads, wherein the first group of attention processing heads is responsible for analyzing temporal features, and the second group of attention processing heads is responsible for analyzing spatial features. Each data block generates a query vector, a key vector, and a value vector through a feature extraction layer, and the combination obtains a data block sequence; The data block sequence is processed by a sliding window mechanism, the temporal features of the data to be written are extracted, the similarity between the query vector of the current data block and the key vectors of other data blocks in the window is calculated within the sliding window, and the temporal attention weight is obtained through normalization processing, the data block sequence is processed by a grid partitioning method, the spatial features of the data to be written are extracted, the adjacent data blocks are organized into a two-dimensional grid structure, and the spatial correlation between the target data block and the surrounding data blocks is calculated to obtain the spatial attention weight; Recursively perform feature fusion on the time series features, perform weighted combination of features of adjacent time windows to obtain a time series feature vector, perform hierarchical feature fusion on the spatial features, perform feature aggregation in a local grid and then fuse layer by layer to obtain a spatial feature vector, concatenate the time series feature vector and the spatial feature vector and generate the high-dimensional data feature vector through full connection layer mapping; Adding the high-dimensional data feature vector to a capsule neural network of dynamic routing, the capsule neural network comprises a primary capsule layer, an intermediate capsule layer and an advanced capsule layer, detecting local feature patterns through the primary capsule layer, reorganizing and abstracting the features of the primary capsule layer through the intermediate capsule layer, and representing data sharding strategies of different granularities through the advanced capsule layer; The primary capsule layer, the intermediate capsule layer and the high-level capsule layer are connected through iterative negotiation between capsules, the feature similarity between the low-level capsules and the high-level capsules is calculated and the connection weight is updated in each round of the iterative negotiation, and the optimal fragmentation strategy and adaptive coding parameters are extracted from the high-level capsule layer; Based on the optimal sharding strategy, the data to be written is divided into layers, a multi-level data shard including a basic block, a time-series shard, a space shard and an aggregate shard is constructed, the redundancy mark of the multi-level data shard is calculated according to the adaptive coding parameter, the multi-level data shard with the redundancy mark is generated, and the hierarchical check information is generated by calculating the parity check code of the basic block, the erasure code corresponding to different shards and the checksum of the aggregate shard; A consistent hash ring of virtual nodes is constructed through a consistent hash algorithm and multiple virtual nodes are evenly distributed on the consistent hash ring for each storage node. A load-aware node selection strategy is adopted to collect the load status of the storage node and calculate the load score of the storage node. A data distribution plan is calculated according to the load score. Based on the data distribution plan, the multi-level data shards and the hierarchical verification information are sent to the storage node and the system initial configuration table is updated.

5. The method according to claim 4, characterized in that A consistent hash ring of virtual nodes is constructed by a consistent hash algorithm, and a plurality of virtual nodes are evenly distributed on the consistent hash ring for each storage node. A load-aware node selection strategy is adopted to collect the load status of the storage node and calculate the load score of the storage node. The data distribution scheme is calculated according to the load score, including: A consistent hash ring is constructed by a consistent hash algorithm, a first hash operation is performed on the storage node identifier to obtain a reference hash value and determine the reference position of the virtual node, a second hash operation is performed on the reference hash value to obtain a modified hash value and determine the final position of the virtual node on the consistent hash ring, and a plurality of virtual nodes including node attribute information, load monitoring parameters and routing tags are generated for each storage node; The node distribution discreteness of the local area on the consistent hash ring is calculated by a sliding window method, and the area where the node distribution discreteness exceeds a preset threshold is marked as an area to be optimized, and the distribution position of the virtual node is adjusted in the area to be optimized, and the distribution of the virtual node is optimized by a decreasing adjustment step until the virtual nodes are evenly distributed on the consistent hash ring; Adopting a load-aware node selection strategy, obtaining the system resource usage status and business load data of the storage node through a load acquisition agent, hierarchically storing the system resource usage status and the business load data according to the time span, performing data smoothing and abnormal filtering on the stored load data, and analyzing the processed load data to obtain the load status of the storage node; Calculating a system resource load score and a business load score of the storage node based on the load status of the storage node, determining a weight coefficient of the system resource load score and the business load score in combination with the system operation status, and combining the weighted system resource load score and the business load score to obtain a load score of the storage node; Calculate the hash value of the data shard and determine the target position on the consistent hash ring, select multiple virtual nodes within a fixed distance from the target position as candidate nodes, sort the storage nodes corresponding to the candidate nodes based on the load scores and select the storage node with the lowest score as the target storage node, check the load status of the target storage node, and if the load status exceeds a preset load limit, select the storage node with the second lowest score as the target storage node; Calculate the difference in load scores between the storage nodes, screen the storage node pairs whose load score difference exceeds a preset threshold, mark the storage nodes with high load scores in the screened storage node pairs as source nodes, and mark the storage nodes with low load scores as target nodes, calculate the number of data shards that need to be migrated in the source nodes, and generate a data distribution plan that includes the source nodes, the target nodes, and the number of data shards.

6. The method according to claim 1, characterized in that After receiving the data read request, the target data storage location and encoding parameters are retrieved in the updated system initial configuration table, multi-channel parallel reading is started based on the storage location, the multi-level data sharding and layered verification information are obtained in combination with the pipeline mechanism, the read data is added to the data format conversion model, the data features are extracted in combination with the graph convolution layer, and the optimal conversion path is calculated according to the data feature mapping rule library, and a conversion execution plan is generated and the format conversion is performed to obtain the initial conversion result, and the integrity of the initial conversion result is verified by a two-layer spatial attention network to generate an integrity score. If the integrity score is lower than a preset integrity threshold, a data dependency graph is constructed through a graph attention network, and the defect position is located and repaired in combination with the layered verification information, and reconstruction is repeated until the preset maximum number of reconstructions is reached, and the target format data is obtained, including: Receive a data read request, retrieve the storage location and encoding parameters of the target data in the updated system initial configuration table, start multi-channel parallel reading based on the storage location, and obtain multi-level data sharding and layered verification information in combination with a pipeline mechanism. The pipeline mechanism divides data processing into four processing stages: reading, decoding, verification, and assembly. Each processing stage maintains an independent task queue, and data is transferred between the processing stages through a producer-consumer model; Add the read data to the data format conversion model, build a data structure graph, set the data fields as graph nodes, set the associations between the fields as graph edges, assign weight values ​​to the graph edges according to the business relevance between the fields, extract data features in combination with the graph convolution layer, extract local structural features in the first layer of graph convolution processing, aggregate information of directly connected nodes for each graph node, fuse adjacent node information in the second layer of graph convolution processing, combine the first layer node features with the features of second-order adjacent nodes, obtain global structural features in the third layer of graph convolution processing, fuse the information of all nodes in the data structure graph, and generate a data feature vector; Retrieving conversion rules in a data feature mapping rule library according to the data feature vector, calculating feature similarity between the feature vector and a preset conversion template, and selecting the conversion template with the highest similarity to calculate an optimal conversion path; Generate a conversion execution plan based on the optimal conversion path and perform format conversion, adjust the hierarchical structure of the source data format to the hierarchical structure of the target format in the structural reorganization phase, establish a corresponding relationship between the source format field and the target format field in the field mapping phase, and convert the field value into the data type required by the target format in the data conversion phase to obtain an initial conversion result; Verify the integrity of the initial conversion result through a two-layer spatial attention network, check the existence, value range and type matching of the data field at the local attention layer, calculate the field-level integrity score, verify the primary and foreign key relationships, uniqueness constraints and business rules between data records at the global attention layer, calculate the record-level integrity score, and combine the field-level integrity score and the record-level integrity score to generate an integrity score; If the integrity score is lower than a preset integrity threshold, a data dependency graph is constructed through a graph attention network, data fields are set as graph nodes, dependencies between fields are set as graph edges, weight values ​​are assigned to the graph edges based on the degree of business impact, the defect location is located in combination with the hierarchical verification information, the repair priority is determined based on the weight of the graph edge, the abnormal fields are sorted in descending order according to the impact range, and the abnormal fields are repaired according to the sorting results, and data reconstruction is performed in an iterative manner; The integrity score is recalculated after each round of reconstruction, and the reconstruction is repeated until the integrity score reaches the integrity threshold or reaches a preset maximum number of reconstruction times, thereby obtaining target format data.

7. The method according to claim 6, characterized in that A data dependency graph is constructed through a graph attention network, data fields are set as graph nodes, dependencies between fields are set as graph edges, weight values ​​are assigned to the graph edges based on the degree of business impact, the defect location is located in combination with the hierarchical verification information, the repair priority is determined based on the weight of the graph edge, the abnormal fields are sorted in descending order according to the impact range, and the abnormal fields are repaired according to the sorting results. Data reconstruction in combination with an iterative method includes: Construct a data dependency graph through a graph attention network, set data fields as graph nodes, parse database constraint information to extract dependencies between fields to construct reference dependency edges, parse triggers and stored procedures to construct calculation dependency edges, parse field update rules to construct timing dependency edges, analyze data content to calculate the mutual information value of field value distribution to construct association dependency edges, analyze the synchronous change pattern of field values ​​to construct collaborative dependency edges, detect the hierarchical inclusion relationship of field values ​​to construct nested dependency edges, and set the reference dependency edges, the calculation dependency edges, the timing dependency edges, the association dependency edges, the collaborative dependency edges, and the nested dependency edges as graph edges of the data dependency graph; Based on the business impact, weight values ​​are assigned to the graph edges of the data dependency graph. At the dependency type layer, the base weight of the reference dependency edge is set to a first weight value, the base weight of the calculation dependency edge is set to a second weight value, and the base weight of the timing dependency edge is set to a third weight value. At the data feature layer, the weight values ​​of the graph edges are adjusted based on the reference data volume ratio, calculation complexity, and time correlation strength. At the business importance layer, the weight values ​​of the graph edges of the dependency relationships involved in the core business process are increased. A three-level verification rule tree is constructed based on hierarchical verification information. The first-level rules check the non-emptiness, type matching, and length constraints of the field, and mark the fields that violate the basic constraints as the first-level abnormal nodes. The second-level rules check the value range, enumeration value set, and field combination constraints of the field, and mark the fields that violate the business rules as the second-level abnormal nodes. The third-level rules check the consistency, timeliness, and accuracy standards of the data, and mark the fields that violate the quality requirements as the third-level abnormal nodes. In the data dependency graph, starting from each abnormal node, traverse all reachable nodes downstream along the graph edge, calculate the impact attenuation coefficient between nodes based on the cumulative graph edge weight of the dependency path, sum the impact attenuation coefficients of the abnormal node on all reachable nodes to obtain the impact range score, and sort the abnormal fields in descending order according to the impact range score to obtain a repair priority sequence; The abnormal fields are repaired in sequence according to the repair priority sequence, the values ​​of the numerical abnormal fields are randomly generated based on the historical data of the time series, the categories of the categorical abnormal fields are inferred based on the association rules between the fields, and the complete information of the text abnormal fields is supplemented based on the string similarity matching, and the repaired field values ​​are updated to the original data; The data is reconstructed in an iterative manner, node features and impact range scores are recalculated on the data dependency graph, the abnormal node set is updated and the data quality score is calculated, and the quality scores of two adjacent rounds are compared. When the score improvement is greater than a preset score improvement threshold or reaches a preset maximum number of iterations, the repair result is output.

8. An erasure code compatible read / write system for a bidirectional data access agent, used to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is used to collect the computing power and storage status of each storage node in the system, and encode and construct a heterogeneous resource portrait in combination with a multi-layer perceptron. Based on the heterogeneous resource portrait, the node state sequence is input into the multi-agent reinforcement learning model, and the optimal task allocation strategy is obtained through interactive iterative calculation of the policy network and the value network. According to the optimal task allocation strategy, a read agent module and a write agent module are deployed in the system, and a data format conversion model is constructed using a space-time graph convolutional network. The model parameters are distributed to the storage nodes based on a federated learning framework, and the data format conversion model is optimized through local training and global aggregation iterations to generate a data feature mapping rule base. The data feature mapping rule base and the erasure code configuration parameters are used to establish a multi-level index structure through a hierarchical Bloom filter, and the query performance is optimized in combination with a skip table mechanism to obtain a system initial configuration table; The second unit is used for, after receiving a data write request, performing multi-head attention analysis on the data to be written through an encoder based on the attention mechanism, extracting the temporal features and spatial features of the data to be written to generate a high-dimensional data feature vector, adding the high-dimensional data feature vector to the capsule neural network of dynamic routing, calculating the optimal sharding strategy and adaptive coding parameters through iterative negotiation between capsules, performing hierarchical segmentation on the data to be written based on the optimal sharding strategy, generating multi-level data sharding and hierarchical verification information with redundancy marks in combination with the adaptive coding parameters, calculating the data distribution plan through a consistent hashing algorithm and a load-aware node selection strategy, sending the multi-level data sharding and the hierarchical verification information to the storage node based on the data distribution plan, and updating the system initial configuration table; The third unit is used to retrieve the target data storage location and encoding parameters in the updated system initial configuration table after receiving a data read request, start multi-channel parallel reading based on the storage location, obtain the multi-level data segmentation and layered verification information in combination with the pipeline mechanism, add the read data to the data format conversion model, extract data features in combination with the graph convolution layer and calculate the optimal conversion path according to the data feature mapping rule library, generate a conversion execution plan and perform format conversion to obtain an initial conversion result, verify the integrity of the initial conversion result through a two-layer spatial attention network, generate an integrity score, and if the integrity score is lower than a preset integrity threshold, construct a data dependency graph through the graph attention network, locate the defect location in combination with the layered verification information and repair it, repeat the reconstruction until the preset maximum number of reconstructions is reached, and obtain the target format data.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images