Distributed data management system and processing method in computer informationization construction

Abstract

The present application relates to computer informationization construction, and proposes a distributed data management system and processing method in computer informationization construction. The system contains data acquisition layer, transmission layer, storage layer and management layer. The intelligent acquisition module of the acquisition layer adaptively adjusts the acquisition strategy; the encryption transmission module of the transmission layer uses a new hybrid encryption algorithm to ensure security, and the dynamic routing module selects the optimal path based on the reinforcement learning algorithm; the intelligent storage scheduling module of the storage layer stores data based on heat; each module of the management layer maintains data consistency, balanced load and data security. The processing method covers the whole process operation of data acquisition, transmission, storage and management. The present application solves the problems of single point failure, poor scalability and difficult data consistency maintenance of traditional data management systems, improves the efficiency, reliability and security of data management, meets the demand of large-scale data processing in multiple fields, and provides strong support for computer informationization construction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer informationization construction, in particular to a distributed data management system and processing method in computer informationization construction. BACKGROUND

[0002] With the rapid development of computer informationization construction, the amount of data is growing explosively, and traditional data management systems face many challenges in coping with large-scale data storage and processing. For example, centralized data management systems have single point failure problems, and once the central server fails, the entire system will be paralyzed. At the same time, its scalability is poor, and it is difficult to meet the growing data processing needs. Although distributed data management systems have been widely used, the existing systems still have deficiencies in data consistency maintenance and other aspects, and cannot fully meet the requirements of efficient and reliable data management in computer informationization construction. SUMMARY

[0003] The purpose of the present application is to provide a distributed data management system and processing method in computer informationization construction to solve the problems raised in the background.

[0004] To achieve the above purpose, the present application provides the following technical scheme: a distributed data management system in computer informationization construction, comprising a data acquisition layer, a data transmission layer, a data storage layer and a data management layer, the data acquisition layer comprising an intelligent acquisition module, the intelligent acquisition module having a self-adaptive acquisition function, and the specific intelligent acquisition module automatically adjusting the acquisition strategy according to the type of data source (such as structured database, unstructured file system);

[0005] The data transmission layer comprises an encryption transmission module and a dynamic routing module, wherein the encryption transmission module adopts a new hybrid encryption algorithm, combining the efficiency of symmetric encryption algorithm and the security of asymmetric encryption algorithm. Before data transmission, the collected data is encrypted using symmetric encryption algorithm to improve the encryption speed, and then the symmetric encryption key is encrypted using the public key of the receiver to ensure the security of key transmission; the dynamic routing module is based on real-time network state monitoring, using reinforcement learning algorithm in machine learning, dynamically selecting the optimal data transmission path according to network bandwidth, delay and packet loss rate indicators, avoiding network congestion and improving data transmission efficiency;

[0006] The data storage layer adopts distributed storage nodes, each node is equipped with an intelligent storage scheduling module, and the intelligent storage scheduling module realizes the data storage strategy based on heat by analyzing the access frequency and timeliness of data;

[0007] The data management layer includes a data consistency maintenance module, a load balancing module and a data security module. The data consistency maintenance module adopts a vector clock consistency algorithm, combines with a distributed transaction processing technology, and ensures the consistency of data in a distributed environment. The vector clock is used to record the version information of data. By comparing the vector clock values, the system can quickly determine the update order of data and avoid data conflicts. The load balancing module collects the load information of each storage node and processing node by using real-time load monitoring technology, and adopts a predictive load balancing algorithm. Through the analysis of historical load data, the time series prediction model is used to predict the load of each node in the future period of time. The tasks are distributed to the nodes with low load in advance to achieve balanced allocation of system load and improve the overall performance of the system. The data security module adopts multi-factor authentication login and data homomorphic encryption technology. When data is stored, homomorphic encryption technology is used so that data can be calculated in an encrypted state. While ensuring data security, it meets the data analysis and processing requirements.

[0008] Preferably, the intelligent collection module automatically adjusts the collection strategy according to the type of data source for structured databases and unstructured file systems. The specific implementation logic is as follows: after the intelligent collection module is started, it first initiates a detection request to the target data source. For structured databases, the intelligent collection module uses a database connection tool to obtain metadata information provided by the database management system. For unstructured data sources, including but not limited to log files and document libraries, the intelligent collection module uses a file system interface to read part of the sample file content. For log files, the intelligent collection module uses regular expression matching and keyword recognition technology to preliminarily determine the log format and the type of key information contained. For document libraries, the intelligent collection module uses the bag-of-words text feature extraction technology to extract and analyze the features of the document content to identify the theme and key content distribution of the document.

[0009] Preferably, the new hybrid encryption algorithm in the encryption transmission module combines the efficiency of symmetric encryption algorithm and the security of asymmetric encryption algorithm. The specific implementation logic of the new hybrid encryption algorithm is as follows:

[0010] Symmetric encryption link: when data is transmitted from the data collection layer to the data transmission layer, the encryption transmission module is started. The encryption transmission module randomly generates a high-strength symmetric encryption key K s . Then, let P be the collected plaintext data. The encryption process follows the formula That is, use the generated symmetric key K s to perform AES encryption on the plaintext P to quickly generate the ciphertext C s .

[0011] Key encryption link: the encryption transmission module obtains the public key K s, and then the symmetric key K p is encrypted by RSA algorithm, where K p = (n, e), n is obtained by multiplying two large prime numbers, e is an integer coprime with (p-1)(q-1), and p and q are two large prime numbers generating n, the encrypted symmetric key C k is obtained by the formula , that is ;

[0012] Data encapsulation and transmission preparation: the encrypted ciphertext C s and the encrypted symmetric key C k are encapsulated by the encryption transmission module to form a complete encrypted data packet.

[0013] Preferably, the implementation logic of the dynamic routing module using reinforcement learning algorithm to dynamically select the optimal data transmission path is as follows:

[0014] Real-time network state monitoring: the dynamic routing module uses the ping command to detect network connectivity and delay, the traceroute command to track the routing path of the data packet, and interacts with the network device to continuously obtain the real-time network bandwidth B, delay D and packet loss rate L key indicators. The above indicators are collected at a preset time interval and sequentially stored in a special state information table to provide data support for subsequent accurate routing decisions;

[0015] Initialization configuration of reinforcement learning algorithm:

[0016] a. State space S definition: the obtained real-time network bandwidth B, delay D and packet loss rate L are divided into multiple different intervals according to the preset, that is, each different interval combination corresponds to a specific state s∈S, realizing the abstraction of real-time network state into a discrete and convenient algorithm processing state set;

[0017] b. Action space A definition: the action space A is the data transmission path for selection. Assuming that there are n different paths available for data transmission in the real-time network environment, then the action space A = {a1, a2, …, an}, where ai represents the specific action of selecting the ith path for data transmission. n i

[0018] ​​c. Construction of reward function R(s,a): The reward function R(s,a) is used to evaluate the quality of the preset action a under the given state s. That is, when the selected path has high bandwidth, low latency, and low packet loss rate, a higher reward value is given to encourage the algorithm to choose such a high-quality path; conversely, when the path performs poorly, a lower reward value is given. R(s,a) = α·B(s,a)―β·D(s,a)―γ·L(s,a), where α, β, and γ are weight coefficients used to flexibly adjust the influence of bandwidth, latency, and packet loss rate on the reward value. B(s,a), D(s,a), and L(s,a) represent the actual bandwidth, latency, and packet loss rate obtained after selecting action a under state s, respectively.

[0019] d. Initialization of Q-table Q(s,a): The Q-table is used to store the expected cumulative reward value for taking each action a in each state s. In the initial stage, the value in the Q-table is set to zero.

[0020] Dynamic routing decision-making and continuous learning process: At each decision time t, the dynamic routing module determines the network state s based on the current real-time monitored network state. t In action space A, an action a is selected using a greedy strategy. t That is, select Q(s) from the current Q-table with probability 1 ∈ . t a) The action with the largest value; select action a. t That is, the data transmission path is determined, and the system immediately transmits data through that path. After the data transmission task is completed, the dynamic routing module determines the data transmission path based on the actual network state changes and the predefined reward function R(s). t ,a t ), calculate the reward value r obtained from this decision. t Then, the values ​​in the Q-table are updated using the Q-learning algorithm, with the update formula as follows: Where η is the learning rate, controlling the step size of each update, δ is the discount factor, used to balance the importance of current and future rewards, and s t+1 It is to perform action a t The new state reached after the network state changes;

[0021] The dynamic routing module continuously repeats the above process of decision-making, transmission, evaluation, and updating, gradually and accurately finding the optimal transmission path under various network conditions.

[0022] Preferably, the specific implementation logic of the intelligent storage scheduling module for implementing the popularity-based data storage strategy is as follows:

[0023] Data receiving and preliminary processing: when the data arrives at the nodes of the data storage layer from the data transmission layer, the intelligent storage scheduling module first performs preliminary processing on the received data, specifically including that the intelligent storage scheduling module analyzes the metadata information of the received data, and the analyzed metadata information includes but is not limited to data type, data size, data source, collection time, through the analysis of the metadata, the intelligent storage scheduling module provides basic data support for subsequent data heat analysis and storage strategy formulation;

[0024] Data heat analysis: the intelligent storage scheduling module updates the access records of the corresponding data in real time according to the analyzed data source; by periodically performing statistical analysis on the access record database, the access times of each data object in a specific time period are calculated;

[0025] Time effectiveness evaluation: the intelligent storage scheduling module evaluates the time effectiveness according to the collection time obtained by data analysis and the business attribute of the data, specifically: in the evaluation based on the collection time, the time effectiveness score formula is S 采集 =I×d n , wherein S 采集 is the time effectiveness score of the collection time, I is the initial time effectiveness score, d is the decay coefficient set according to the business attribute, and n is the period number calculated according to the decay period from the collection time to the current time;

[0026] Comprehensive heat calculation: the intelligent storage scheduling module comprehensively calculates the access frequency and the time effectiveness score to obtain the final heat value of the data, wherein the heat value = access frequency score x weight 1 + time effectiveness score x weight 2, and the weight 1 and the weight 2 are dynamically adjusted according to the business characteristics and performance requirements of the data storage system;

[0027] Data storage decision based on heat: a heat value threshold is set, for data whose heat value is greater than the heat value threshold, that is, hot data, the intelligent storage scheduling module stores it in the high-speed solid state disk; for data whose heat value is less than the heat value threshold, that is, cold data, the intelligent storage scheduling module stores it in the large-capacity mechanical hard disk.

[0028] Preferably, the data consistency maintenance module, after data update in the data storage layer, ensures that the data states on different nodes remain consistent through the vector clock consistency algorithm combined with distributed transaction processing technology, and the specific working logic is:

[0029] Vector clock initialization: initialize the vector clock of the stored data on each node of the data storage layer, set N distributed storage nodes, and represent the vector clock as V = [v1, v2, …, v N ], wherein v irepresents the version count of the data object on the i-th node, initially, for all nodes i, v i = 0, i.e., V = [0, 0, …, 0];

[0030] Data update operation: assuming that an update occurs on the data object on node j, the following steps are performed: node j adds 1 to its corresponding version count in the vector clock, i.e., V[j] = V[j] + 1; then node j broadcasts the update operation and the new vector clock value V to all other nodes;

[0031] Update reception and conflict detection: assuming that node k receives the update message from node j, the following operations are performed:

[0032] Update local vector clock: node k updates the version count of node j in its vector clock to the received value, and adds 1 to its own version count, i.e., V[k] = V[k] + 1, V[j] = max(V[j] 本地 , V[j] 接收 );

[0033] Conflict detection: by comparing the local vector clock V 本地 and the received vector clock V 接收 , it is determined whether there is a data conflict, and the determination rule is as follows: if for all i = 1, 2, …, N, V 本地 [i] ≤ V 接收 [i], it is determined that the received update is the update operation, and there is no conflict; if there is an i such that V 本地 [i] > V 接收 [i], there is a data conflict, and at this time, distributed transaction processing technology needs to be used for coordination;

[0034] Distributed transaction processing coordination: when a data conflict is detected, a two-phase commit protocol is used for coordination, and the specific steps are as follows:

[0035] Preparation phase: assuming that the coordinator, i.e., node j, initiates a request: the coordinator sends a "prepare to commit" message to all participants, i.e., node k, including itself, and the message contains the update operation and the new vector clock value; after each participant receives the "prepare to commit" message, it checks whether it can execute the update operation, and specifically checks whether the data satisfies the preset constraint condition, if it can execute, the update operation is recorded in the local log, and a "ready" message is sent to the coordinator; otherwise, a "not ready" message is sent;

[0036] Commit / Rollback Phase: The coordinator collects response messages from all participants. If all participants send a "ready" message, the coordinator decides to commit the transaction and sends a "commit" message to all participants. If any participant sends a "not ready" message, the coordinator decides to roll back the transaction and sends a "rollback" message to all participants. After receiving the "commit" message, participants execute update operations and record the results in their local logs. After receiving the "rollback" message, participants undo the previously recorded update operations and restore the state to its previous state.

[0037] Data read operation: When performing a data read operation, the node can ensure that it is reading the latest data by comparing the value of the vector clock. The specific rules are as follows: When reading data, the node first obtains the vector clock value V of the data object. 读取 , will node V 读取 With local vector clock V 本地 Comparison: If for all i = 1, 2, ..., N, V 本地 [i]≥V 读取 [i] indicates that the local data is up-to-date and can be read directly. If there exists some i such that V 本地 [i] <V 读取 [i] indicates that the local data is not up-to-date and data needs to be obtained from the node that has the latest data.

[0038] A preferred distributed data management method in computer information infrastructure development includes the following steps:

[0039] Step 1, Data Acquisition Process: The intelligent acquisition module first probes the data source and identifies its type. For structured data sources, it parses the database metadata to determine the data update identifier and executes data acquisition operations based on a pre-set acquisition frequency or a trigger mechanism based on data changes. For unstructured data sources, it uses text analysis tools to preprocess the data and extract key information. The acquired data undergoes preliminary cleaning and format conversion locally before being sent to the data storage layer through the data transmission layer.

[0040] Step 2, Data Transmission Process: The encrypted transmission module encrypts the collected data to generate encrypted data blocks. The dynamic routing module selects the optimal transmission path based on real-time network status information. During transmission, the data is encapsulated and decapsulated through the network protocol stack to ensure reliable data transmission.

[0041] Step 3, data storage process: the intelligent storage scheduling module receives data from the data transmission layer, decides to store the data in the high-speed storage device or the mass storage device according to the heat label of the data, and updates the metadata information of the data during the storage process, including but not limited to data storage location, number of copies, access frequency;

[0042] Step 4, data management process: the data consistency maintenance module regularly checks the vector clock value of the data on each node, and when inconsistent data is found, synchronizes the data through coordination of each node; the load balancing module monitors the load of each node in real time, adjusts the task allocation strategy according to the prediction result, and when a new task request arrives, allocates the task to the node with lower load for processing; the data security module performs a multi-factor identity authentication process when a user accesses the system, and provides corresponding data access services to the authenticated user according to his / her permissions; during data storage and processing, the data is always kept in an encrypted state to ensure data security.

[0043] Compared with the prior art, the beneficial effects of the present application are:

[0044] The present application adopts a distributed architecture, and the data is dispersedly stored in multiple nodes, avoiding the single point failure problem of centralized data management systems. When a node fails, other nodes can still work normally, ensuring continuous operation of the system. Moreover, the distributed architecture naturally has good scalability, and as the data volume and business demand grow, new storage nodes and processing nodes can be easily added to meet the growing data processing demand, overcoming the poor scalability of traditional centralized systems.

[0045] The data consistency maintenance module uses the vector clock consistency algorithm combined with distributed transaction processing technology. The vector clock accurately records data version information and can quickly determine the data update order, combined with distributed transaction processing technology, to ensure that the data state on different nodes remains consistent during data update, reading and other operations. Compared with existing distributed data management systems, data conflicts are effectively avoided, data consistency is fully guaranteed, and the reliability and availability of data are improved.

[0046] The encryption transmission module of the data transmission layer uses a new hybrid encryption algorithm, which first uses a symmetric encryption algorithm to quickly encrypt a large amount of data, and then uses an asymmetric encryption algorithm to ensure the security of key transmission, solving the security problem in the data transmission process. The dynamic routing module dynamically selects the optimal transmission path based on real-time network state monitoring using reinforcement learning algorithm, avoids network congestion, improves data transmission efficiency, optimizes the data transmission process, and makes up for the shortcomings of existing systems in data transmission.

[0047] The intelligent storage scheduling module in the data storage layer analyzes the access frequency and timeliness of data and adopts a hot-based data storage strategy. Hot data is stored on high-speed solid-state drives (SSDs) to improve data access speed, while cold data is stored on large-capacity hard disk drives (HDDs) to reduce storage costs. Simultaneously, metadata information is updated during the storage process to facilitate better data management and improve the efficiency and rationality of data storage. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0049] Figure 2 This is a schematic diagram of the working process of the encrypted transmission module of the present invention;

[0050] Figure 3 This is a schematic diagram of the dynamic routing module workflow of the present invention;

[0051] Figure 4 This is a schematic diagram of the workflow of the data consistency maintenance module of the present invention;

[0052] Figure 5 This is a schematic diagram of the system management method of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1

[0055] Please see Figures 1-4 This invention provides a technical solution: a distributed data management system in computer information construction, including a data acquisition layer, a data transmission layer, a data storage layer and a data management layer. The data acquisition layer includes an intelligent acquisition module, which has an adaptive acquisition function. Specifically, the intelligent acquisition module automatically adjusts the acquisition strategy according to the type of data source (such as structured database, unstructured file system, etc.).

[0056] The intelligent collection module adjusts the collection strategy automatically according to the type of the data source for a structured database and an unstructured file system. The specific implementation logic is as follows: after the intelligent collection module is started, a detection request is first sent to the target data source. For a structured database, the intelligent collection module uses a database connection tool to obtain metadata information provided by the database management system. For example, for a common relational database (such as MySQL or Oracle), the structure of each table in the database is parsed by querying the system table (such as the related tables in the information_schema database of MySQL), including the field name, data type, primary key constraint and other information. At the same time, the module also analyzes the transaction log or related system view of the database to obtain statistical information of the data update frequency. For an unstructured data source, including but not limited to log files and document libraries, the intelligent collection module uses a file system interface to read part of the sample of the file content. For log files, the intelligent collection module uses regular expression matching and keyword recognition technology to preliminarily judge the log format and the type of the key information contained. For document libraries, the intelligent collection module uses the bag-of-words model text feature extraction technology to extract and analyze the features of the document content to identify the theme and key content distribution of the document.

[0057] The data transmission layer includes an encryption transmission module and a dynamic routing module. The encryption transmission module uses a new hybrid encryption algorithm that combines the efficiency of a symmetric encryption algorithm and the security of an asymmetric encryption algorithm. Before data transmission, the collected data is first encrypted using a symmetric encryption algorithm to improve the encryption speed, and then the symmetric encryption key is encrypted using the public key of the receiving party to ensure the security of the key transmission. The dynamic routing module uses a reinforcement learning algorithm in machine learning to dynamically select the optimal data transmission path based on real-time network state monitoring according to network bandwidth, delay and packet loss rate indicators to avoid network congestion and improve data transmission efficiency.

[0058] It should be noted that the new hybrid encryption algorithm in the encryption transmission module combines the efficiency of a symmetric encryption algorithm and the security of an asymmetric encryption algorithm. The specific implementation logic of the new hybrid encryption algorithm is as follows:

[0059] Symmetric encryption link: when data is transmitted from the data collection layer to the data transmission layer, the encryption transmission module is started, and the encryption transmission module randomly generates a high-strength symmetric encryption key K s , and then sets the collected plaintext data P. The encryption process follows the formula That is, the generated symmetric key K s is used to perform AES encryption on the plaintext P to quickly generate the ciphertext C s; wherein due to the advantage of fast calculation speed of the symmetric encryption algorithm in processing a large amount of data, the data can be efficiently encrypted, greatly improving the encryption efficiency and laying a foundation for subsequent data transmission security;

[0060] Key encryption link: the encryption transmission module obtains the public key K of the receiving party s , and then uses the RSA algorithm to encrypt the symmetric key K p , wherein K p =(n, e), n is obtained by multiplying two large prime numbers, e is an integer coprime with (p-1)(q-1), and p and q are two large prime numbers generating n, and the encrypted symmetric key C k is obtained by the formula , that is, ; in this way, even if the encrypted symmetric key C k is unfortunately intercepted during transmission, the attacker cannot decrypt it to obtain the original symmetric key K s due to the lack of the corresponding private key, thereby effectively ensuring the security of key transmission;

[0061] Data encapsulation and transmission preparation: the encryption transmission module encapsulates the encrypted ciphertext C s and the encrypted symmetric key C k to combine into complete encrypted data packets; these encrypted data packets then strictly follow the established network protocol, such as the widely used TCP / IP protocol, and are prepared for transmission in the network.

[0062] In the present application, the efficiency of the symmetric encryption algorithm and the security of the asymmetric encryption algorithm are combined. In the encryption process, the symmetric encryption algorithm is used to quickly encrypt a large amount of data to improve the encryption speed; at the same time, the asymmetric encryption algorithm is used to encrypt the key of the symmetric encryption to ensure the security of the key transmission. This combination can significantly improve the efficiency of encryption and decryption while ensuring the security of data encryption; by using the asymmetric encryption algorithm to encrypt and transmit the symmetric encryption key, the problem of key distribution and management in the symmetric encryption algorithm is solved. The receiving party can use its own private key to decrypt the symmetric encryption key to achieve decryption of the data. This method enhances the security of the key and reduces the risk of key leakage.

[0063] Compared with the traditional single asymmetric encryption algorithm, such as RSA algorithm, the new hybrid encryption algorithm greatly improves the encryption speed when processing a large amount of data, because the symmetric encryption algorithm is used for data encryption. For example, the RSA algorithm has a large amount of calculation and slow speed when encrypting a large amount of data, while the new hybrid encryption algorithm can quickly complete data encryption through the preprocessing of the symmetric encryption algorithm, reducing the encryption time. Compared with the pure symmetric encryption algorithm, such as the AES algorithm, the new hybrid encryption algorithm increases the asymmetric encryption protection of the key. The AES algorithm has certain security risks in the key distribution process. If the key is stolen, the security of the data cannot be guaranteed. The new hybrid encryption algorithm encrypts the key through the asymmetric encryption algorithm for transmission. Even if the key is intercepted during transmission, the attacker cannot obtain the symmetric encryption key, thereby improving the security of the data.

[0064] The implementation logic of the dynamic routing module using the reinforcement learning algorithm to dynamically select the optimal data transmission path is as follows:

[0065] Real-time network state monitoring: the dynamic routing module uses the ping command to detect network connectivity and delay, the traceroute command to track the routing path of the data packet, and interacts with the network device to continuously obtain the real-time network bandwidth B, delay D and packet loss rate L key indicators. The above indicators are collected at a preset time interval and sequentially stored in a special state information table, providing data support for subsequent accurate routing decisions;

[0066] Initialization configuration of the reinforcement learning algorithm:

[0067] a, state space S definition: the obtained real-time network bandwidth B, delay D and packet loss rate L are divided into multiple different intervals according to the preset, that is, each different interval combination corresponds to a specific state s∈S, realizing the abstraction of the real-time network state into a discrete, algorithm-friendly state set. In this way, the complex network state is abstracted into a discrete, algorithm-friendly state set;

[0068] b, action space A definition: the action space A is the data transmission path for selection. Assuming that there are n different paths available for data transmission in the real-time network environment, then the action space A = {a1, a2, …, an}, where a n} represents the specific action of selecting the i-th path for data transmission; i

[0069] ​c. Construction of reward function R(s,a): The reward function R(s,a) is used to evaluate the quality of the preset action a under the given state s. That is, when the selected path has high bandwidth, low latency, and low packet loss rate, a higher reward value is given to encourage the algorithm to choose such a high-quality path; conversely, when the path performs poorly, a lower reward value is given. R(s,a) = α·B(s,a)―β·D(s,a)―γ·L(s,a), where α, β, and γ are weight coefficients used to flexibly adjust the influence of bandwidth, latency, and packet loss rate on the reward value. B(s,a), D(s,a), and L(s,a) represent the actual bandwidth, latency, and packet loss rate obtained after selecting action a under state s, respectively.

[0070] d. Initialization of Q-table Q(s,a): The Q-table is used to store the expected cumulative reward value for each action a in each state s. In the initial stage, the value in the Q-table is set to zero to provide initial basic data for the subsequent learning process.

[0071] Dynamic routing decision-making and continuous learning process: At each decision time t, the dynamic routing module determines the network state s based on the current real-time monitored network state. t In action space A, an action a is selected using a greedy strategy. t That is, select Q(s) from the current Q-table with probability 1 ∈ . t a) The action with the largest value; select action a. t That is, the data transmission path is determined, and the system immediately transmits data through that path. After the data transmission task is completed, the dynamic routing module determines the data transmission path based on the actual network state changes and the predefined reward function R(s). t ,a t ), calculate the reward value r obtained from this decision. t Then, the values ​​in the Q-table are updated using the Q-learning algorithm, with the update formula as follows: Where η is the learning rate, controlling the step size of each update, δ is the discount factor, used to balance the importance of current and future rewards, and s t+1 It is to perform action a t The network state changes and the new state is reached. As time goes on, the dynamic routing module continuously repeats the above process of decision-making, transmission, evaluation, and updating. Through this continuous learning and optimization of the Q-table, the optimal transmission path is gradually and accurately found under various network conditions, effectively avoiding network congestion and significantly improving data transmission efficiency.

[0072] The reinforcement learning algorithm of the present application can monitor the network state in real time, including network bandwidth, delay, packet loss rate and other indicators. By continuously collecting and analyzing these real-time data, the algorithm can timely understand the dynamic changes of the network and provide accurate basis for dynamic routing selection; based on the mechanism of reinforcement learning, the algorithm can automatically learn and adjust the data transmission path according to the changes of the network state. It gradually finds the optimal transmission path by constantly trying different paths and evaluating the pros and cons of the paths according to the results of path selection (such as whether the data is successfully transmitted, transmission efficiency, etc.). This intelligent path selection method can adapt to the dynamic changes of the network environment, avoid network congestion and improve data transmission efficiency;

[0073] Compared with traditional routing algorithms such as static routing algorithm, which usually selects data transmission path according to pre-set rules and cannot real-time perceive the changes of network state, the reinforcement learning algorithm can dynamically adjust the route according to real-time network state, and has stronger adaptability and flexibility. For example, when there is a sudden traffic peak or link failure in the network, the reinforcement learning algorithm can quickly find and switch to other available optimal path, while the static routing algorithm may cause increased data transmission delay or loss; compared with some dynamic routing algorithms based on simple heuristic rules, the reinforcement learning algorithm can more accurately find the optimal path through continuous learning and optimization. For example, some heuristic algorithms may simply choose the path with the largest current bandwidth, but do not consider the stability of the path and the potential congestion risk. While the reinforcement learning algorithm considers multiple network indicators and optimizes path selection through long-term learning, it can more effectively improve data transmission efficiency, reduce transmission delay and packet loss rate.

[0074] During the entire data transmission process, the encryption transmission module ensures the security of the data, and the dynamic routing module improves the transmission efficiency. The two modules cooperate with each other to ensure that the data obtained from the data acquisition layer can be safely and efficiently transmitted to the data storage layer, and effectively support the stable operation of the distributed data management system.

[0075] The dynamic routing module continuously repeats the above decision-making, transmission, evaluation and updating process, and gradually accurately finds the optimal transmission path under various network states;

[0076] The data storage layer is composed of distributed storage nodes, each node is equipped with an intelligent storage scheduling module, and the intelligent storage scheduling module realizes the data storage strategy based on heat by analyzing the access frequency and timeliness of the data;

[0077] The specific implementation logic of the data storage strategy based on heat is as follows:

[0078] Data receiving and preliminary processing: when the data arrives at the nodes of the data storage layer from the data transmission layer, the intelligent storage scheduling module first performs preliminary processing on the received data, specifically including the intelligent storage scheduling module analyzing the metadata information of the received data. The obtained metadata information includes but is not limited to data type, data size, data source, collection time, etc. Through the analysis of the metadata, the intelligent storage scheduling module provides basic data support for subsequent data heat analysis and storage strategy formulation;

[0079] Data heat analysis: the intelligent storage scheduling module updates the access records of the corresponding data in real time according to the data source obtained by analysis. For example, it is known that a batch of data comes from the customer order table of the sales system. The module can quickly filter out all access records of the table data in the access record database, so as to accurately calculate the access frequency of the data in a specific time period and provide accurate data basis for access frequency statistics. Through regular statistical analysis of the access record database, the access frequency of each data object in a specific time period is calculated;

[0080] Time effectiveness evaluation: the intelligent storage scheduling module evaluates the time effectiveness according to the collection time obtained by data analysis and the business attribute of the data, by using a time effectiveness evaluation model. Specifically, in the evaluation based on the collection time, the time effectiveness score formula is S 采集 =I×d n , where S 采集 is the time effectiveness score of the collection time, I is the initial time effectiveness score, d is the decay coefficient set according to the business attribute, and n is the number of periods calculated according to the decay period from the collection time to the current time. For example, for real-time transaction data, the time effectiveness requirement is extremely high, and the value of the data decreases rapidly with time. For some historical archive data, the time effectiveness is relatively low. The module will set different time effectiveness evaluation models for different types of data. For example, for real-time transaction data, through the set time decay function, the time effectiveness score of the data decreases with the distance from the current time. Assuming that the current time is taken as the reference, the time effectiveness score decreases by a certain percentage (such as a decay coefficient of 0.9) every 1 hour. In this way, a time effectiveness score is calculated for each data.

[0081] Comprehensive heat calculation: the intelligent storage scheduling module comprehensively calculates the access frequency and the time effectiveness score to obtain the final heat value of the data, where the heat value = access frequency score x weight 1 + time effectiveness score x weight 2, and the weights 1 and 2 are dynamically adjusted according to the business characteristics and performance requirements of the data storage system;

[0082] The data storage decision based on the heat degree is as follows: a heat degree value threshold is set, and the intelligent storage scheduling module stores data with a heat degree value greater than the heat degree value threshold, i.e. hot data, in a high-speed solid state disk; the intelligent storage scheduling module stores data with a heat degree value less than the heat degree value threshold, i.e. cold data, in a large-capacity mechanical hard disk. For example, for hot commodity information data of an online e-commerce platform, the data belongs to the hot data category due to frequent user inquiries and browsing, and the intelligent storage scheduling module stores such data in an SSD device, so that the user can quickly obtain data when inquiring about commodity information, and the user experience is improved. For some historical data, such as transaction records in the past few years, although the access frequency is low, the data still needs to be stored due to the requirement of data backup and occasional inquiries. The intelligent storage scheduling module stores such cold data in a mechanical hard disk, thereby meeting the data storage requirement and effectively reducing the storage cost.

[0083] The data management layer includes a data consistency maintenance module, a load balancing module and a data security module. The data consistency maintenance module adopts a vector clock consistency algorithm combined with a distributed transaction processing technology to ensure the consistency of data in a distributed environment. The vector clock is used to record the version information of data, and the system can quickly determine the update order of data by comparing the vector clock values to avoid data conflicts. The load balancing module collects the load information of each storage node and processing node by using real-time load monitoring technology, and adopts a predictive load balancing algorithm. The algorithm analyzes the historical load data and predicts the load of each node in the future period of time by using a time series prediction model, so as to distribute tasks to nodes with low load in advance, realize balanced distribution of system load, and improve the overall performance of the system. The data security module adopts multi-factor identity authentication login and data homomorphic encryption technology. Multi-factor identity authentication combines user password, fingerprint recognition and dynamic verification code and other authentication methods to ensure that only legitimate users can access the system. The data is encrypted when stored, and the data can still be calculated in the encrypted state, thereby meeting the data analysis and processing requirements while ensuring data security. Multi-factor identity authentication and homomorphic encryption technology are existing mature technologies, and therefore will not be described here.

[0084] It should be noted that the load balancing module uses real-time load monitoring technology to analyze data by using a load balancing algorithm, which is an existing technology. The working steps are briefly described as follows:

[0085] Load information collection: real-time load monitoring technology is used to collect the load information of each storage node and processing node at regular intervals (e.g. every 1 minute). For storage nodes, the disk I / O usage rate, remaining storage space, etc. are monitored; for processing nodes, the CPU usage rate, memory usage rate, task queue length, etc. are monitored.

[0086] Historical load data processing: The collected load information is stored in a historical load database according to time series for easy subsequent analysis. For example, the data for each hour constitutes a data point, covering the average of different load indicators for each node within that hour, forming a time series dataset;

[0087] Application of Time Series Forecasting Models: Time series forecasting models such as the Autoregressive Integrated Moving Average (ARIMA) model are used to predict the load of each node over a future period. Taking the prediction of node A's CPU utilization for the next hour as an example, the ARIMA model analyzes historical CPU utilization data sequences, identifying trends, seasonality, and periodicity to build the forecasting model. After training, inputting CPU utilization data from a period prior to the current time (e.g., the past 12 hours) yields the predicted value.

[0088] Task allocation decision: Based on the predicted load of each node, tasks are pre-assigned to nodes with lower loads. For example, if a new task request arrives and node A is predicted to have a 70% load while node B is predicted to have a 40% load, the task will be assigned to node B, achieving a balanced distribution of system load. Simultaneously, the module monitors task execution in real time, and if the actual load deviates significantly from the prediction, it adjusts the allocation strategy for subsequent tasks accordingly.

[0089] Specifically, for the data consistency maintenance module, after data is updated in the data storage layer, it uses a vector clock consistency algorithm combined with distributed transaction processing technology to ensure that the data state on different nodes remains consistent. The specific working logic is as follows:

[0090] Vector clock initialization: On each node of the data storage layer, the stored data is initialized with a vector clock. Given N distributed storage nodes, denoted by V = [v1, v2, ..., v...], ... N ] represents a vector clock, where v i This represents the version count of the data object on the i-th node. Initially, for all nodes i, v i =0, that is, V = [0,0,…,0];

[0091] Data update operation: Assuming that a data object on node j is updated, the following steps are performed: Node j increments its version count in the vector clock by 1, i.e., V[j] = V[j] + 1; then node j broadcasts the update operation and the new vector clock value V to all other nodes;

[0092] Update reception and collision detection: Suppose that after node k receives the update message from node j, it performs the following operations:

[0093] Update local vector clock: node k updates the version count of node j in its vector clock to the received value, while adding 1 to its own version count, i.e. V[k] = V[k] + 1, V[j] = max(V[j] 本地 , V[j] 接收 );

[0094] Conflict detection: by comparing the local vector clock V 本地 and the received vector clock V 接收 , it is determined whether there is a data conflict, and the determination rule is as follows: if V 本地 [i]≤V 接收 [i] for all i = 1, 2, …, N, it means that the received update is an update operation, and there is no conflict; if there is an i such that V 本地 [i]>V 接收 [i], there is a data conflict, and at this time, it needs to be coordinated by combining the distributed transaction processing technology;

[0095] Distributed transaction processing coordination: when a data conflict is detected, a two-phase commit protocol is used for coordination, and the specific steps are as follows:

[0096] Preparation phase: assuming that the coordinator, i.e. node j, initiates a request: the coordinator sends a "prepare to commit" message to all participants, i.e. nodes k, including itself, and the message contains an update operation and a new vector clock value; after each participant receives the "prepare to commit" message, it checks whether it can execute the update operation, and specifically checks whether the data meets the preset constraint condition, and if it can execute, it records the update operation in the local log and sends a "ready" message to the coordinator; otherwise, a "not ready" message is sent;

[0097] Commit / rollback phase: the coordinator collects the response messages of all participants, and if all participants send a "ready" message, the coordinator decides to commit the transaction and sends a "commit" message to all participants; if any participant sends a "not ready" message, the coordinator decides to roll back the transaction and sends a "rollback" message to all participants; after the participant receives the "commit" message, it executes the update operation and records the operation result in the local log; after the participant receives the "rollback" message, it cancels the previously recorded update operation and restores to the state before the update;

[0098] Data read operation: when performing a data read operation, the node can ensure that the latest data is read by comparing the values of the vector clocks, and the specific rules are as follows: when the node reads data, it first obtains the vector clock value V 读取 of the data object, compares the node V 读取 with the local vector clock V 本地For comparison, if for all i = 1, 2, …, N, V 本地 [i] ≥ V 读取 [i], it is indicated that the local data is the latest and can be directly read, if there is some i, such that V 本地 [i] < V 读取 [i], it is indicated that the local data is not the latest and needs to obtain data from a node that has the latest data.

[0099] The application combines the vector clock consistency algorithm with the distributed transaction processing technology. The vector clock can accurately record data version information and quickly determine the update order. On this basis, combined with distributed transaction processing, it ensures that in a distributed environment, whether it is data update, reading or other operations, it can maintain high consistency between multiple nodes, avoiding data conflicts. This combination method more comprehensively guarantees data consistency. Further, through the vector clock to record version information, the system can quickly determine the update order of the data. For example, in a distributed database, when multiple nodes simultaneously operate on the same data, the traditional algorithm may need a complex comparison and coordination process, while the application can quickly determine which operation is the latest and which needs to be adjusted, greatly improving the efficiency of conflict detection and resolution.

[0100] Embodiment 2

[0101] As Figure 5 , the distributed data management method in computer informationization construction comprises the following steps:

[0102] Step 1, data acquisition process: the intelligent acquisition module first detects the data source and identifies the data source type. For structured data sources, parse the database metadata, determine the data update identifier, and perform data acquisition operations according to the pre-set acquisition frequency or based on the data change trigger mechanism. For unstructured data sources, use text analysis tools to preprocess the data and extract key information; the collected data is preliminarily cleaned and format-converted locally, and then sent to the data storage layer through the data transmission layer;

[0103] Step 2, data transmission process: the encrypted transmission module encrypts the collected data to generate encrypted data blocks, and the dynamic routing module selects the optimal transmission path according to real-time network state information. During data transmission, the data is encapsulated and decapsulated through the network protocol stack to ensure reliable data transmission;

[0104] Step 3, data storage process: the intelligent storage scheduling module receives data from the data transmission layer, decides to store the data in the high-speed storage device or the mass storage device according to the heat label of the data, and updates the metadata information of the data during the storage process, including but not limited to data storage location, number of copies, access frequency;

[0105] Step 4, data management process: the data consistency maintenance module regularly checks the vector clock value of the data on each node, and when inconsistent data is found, synchronizes the data through coordination of each node; the load balancing module monitors the load of each node in real time, adjusts the task allocation strategy according to the prediction result, and when a new task request arrives, allocates the task to the node with lower load for processing; the data security module performs multi-factor identity authentication process when the user accesses the system, and provides corresponding data access service according to the user's authority; during data storage and processing, the data is always kept in an encrypted state to ensure data security.

[0106] The present application focuses on the field of computer informationization construction, and aims at the pain points of traditional data management system, and builds an innovative distributed data management system and processing method, which covers the optimization of the whole process of data collection, transmission, storage and management.

[0107] System architecture and core modules: the system adopts a layered architecture, and works collaboratively by data collection layer, data transmission layer, data storage layer and data management layer. The intelligent collection module of the data collection layer can automatically adjust the collection strategy according to the data source type (structured database, unstructured file system), accurately collect data for different data sources, and reduce transmission overhead. The encryption transmission module of the data transmission layer uses a new hybrid encryption algorithm, which combines the efficiency of symmetric encryption and the security of asymmetric encryption, to ensure the safety of data transmission; the dynamic routing module dynamically selects the optimal transmission path based on real-time network state and reinforcement learning algorithm, to improve the transmission efficiency. The intelligent storage scheduling module of the data storage layer analyzes the data access frequency and timeliness, implements the data storage strategy based on heat, reasonably allocates storage resources, reduces cost and improves access speed. The data management layer includes data consistency maintenance module, load balancing module and data security module. The data consistency maintenance module uses vector clock and distributed transaction processing technology to ensure the data consistency in distributed environment; the load balancing module uses real-time load monitoring and prediction algorithm to balance the system load and improve the overall performance; the data security module uses multi-factor identity authentication and homomorphic encryption technology to ensure user access security and data processing security.

[0108] Processing method and flow optimization: The processing method covers the complete flow of data collection, transmission, storage and management. During data collection, the intelligent collection module identifies the data source type, parses the metadata of structured data sources, extracts key information from unstructured data sources, and after data collection, the data is cleaned and format-converted before transmission. During data transmission, the encryption transmission module encrypts the data, the dynamic routing module selects the optimal path, and the data is reliably transmitted under the support of the network protocol stack. In the data storage stage, the intelligent storage scheduling module stores data according to the heat label and updates the metadata information. In the data management process, the data consistency maintenance module checks the vector clock value and synchronizes the data, the load balancing module monitors the load and allocates tasks, and the data security module performs multi-factor identity authentication and ensures data encryption processing.

[0109] Innovation advantage and application value: The present application realizes innovation breakthrough in multiple aspects. The new hybrid encryption algorithm, the dynamic routing algorithm based on reinforcement learning, the data storage strategy based on heat, and the vector clock combined with distributed transaction processing, effectively solve the problems of single point failure, poor scalability, and difficult data consistency maintenance of traditional systems. In practical application, the efficiency, reliability and security of data management are improved, which can meet the needs of large-scale data processing and management in multiple fields such as finance, e-commerce and scientific research, providing strong technical support for computer informationization construction and promoting the digital development process of related industries.

[0110] Although embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A distributed data management system in computer informationization construction, comprising a data acquisition layer, a data transmission layer, a data storage layer and a data management layer, characterized in that: The data acquisition layer includes an intelligent acquisition module, which uses an adaptive acquisition function to automatically adjust the acquisition strategy. The data transmission layer includes an encrypted transmission module and a dynamic routing module. The encrypted transmission module uses a novel hybrid encryption algorithm to ensure the security of key transmission. The dynamic routing module dynamically selects the optimal data transmission path based on real-time network status monitoring and reinforcement learning algorithms in machine learning. The data storage layer consists of distributed storage nodes, each equipped with an intelligent storage scheduling module. This module analyzes the access frequency and timeliness of data to implement a popularity-based data storage strategy. The specific implementation logic of this popularity-based data storage strategy is as follows: Data reception and preliminary processing: When data arrives at each node of the data storage layer from the data transmission layer, the intelligent storage scheduling module first performs preliminary processing on the received data. Specifically, the intelligent storage scheduling module parses the metadata information of the received data. The parsed metadata information includes, but is not limited to, data type, data size, data source, and collection time. Through the parsing of metadata, the intelligent storage scheduling module provides basic data support for subsequent data heat analysis and storage strategy formulation. Data popularity analysis: The intelligent storage scheduling module updates the access records of the corresponding data in real time based on the parsed data source; by periodically performing statistical analysis on the access record database, it calculates the number of times each data object is accessed within a specific time period; Timeliness evaluation: the intelligent storage scheduling module evaluates the timeliness according to the collection time obtained by data analysis and the business attribute of the data by a timeliness evaluation model. Specifically, in the evaluation based on the collection time, the timeliness score formula is: wherein is the timeliness score of the collection time, is the initial timeliness score, is the decay coefficient set according to the business attribute, is the number of periods calculated according to the decay period from the collection time to the current time. Comprehensive popularity calculation: The intelligent storage scheduling module comprehensively calculates the access frequency and timeliness score to obtain the final popularity value of the data, where the popularity value = access frequency score × weight 1 + timeliness score × weight 2. Weight 1 and weight 2 are dynamically adjusted according to the business characteristics and performance requirements of the data storage system. Data storage decisions based on popularity: A popularity threshold is set. For data with a popularity value greater than the threshold (i.e., hot data), the intelligent storage scheduling module stores it on a high-speed solid-state drive. For data with a popularity value less than the threshold (i.e., cold data), the intelligent storage scheduling module stores it on a large-capacity hard disk drive. The data management layer includes a data consistency maintenance module, a load balancing module, and a data security module. The data consistency maintenance module adopts a vector clock consistency algorithm combined with distributed transaction processing technology to ensure data consistency in a distributed environment. The load balancing module utilizes real-time load monitoring technology to collect load information from each storage and processing node. It employs a predictive load balancing algorithm to pre-allocate tasks to nodes with lower loads, achieving balanced distribution of system load. The data security module uses multi-factor authentication login and homomorphic encryption technology. When storing data, homomorphic encryption is used, allowing computation to continue even when the data is encrypted. This ensures data security while meeting the needs of data analysis and processing.

2. The distributed data management system in computer informatization construction of claim 1, wherein: The intelligent data acquisition module automatically adjusts its acquisition strategy based on the type of data source, whether it's a structured database or an unstructured file system. Specifically, after startup, the module initiates a probe request to the target data source. For structured databases, it uses database connection tools to obtain metadata information provided by the database management system. For unstructured data sources, including but not limited to log files and document libraries, the module uses a file system interface to read partial samples of the file content. For log files, the module uses regular expression matching and keyword recognition technology to initially determine the log format and the types of key information contained within. For document libraries, the module uses bag-of-words model text feature extraction technology to extract and analyze features from the document content to identify the document's theme and key content distribution.

3. The distributed data management system in computer informatization construction of claim 1, wherein: The novel hybrid encryption algorithm in the encrypted transmission module combines the efficiency of symmetric encryption algorithms with the security of asymmetric encryption algorithms. The specific implementation logic of the novel hybrid encryption algorithm is as follows: Symmetric encryption link: when the data is transmitted from the data acquisition layer to the data transmission layer, the encryption transmission module is started, and the encryption transmission module randomly generates a high-strength symmetric encryption key , and then the collected plaintext data is encrypted The encryption process follows the formula , that is, the generated symmetric key is used to encrypt the plaintext The encryption operation quickly generates ciphertext ; Key encryption step: The encrypted transmission module obtains the recipient's public key. Then, the RSA algorithm was used to process the symmetric key. Encryption is performed, among which , It is obtained by multiplying two large prime numbers. Is with Coprime integers, and and To generate Two large prime numbers, and the encrypted symmetric key. Through formula ,Right now It can be concluded that; Data encapsulation and transmission preparation: The encryption transmission module encapsulates the ciphertext. and the encrypted symmetric key They are then encapsulated and combined into complete encrypted data packets.

4. The distributed data management system in computer information construction according to claim 1, characterized in that: The implementation logic of the dynamic routing module, which uses reinforcement learning algorithms to dynamically select the optimal data transmission path, is as follows: Real-time network status monitoring: The dynamic routing module uses the ping command to detect network connectivity and latency, the traceroute command to trace the routing path of data packets, and interacts with network devices to continuously obtain real-time network bandwidth. ,Delay and packet loss rate The key metrics mentioned above are collected at preset time intervals and stored in an orderly manner in a dedicated status information table to provide data support for subsequent accurate routing decisions. Initialization configuration of reinforcement learning algorithm: a. State space Definition: The real-time network bandwidth to be acquired ,Delay and packet loss rate The system is divided into multiple different intervals based on a preset structure, meaning that each different combination of intervals corresponds to a specific state. This allows the real-time network state to be abstracted into a discrete set of states that are easy for algorithms to process; b. Action Space Definition: Action Space Assuming that there are data transmission paths to choose from in a real-time network environment... Different paths can be used for data transmission, so the action space... ,in Representative selects the first The specific actions for data transmission along each path; c. Reward Function Construction: Through the reward function To evaluate the setting status Preset actions The quality of the selected path is determined by its performance. A higher reward is given when the chosen path has high bandwidth, low latency, and low packet loss rate, encouraging the algorithm to choose such high-quality paths; conversely, a lower reward is given when the path performs poorly. , and These are weighting coefficients used to flexibly adjust the impact of bandwidth, latency, and packet loss rate on the reward value. , and These represent the states respectively. Select action The actual bandwidth, latency, and packet loss rate obtained afterwards; d, Q-table Initialization: Q-tables are used to store data for each state. Take each action below The expected cumulative reward value is set to zero in the Q-table during the initial phase; Dynamic routing decision-making and continuous learning process: at each decision-making moment The dynamic routing module determines the network status based on real-time monitoring. Action space China adopts Greedy strategy selects one action That is, based on probability Select the current Q-table The action with the highest value; the selected action. That is, once the data transmission path is determined, the system immediately transmits data through that path. After the data transmission task is completed, the dynamic routing module adjusts the routing based on actual network state changes and a predefined reward function. Calculate the reward value obtained from this decision. Then, the values ​​in the Q-table are updated using the Q-learning algorithm, with the update formula as follows: ,in The learning rate controls the step size for each update. It is a discount factor used to balance the importance of current rewards and future rewards. It is to perform an action The new state reached after the network state changes; The dynamic routing module continuously repeats the above process of decision-making, transmission, evaluation, and updating, gradually and accurately finding the optimal transmission path under various network conditions.

5. The distributed data management system in computer information construction according to claim 1, characterized in that: After data is updated in the data storage layer, the data consistency maintenance module ensures that the data state on different nodes remains consistent through a vector clock consistency algorithm combined with distributed transaction processing technology. The specific working logic is as follows: Vector clock initialization: At each node of the data storage layer, the vector clock is initialized for the stored data. A distributed storage node, used Denotes a vector clock, where Indicates the first The version count of the data object on each node, initially, for all nodes... ,Right now ; Data update operation: Assuming node When a data object is updated, the following steps are performed: Node Increment its corresponding version count in the vector clock by 1, that is ; Subsequent nodes The update operation and the new vector clock value will be implemented. Broadcast to all other nodes; Update reception and collision detection: Set up a node Received node After receiving the update message, perform the following operations: Update local vector clock: node The corresponding node in its vector clock The version count is updated to the received value, and its own version count is incremented by 1. ; Collision detection: by comparing local vector clocks and received vector clock To determine if there are data conflicts, the following rules apply: If for all They all If the received update is a valid update operation, then there is no conflict; if there is a conflict... , making If this happens, data conflicts will occur, and in this case, it is necessary to use distributed transaction processing technology for coordination. Distributed transaction processing coordination: When data conflicts are detected, a two-phase commit protocol is used for coordination. The specific steps are as follows: Preparation phase: Assume a coordinator, i.e., a node. Initiating a request: The coordinator sends a request to all participants, i.e., nodes. This includes sending a "ready to commit" message, which contains the update operation and the new vector clock value; after receiving the "ready to commit" message, each participant checks whether it can execute the update operation, specifically whether the data meets the preset constraints. If it can be executed, the update operation is recorded in the local log and a "ready" message is sent to the coordinator; otherwise, a "not ready" message is sent. Commit / Rollback Phase: The coordinator collects response messages from all participants. If all participants send a "ready" message, the coordinator decides to commit the transaction and sends a "commit" message to all participants. If any participant sends a "not ready" message, the coordinator decides to roll back the transaction and sends a "rollback" message to all participants. After receiving the "commit" message, participants execute update operations and record the results in their local logs. After receiving the "rollback" message, participants undo the previously recorded update operations and restore the state to its previous state. Data read operation: When performing a data read operation, the node can ensure that it is reading the latest data by comparing the value of the vector clock. The specific rules are as follows: When reading data, the node first obtains the vector clock value of the data object. , will node With local vector clock Compare, if for all They all This indicates that the local data is up-to-date and can be read directly. If a certain... , making If the local data is not up-to-date, then data needs to be obtained from the node that has the latest data.

6. The method for implementing a distributed data management system in computer information construction according to any one of claims 1-5, characterized in that, Includes the following steps: Step 1, Data Acquisition Process: The intelligent acquisition module first detects the data source and identifies the data source type; For structured data sources, the database metadata is parsed to identify data update indicators. Data collection operations are performed according to the pre-set collection frequency or the trigger mechanism based on data changes. For unstructured data sources, text analysis tools are used to preprocess the data and extract key information. The collected data is initially cleaned and format converted locally before being sent to the data storage layer through the data transmission layer. Step 2, Data Transmission Process: The encrypted transmission module encrypts the collected data to generate encrypted data blocks. The dynamic routing module selects the optimal transmission path based on real-time network status information. During transmission, the data is encapsulated and decapsulated through the network protocol stack to ensure reliable data transmission. Step 3, Data storage process: The intelligent storage scheduling module receives data from the data transmission layer and decides whether to store the data in a high-speed storage device or a large-capacity storage device based on the data's popularity tags. During the storage process, the module updates the data's metadata information, including but not limited to the data storage location, number of copies, and access frequency. Step 4, Data Management Process: The data consistency maintenance module periodically checks the vector clock values ​​of the data on each node. When inconsistent data is found, it coordinates the nodes to perform data synchronization operations. The load balancing module monitors the load of each node in real time and adjusts the task allocation strategy based on the prediction results. When a new task request arrives, the task is assigned to a node with a lower load for processing. The data security module executes a multi-factor authentication process when users access the system. For authenticated users, the module provides corresponding data access services according to their permissions. During data storage and processing, the data is always kept encrypted to ensure data security.

Images