Centralized filing processing system and method for electric power engineering data based on cloud storage

Abstract

The invention discloses an electric power engineering data centralized filing processing system and method based on cloud storage. The method comprises the following steps: carrying out digital formatting processing on all electric power engineering data through a scanner; uploading the digitized electric power engineering data to a cloud storage platform based on a distributed storage technology; analyzing relevance among the data to obtain a relevance matrix; constructing a hash space of distributed storage based on a hash ring; based on the obtained relevance matrix, data are stored in a hash ring in a scattered mode through a distributed hash algorithm according to relevance; and carrying out dynamic archiving management on the electric power engineering data based on load balancing storage. The method has the advantages that high-relevance data are stored in a scattered mode by analyzing data relevance, data safety is improved, efficient scattered storage and dynamic management of electric power engineering data are achieved through the distributed hash algorithm and the load balancing technology, and data storage reliability and system operation efficiency are improved.

Description

Technical Field

[0001] The present invention relates to data processing load balancing, and in particular to a centralized archiving processing system and method for electric power engineering data based on cloud storage. Background Art

[0002] A centralized archiving and processing system and method for electric power engineering data based on cloud storage, which uploads digital electric power engineering data to a cloud storage platform, uses correlation analysis technology to obtain the union of random data packets, and generates a correlation matrix by the proportion of the union to the whole set. Subsequently, a hash space for distributed storage is constructed based on a hash ring, and the data data is dispersedly stored according to the correlation using a distributed hash algorithm. By introducing a load balancing mechanism, dynamic and intelligent archiving management of electric power engineering data is achieved. This method not only improves the flexibility and reliability of data storage, but also effectively optimizes storage resource allocation and access performance, providing technical support for the efficient management of electric power engineering data.

[0003] The centralized archiving and processing systems and methods for power engineering data currently on the market mainly rely on cloud computing, distributed storage and big data technologies to achieve efficient management of massive engineering data. Such systems usually collect and classify the design drawings, construction records, equipment information, maintenance data, etc. generated in power engineering projects. The system supports the rapid entry, encrypted storage and intelligent retrieval of files in various formats to ensure the security and traceability of data. At the same time, the integrated archiving method also supports permission management and collaboration functions, and can provide hierarchical access rights according to different user needs to improve work efficiency. In addition, by introducing artificial intelligence technology, the system can achieve in-depth analysis of historical data to provide support for engineering optimization and decision-making. Compared with the traditional decentralized management method, this method significantly improves the integrity, query efficiency and archiving quality of the data, and is an important part of the digital transformation of power engineering. Summary of the invention

[0004] In order to improve the existing centralized archiving and processing system and method for electric power engineering data, a centralized archiving and processing system and method for electric power engineering data based on cloud storage is provided. The method realizes efficient management of data through cloud computing and distributed storage technology, integrates data uploading, correlation analysis, intelligent storage and dynamic optimization functions, and improves archiving efficiency and security.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A centralized archiving and processing method for electric power engineering data based on cloud storage, comprising:

[0007] All power engineering data are digitally formatted through scanners;

[0008] Upload the digitized power engineering data to the cloud storage platform based on distributed storage technology;

[0009] Based on the acquired power engineering data, analyze the correlation between the data and obtain the correlation matrix;

[0010] Based on the data management requirements of power engineering materials, a hash space with distributed storage based on hash ring is constructed;

[0011] Based on the obtained correlation matrix, the data is dispersedly stored in the hash ring according to the correlation through the distributed hash algorithm;

[0012] Dynamic archiving management of power engineering data based on load balancing storage.

[0013] Preferably, the digital formatting of all power engineering data by means of a scanner specifically includes:

[0014] Obtain electronic image data of all power engineering materials through scanners;

[0015] Based on the image data, digital image processing is performed to obtain the content data of the power engineering materials;

[0016] Based on the content data of power engineering materials, basic information data of each material is obtained.

[0017] Preferably, the step of analyzing the correlation between the data based on the acquired power engineering data to obtain the correlation matrix specifically includes:

[0018] Based on the data in the power engineering data of each project, obtain all data packages of the data;

[0019] Based on the data contained in the data packages of all projects, the percentage of the combined data of each data package in the total project data is analyzed to obtain the correlation matrix of the data packages.

[0020] Preferably, the step of analyzing the percentage of the combined data of each data package in the total data of the project based on the data contained in the data packages of all the projects to obtain the correlation matrix of the data packages specifically includes:

[0021] Based on the data contained in the data packets of all the acquired items, a data packet set P is acquired = {P 1 ,P 2 ,P 3 ,...,P n}, each data packet P i Contains data subset D i ={d 1 ,d 2 ,d 3 ,...,d k}, the global data set is

[0022] For each data item d in the data set D k Mapped to index k, and represent each packet P with a binary vector i , the formula is:

[0023]

[0024] Based on two random data packets P from all projects i and P j , calculate and obtain the union D of two data packets i ∪D j , the formula is:

[0025] v i ∨v j =[max(x i1 ,x j1 ),max(x i2 ,x j2 ),...,max(x im ,x jm )]

[0026] Based on the obtained union, calculate its size, the formula is:

[0027]

[0028] Based on the size of the obtained union, calculate the percentage of the obtained union in the entire set. The formula is:

[0029]

[0030] Based on calculating the union ratio data between all data packets, the correlation matrix R is constructed. The matrix is:

[0031]

[0032] Preferably, the construction of a hash space for distributed storage based on a hash ring based on the power engineering data management requirements specifically includes:

[0033] Construct a hash ring of length N and determine the hash value range to be [0, 2 N -1];

[0034] Add a unique identifier to each node;

[0035] Based on the hash function H(x), the data packet P i Mapped onto the ring.

[0036] Preferably, based on the acquired correlation matrix, dispersively storing the data packets in the hash ring according to the correlation by using a distributed hash algorithm specifically includes:

[0037] Based on the correlation matrix, data packets with high correlation are placed at both ends of a diameter of the hash ring, and data packets with low correlation are randomly and evenly distributed on the hash ring.

[0038] Preferably, placing data packets with high correlation at both ends of a diameter of a hash ring based on the correlation matrix and randomly and evenly distributing data packets with low correlation on the hash ring specifically includes:

[0039] If P i and P j is a group of highly correlated data packets placed at both ends of the diameter, with the formula:

[0040]

[0041] Where N is the total length of the hash ring, H(P i ) is always in H(P j ) on the opposite side;

[0042] The data packets with low correlation are randomly placed in the free positions on the ring. The formula is:

[0043] H(P k )=rand()mod N

[0044] If the position H(P j ) is already occupied, a virtual node is created and the data packet is mapped to the virtual node. The formula is:

[0045] H'(P j )=(H(P j )+k·Δ)mod N

[0046] Where k is the virtual machine node number and Δ is the offset of the virtual node.

[0047] Preferably, the dynamic archiving management of power engineering data based on load balancing storage specifically includes:

[0048] Calculate the number of packets at each position on the hash ring and determine whether the distribution is uniform by the ratio of the maximum load to the minimum load. The formula is:

[0049]

[0050] Based on the polling method, part of the data in the high-load area is migrated to other nodes, and the storage location of the data is determined according to the real-time load situation of each node.

[0051] Furthermore, a centralized archiving and processing system for power engineering data based on cloud storage is proposed, comprising:

[0052] Scanning module: The scanning module is mainly used to scan the power engineering data to obtain clear image data of the power engineering data, which is convenient for subsequent analysis and classification operations;

[0053] A cloud storage module, which is mainly used to store power engineering data on a remote server through cloud computing technology, and can access, manage, share and back up the power engineering data through the network anytime and anywhere;

[0054] A correlation module, wherein the correlation module is mainly used to calculate the correlation between the data packets by comparing the data in the data packets;

[0055] Construction module: The construction module is mainly used to construct a hash space of distributed storage based on a hash ring;

[0056] A storage processing module, wherein the storage processing module is mainly used to disperse and store the associated data packets on a hash ring;

[0057] A load processing module, which is mainly used to optimize the use of storage resources to avoid performance degradation or service unavailability due to overload of a single storage node;

[0058] A cloud storage and a processor, wherein the storage and the processor are connected to each other via a network, and the processor executes the above method by executing computer instructions.

[0059] Compared with the prior art, the advantages of the present invention are:

[0060] By uploading the digitized data to the cloud storage platform, efficient centralized management of the data is achieved, and the storage and retrieval efficiency is improved. Secondly, using the correlation matrix to analyze the relationship between the data helps to discover the inherent correlation of the data and optimize the data organization structure. Based on the distributed hash algorithm, the highly correlated data is stored in the hash ring in a dispersed manner, which not only improves the reliability of storage, but also effectively avoids single point failures and improves the disaster recovery capability of the system. In addition, combined with the load balancing mechanism, dynamic archiving and balanced distribution of data are achieved, which avoids overload of storage nodes and further ensures the efficient operation of the system. This method makes full use of the technical advantages of cloud computing and distributed storage, and provides strong support for the intelligent management and long-term storage of power engineering data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a schematic diagram of the data archiving processing method proposed by the present invention;

[0062] Figure 2This is a schematic diagram of the digital formatting process proposed by the present invention;

[0063] Figure 3 This is a schematic diagram of obtaining the relevance proposed by the present invention;

[0064] Figure 4 This is a schematic diagram of the correlation matrix calculation proposed by the present invention;

[0065] Figure 5 A schematic diagram of the hash ring construction proposed by the present invention;

[0066] Figure 6 A schematic diagram of data storage proposed by the present invention;

[0067] Figure 7 A schematic diagram of storing high and low correlation data proposed by the present invention;

[0068] Figure 8 This is a schematic diagram of the load balancing storage proposed by the present invention. DETAILED DESCRIPTION

[0069] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0070] A centralized archiving and processing system for electric power engineering data based on cloud storage, comprising:

[0071] Scanning module: The scanning module is mainly used to scan the power engineering data to obtain clear image data of the power engineering data, which is convenient for subsequent analysis and classification operations;

[0072] A cloud storage module, which is mainly used to store power engineering data on a remote server through cloud computing technology, and can access, manage, share and back up the power engineering data through the network anytime and anywhere;

[0073] A correlation module, wherein the correlation module is mainly used to calculate the correlation between the data packets by comparing the data in the data packets;

[0074] Construction module: The construction module is mainly used to construct a hash space of distributed storage based on a hash ring;

[0075] A storage processing module, wherein the storage processing module is mainly used to disperse and store the associated data packets on a hash ring;

[0076] A load processing module, which is mainly used to optimize the use of storage resources to avoid performance degradation or service unavailability due to overload of a single storage node;

[0077] A cloud storage and a processor, wherein the storage and the processor are connected to each other via a network communication, and the processor executes a centralized archiving and processing method for power engineering data based on cloud storage by executing computer instructions.

[0078] See also Figure 1 As shown, the centralized archiving and processing method for power engineering data based on cloud storage specifically includes:

[0079] Step 1: Use a scanner to digitally format all power engineering data;

[0080] Step 2: Upload the digitized power engineering data to the cloud storage platform based on distributed storage technology;

[0081] Step 3: Based on the acquired power engineering data, analyze the correlation between the data and obtain the correlation matrix;

[0082] Step 4: Based on the data management requirements of power engineering materials, a hash space based on distributed storage of hash rings is constructed;

[0083] Step 5: Based on the obtained correlation matrix, the data is dispersedly stored in the hash ring according to the correlation through the distributed hash algorithm;

[0084] Step 6: Dynamically archive and manage power engineering data based on load balancing storage.

[0085] See also Figure 2 As shown, all power engineering data are digitally formatted through a scanner, including:

[0086] Obtain electronic image data of all power engineering materials through scanners;

[0087] Based on the image data, digital image processing is performed to obtain the content data of the power engineering materials;

[0088] Based on the content data of power engineering materials, basic information data of each material is obtained.

[0089] See also Figure 3 As shown, based on the acquired power engineering data, the correlation between the data is analyzed, and the correlation matrix is ​​obtained, which specifically includes:

[0090] Based on the data in the power engineering data of each project, obtain all data packages of the data;

[0091] Based on the data contained in the data packages of all projects, the percentage of the combined data of each data package in the total project data is analyzed to obtain the correlation matrix of the data packages.

[0092] See also Figure 4As shown, based on the data contained in the data packages of all projects, the percentage of the combined data of each data package in the total project data is analyzed, and the correlation matrix of the data packages is obtained, which specifically includes:

[0093] Based on the data contained in the data packets of all the acquired items, a data packet set P is acquired = {P 1 ,P 2 ,P 3 ,...,P n}, each data packet P i Contains data subset D i ={d 1 ,d 2 ,d 3 ,...,d k}, the global data set is

[0094] For each data item d in the data set D k Mapped to index k, and representing each packet P with a binary vector i , the formula is:

[0095]

[0096] Based on two random data packets P from all projects i and P j , calculate and obtain the union D of two data packets i ∪D j , the formula is:

[0097] v i ∨v j =[max(x i1 ,x j1 ),max(x i2 ,x j2 ),...,max(x im ,x jm )]

[0098] Based on the obtained union, calculate its size, the formula is:

[0099]

[0100] Based on the size of the obtained union, calculate the percentage of the obtained union in the entire set. The formula is:

[0101]

[0102] Based on calculating the union ratio data between all data packets, the correlation matrix R is constructed. The matrix is:

[0103]

[0104] See also Figure 5 As shown in the figure, based on the data management requirements of power engineering materials, the construction of a hash space for distributed storage based on a hash ring specifically includes:

[0105] Construct a hash ring of length N and determine the hash value range to be [0, 2 N -1];

[0106] Add a unique identifier to each node;

[0107] Based on the hash function H(x), the data packet P i Mapped onto the ring.

[0108] Specifically, the hash ring distributes data in a ring-shaped node space through a consistent hashing algorithm, so that data can be stored in a distributed manner based on association, avoiding the node load that may be caused by traditional centralized storage. Secondly, the design of the hash ring makes it possible to reallocate only a small part of the data when adding or deleting storage nodes, without the need for full migration. This dynamic expansion capability is very suitable for coping with the growing storage needs of power engineering data, and has little impact on the system, ensuring the continuity of services.

[0109] See also Figure 6 As shown, based on the obtained correlation matrix, the distributed hash algorithm is used to disperse and store the data packets in the hash ring according to the correlation, which specifically includes:

[0110] Based on the correlation matrix, data packets with high correlation are placed at both ends of a diameter of the hash ring, and data packets with low correlation are randomly and evenly distributed on the hash ring.

[0111] It is understandable that placing highly correlated data at both ends of the diameter, although it can reflect the logical association, will cause the data to physically span a larger hash ring distance. By introducing logical partitioning through virtualization technology of hash ring nodes, the access path of highly correlated data is optimized to the shortest path. At the same time, combined with local caching mechanisms, such as hot data caching, the cross-node access cost of highly correlated data is reduced. At the same time, randomly distributed low-correlation data may be completely irregularly scattered on the ring, resulting in a lack of correlation between data stored in some nodes, increasing the coordination overhead during overall data processing. By establishing weak association rules for low-correlation data during the distribution process, it is ensured that even random distribution can optimize the storage layout to a certain extent. Through global analysis of low-correlation data, a weighted hash distribution algorithm can be introduced to enhance the intelligence of data layout.

[0112] See also Figure 7As shown, based on the correlation matrix, data packets with high correlation are placed at both ends of a diameter of the hash ring, and data packets with low correlation are randomly and evenly distributed on the hash ring. Specifically, the method includes:

[0113] If P i and P j is a group of highly correlated data packets placed at both ends of the diameter, with the formula:

[0114]

[0115] Where N is the total length of the hash ring, H(P i ) is always in H(P j ) on the opposite side;

[0116] The data packets with low correlation are randomly placed in the free positions on the ring. The formula is:

[0117] H(P k )=rand()mod N

[0118] If the position H(P j ) is already occupied, a virtual node is created and the data packet is mapped to the virtual node. The formula is:

[0119] H'(P j )=(H(P j )+k·Δ)mod N

[0120] Where k is the virtual machine node number and Δ is the offset of the virtual node.

[0121] See also Figure 8 As shown in the figure, dynamic archiving management of power engineering data based on load balancing storage specifically includes:

[0122] Calculate the number of packets at each position on the hash ring and determine whether the distribution is uniform by the ratio of the maximum load to the minimum load. The formula is:

[0123]

[0124] Based on the polling method, part of the data in the high-load area is migrated to other nodes, and the storage location of the data is determined according to the real-time load situation of each node.

[0125] Specifically, for data with high access frequency, a multi-copy mechanism can be used to retain a copy on the target node while retaining the master copy on the original node to reduce changes in user access paths. A temporary cache is established for hot data to reduce the impact of migration on user requests. At the same time, the migration process is optimized by classifying hot and cold data. Cold data is migrated first to reduce the impact of migrated data on node performance. Dynamic hot and cold classification is used to monitor the access frequency of data in real time and dynamically adjust the hot and cold data division. During the migration process, if the target node fails, the migration task will be reallocated to the next target node or the list of available nodes will be maintained in advance to quickly switch the target node.

[0126] In summary, the advantages of the present invention are: by uploading digital data to the cloud storage platform, efficient centralized management of data is achieved, and the access speed and convenience of data are improved; by using union analysis and correlation matrix, the correlation between different data is accurately mined, which helps to optimize the data organization structure and query efficiency; constructing a distributed storage hash space of a hash ring and combining the correlation to disperse the storage data, not only improves the reliability and scalability of storage, but also effectively reduces the risk of single point failure; dynamic archiving management is achieved through load balancing storage, making storage resource allocation more reasonable, ensuring the efficient operation and stability of the system. This method combines efficient storage with intelligent management, is suitable for the large-scale and complex storage needs of power engineering data, and helps the industry's digital transformation and upgrading.

[0127] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A centralized archiving and processing method for electric power engineering data based on cloud storage, characterized in that: include: All power engineering data are digitally formatted through scanners; Upload the digitized power engineering data to the cloud storage platform based on distributed storage technology; Based on the acquired power engineering data, analyze the correlation between the data and obtain the correlation matrix; Based on the data management requirements of power engineering materials, a hash space with distributed storage based on hash ring is constructed; Based on the obtained correlation matrix, the data is dispersedly stored in the hash ring according to the correlation through the distributed hash algorithm; Dynamic archiving management of power engineering data based on load balancing storage.

2. The centralized archiving and processing method for power engineering data based on cloud storage according to claim 1 is characterized in that: The digital formatting of all power engineering data by means of a scanner specifically includes: Obtain electronic image data of all power engineering materials through scanners; Based on the image data, digital image processing is performed to obtain the content data of the power engineering materials; Based on the content data of power engineering materials, basic information data of each material is obtained.

3. The centralized archiving and processing method for power engineering data based on cloud storage according to claim 1 is characterized in that: The step of analyzing the correlation between various data based on the acquired power engineering data to obtain a correlation matrix specifically includes: Based on the data in the power engineering data of each project, obtain all data packages of the data; Based on the data contained in the data packages of all projects, the percentage of the combined data of each data package in the total project data is analyzed to obtain the correlation matrix of the data packages.

4. The centralized archiving and processing method for power engineering data based on cloud storage according to claim 3 is characterized in that: The data contained in the data packages of all projects, analyzing the percentage of the combined data of each data package in the total data of the project, and obtaining the correlation matrix of the data packages specifically includes: Based on the data contained in the data packets of all the acquired items, a data packet set P = {P1, P2, P3, ..., P n }, each data packet P i Contains data subset D i ={d1,d2,d3,...,d k }, the global data set is For each data item d in the data set D k Mapped to index k, and representing each packet P with a binary vector i , the formula is: v i =[x1,x2,x3,...,x m ], Based on two random data packets P from all projects i and P j , calculate and obtain the union D of two data packets i ∪D j , the formula is: v i ∨v j =[max(x i1 ,x j1 ),max(x i2 ,x j2 ),...,max(x im ,x jm )] Based on the obtained union, calculate its size, the formula is: Based on the size of the obtained union, calculate the percentage of the obtained union in the entire set. The formula is: Based on calculating the union ratio data between all data packets, the correlation matrix R is constructed. The matrix is:

5. The centralized archiving and processing method for power engineering data based on cloud storage according to claim 1 is characterized in that: The construction of a hash space for distributed storage based on a hash ring based on the data management requirements of electric power engineering materials specifically includes: Construct a hash ring of length N and determine the hash value range to be [0, 2 N -1]; Add a unique identifier to each node; Based on the hash function H(x), the data packet P i Mapped onto the ring.

6. The centralized archiving and processing method for power engineering data based on cloud storage according to claim 1 is characterized in that: The method of distributing and storing the data packets in the hash ring according to the association based on the obtained association matrix by using the distributed hash algorithm specifically includes: Based on the correlation matrix, data packets with high correlation are placed at both ends of a diameter of the hash ring, and data packets with low correlation are randomly and evenly distributed on the hash ring.

7. The method for centralized archiving and processing of electric power engineering data based on cloud storage according to claim 5, characterized in that: The method of placing data packets with high correlation at two ends of a diameter of a hash ring based on the correlation matrix and randomly and evenly distributing data packets with low correlation on the hash ring specifically includes: If P i and P j is a group of highly correlated data packets placed at both ends of the diameter, with the formula: Where N is the total length of the hash ring, H(P i ) is always in H(P j ) on the opposite side; The data packets with low correlation are randomly placed in the free positions on the ring. The formula is: H(P k )=rand()mod N If the position H(P j ) is already occupied, a virtual node is created and the data packet is mapped to the virtual node. The formula is: H’(P j )=(H(P j )+k·Δ)mod N Where k is the virtual machine node number and Δ is the offset of the virtual node.

8. The method for centralized archiving and processing of electric power engineering data based on cloud storage according to claim 1, characterized in that: The dynamic archiving management of power engineering data based on load balancing storage specifically includes: Calculate the number of packets at each position on the hash ring and determine whether the distribution is uniform by the ratio of the maximum load to the minimum load. The formula is: Based on the polling method, part of the data in the high-load area is migrated to other nodes, and the storage location of the data is determined according to the real-time load situation of each node.

9. A centralized archiving and processing system for electric power engineering data based on cloud storage, used to implement the centralized archiving and processing method for electric power engineering data based on cloud storage as claimed in any one of claims 1 to 8, characterized in that: include: Scanning module: The scanning module is mainly used to scan the power engineering data to obtain clear image data of the power engineering data, which is convenient for subsequent analysis and classification operations; A cloud storage module, which is mainly used to store power engineering data on a remote server through cloud computing technology, and can access, manage, share and back up the power engineering data through the network anytime and anywhere; A correlation module, wherein the correlation module is mainly used to calculate the correlation between the data packets by comparing the data in the data packets; Construction module: The construction module is mainly used to construct a hash space of distributed storage based on a hash ring; A storage processing module, wherein the storage processing module is mainly used to disperse and store the associated data packets on a hash ring; A load processing module, which is mainly used to optimize the use of storage resources to avoid performance degradation or service unavailability due to overload of a single storage node; A cloud storage and a processor, wherein the storage and the processor are connected to each other via a network communication, and the processor executes the method according to any one of claims 1 to 8 by executing computer instructions.

Images