A method and system for splitting and storing electronic archives based on distributed storage

Abstract

The present invention relates to the field of data processing technology, and discloses a method and system for splitting and storing electronic archives based on distributed storage, the system comprising: a collection unit collects all documents to be stored within a preset period of time, extracts document features and performs cluster analysis to divide the collection; a processing unit determines the importance index of the document, and integrates to determine the average index of each collection; a judgment unit compares the average index with the index threshold to determine the storage location; an adjustment unit compares the importance index with historical data when determining to store the document to be stored in the first storage unit or the second storage unit, adjusts the importance index, obtains the adjusted importance index, and determines the storage location of the document to be stored; the storage unit comprises a first storage unit, a second storage unit and a third storage unit. The present application realizes intelligent grouping and efficient management of documents by introducing a distributed storage architecture, combining document feature extraction, cluster analysis and importance index calculation.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for splitting and storing electronic archives based on distributed storage. Background Art

[0002] With the popularization of digital management and the rapid growth of information storage needs, enterprises and institutions are increasingly relying on electronic archive management systems to store and manage large amounts of document data.

[0003] However, traditional centralized storage methods face many challenges: first, the types of documents are diverse, and the confidentiality and importance of the content vary significantly, making it difficult to achieve efficient classification management and protection in a unified storage architecture; second, the security of sensitive documents is difficult to guarantee, and unified encryption methods are not enough to deal with complex security threats; third, storage resource allocation is inefficient, especially in a distributed storage architecture, how to optimize storage locations based on document characteristics and access frequency is still a difficult problem. In addition, existing methods often lack a dynamic adjustment mechanism and cannot flexibly optimize storage strategies based on changes in historical data.

[0004] Therefore, it is necessary to design an electronic archive splitting storage method and system based on distributed storage to solve the problems existing in the current technology. Summary of the invention

[0005] In view of this, the present invention proposes an electronic archive splitting storage method and system based on distributed storage, aiming to solve the problems existing in the current electronic archive storage technology, such as lack of classified storage and lack of dynamic adjustment mechanism of storage resource allocation strategy, resulting in low archive storage security and low allocation efficiency.

[0006] In one aspect, the present invention proposes an electronic archive splitting storage system based on distributed storage, comprising:

[0007] A collection unit is configured to collect all documents to be stored within a preset time period, extract document features of each document to be stored, perform cluster analysis on all documents to be stored according to the document features, and divide the documents into sets according to the clustering results;

[0008] A processing unit is configured to determine the importance index of each of the to-be-stored documents according to the size and confidentiality of the documents, and integrate the importance indexes in each set to determine an average index of each set;

[0009] A judgment unit is configured to compare the average index with an index threshold, and determine a storage location of each of the to-be-stored documents in the set according to the comparison result; when the average index of the set is greater than the index threshold, the judgment unit determines to perform AES (Advanced Encryption Standard)-256 encryption on the to-be-stored documents in the set, and store them in the first storage unit or the second storage unit;

[0010] an adjusting unit, configured to, when the judging unit determines to store the document to be stored in the collection in the first storage unit or the second storage unit, compare the importance index of each document to be stored in the collection with the historical data, adjust the importance index according to the comparison result, obtain the adjusted importance index, and determine the storage location of the document to be stored according to the adjusted importance index;

[0011] The storage unit includes a first storage unit, a second storage unit and a third storage unit.

[0012] Furthermore, when the acquisition unit extracts the document features of each of the documents to be stored, it includes:

[0013] Analyze all documents to be stored based on NLP (Natural Language Processing), perform word segmentation, remove stop words, and extract stems on the documents to be stored to obtain valid text;

[0014] A dependency syntactic analysis model is used to parse the relationship between words to extract noun phrases, and the noun phrases that conform to the syntactic structure are used as keywords;

[0015] Compare the keyword with the confidential word set, if it appears, it is recorded as 1, otherwise it is recorded as 0, and the number of occurrences of the confidential word is recorded;

[0016] The document size, creation time and number of occurrences of confidential words of each document to be stored are used as the document features.

[0017] Furthermore, the collection unit performs cluster analysis on all the documents to be stored according to the document features, and divides the collection according to the clustering results, including:

[0018] S101: the acquisition unit normalizes each parameter in the document feature to form a document feature vector;

[0019] S102: combining document feature vectors of all the documents to be stored into a feature matrix;

[0020] S103: setting the number of clusters k, and calculating the Euclidean distance from each document feature vector to each cluster center;

[0021]

[0022] Where d represents the Euclidean distance, fjm represents the mth eigenvalue of document j, and cim represents the mth eigenvalue of cluster center i;

[0023] S104: Allocate the document to the set corresponding to the nearest cluster center;

[0024] S105: recalculate the position of each cluster center;

[0025] S106: Repeat S104 and S105 until the position of the cluster center no longer changes or the preset number of iterations is reached, and all documents to be stored are divided into k sets.

[0026] Furthermore, when the processing unit determines the importance index of each of the documents to be stored according to the size of the document and the amount of confidentiality involved, it includes:

[0027]

[0028] Where I represents the importance index, Sd represents the normalized result of the document size, Sc represents the normalized result of the number of occurrences of confidential words, Sp represents the frequency of occurrence of confidential words, Tc represents the creation time, a, b, c represent weight coefficients, and a+b+c=1, α represents the sensitivity adjustment coefficient, and the value range of α is [0.01, 10].

[0029] Furthermore, when the judgment unit determines the storage location of each of the to-be-stored documents in the collection according to the comparison result, it includes:

[0030] When the average index of the set is greater than the index threshold, the judgment unit determines to perform AES-256 encryption on the documents to be stored in the set and store them in the first storage unit or the second storage unit;

[0031] When the average index of the set is less than or equal to the index threshold, the judgment unit determines to perform AES-168 encryption on all the documents to be stored in the set and store them in the third storage unit.

[0032] Furthermore, the adjustment unit compares the importance index of each document to be stored in the collection with the historical data, adjusts the importance index according to the comparison result, and determines the storage location of the document to be stored according to the adjusted importance index, including:

[0033] The adjustment unit compares the importance index of each document to be stored with each historical importance index in the historical data to determine the importance index difference ratio, and when the importance index difference ratio is less than 10%, the historical data corresponding to the historical importance index is included in the similar set;

[0034] When there is data in the similar set of the document to be stored, the adjustment unit adjusts the importance index according to the similar set to obtain the adjusted importance index;

[0035] When there is no data in the similar set of the to-be-stored document, the adjustment unit adjusts the importance index according to the difference between the importance index of the to-be-stored document and the average index of the set to obtain the adjusted importance index;

[0036] The adjustment unit also compares the adjusted importance index of each document to be stored with the index threshold. When the adjusted importance index is greater than or equal to the index threshold, the adjustment unit stores the document to be stored in the first storage unit; when the adjusted importance index is less than the index threshold, the adjustment unit stores the document to be stored in the second storage unit.

[0037] Further, when the adjustment unit determines to adjust the importance index according to the similar set, it includes:

[0038] The adjustment unit extracts the number of accesses of each historical document in the similar set, and obtains an average number of accesses according to the number of accesses;

[0039] The adjustment unit includes historical documents with access times greater than the average access times in a first adjustment set, and includes historical documents with access times less than the average access times in a second adjustment set;

[0040] The adjustment unit determines an adjustment coefficient according to the average number of visits and the first adjustment set and / or the second adjustment set to adjust the importance index.

[0041] Further, when the adjustment unit determines the adjustment coefficient to adjust the importance index according to the average number of visits and the first adjustment set and / or the second adjustment set, it includes:

[0042] When the historical importance indexes in the similar set are all greater than or equal to the importance index of the document to be stored, the adjustment unit determines an adjustment coefficient according to the average number of accesses and a second adjustment set to adjust the importance index;

[0043]

[0044] When the historical importance indexes in the similar set are all less than or equal to the importance index of the document to be stored, the adjustment unit determines an adjustment coefficient according to the average number of accesses and the first adjustment set to adjust the importance index;

[0045]

[0046] When the existence history importance index in the similar set is greater than the importance index of the document to be stored and the existence history importance index is less than the importance index of the document to be stored, the adjustment unit determines an adjustment coefficient according to the average number of accesses and the first adjustment set and the second adjustment set to adjust the importance index;

[0047] ;

[0048] Among them, P represents the adjustment coefficient, Ct represents the number of visits to the t-th historical document in the first adjustment set, N1 represents the number of documents in the first adjustment set, C0 represents the average number of visits, Cn represents the number of visits to the n-th historical document in the second adjustment set, and N2 represents the number of documents in the second adjustment set.

[0049] Furthermore, when the adjusting unit adjusts the importance index according to the difference between the importance index of the document to be stored and the set average index, it includes:

[0050] The adjustment unit determines an adjustment coefficient according to a difference between the importance index of the document to be stored and the set average index to adjust the importance index, and the adjustment coefficient is proportional to the difference.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows: by introducing a distributed storage architecture, combined with document feature extraction, cluster analysis and important index calculation, intelligent grouping and efficient management of documents are achieved, and cluster analysis based on document features can accurately classify documents, thereby improving the efficiency and pertinence of document management; by calculating the important index of a document based on the comprehensive document size and confidentiality, and making dynamic adjustments based on this, storage resource allocation is optimized, and the protection capability for sensitive documents is improved; AES-256 encryption is used to perform hierarchical storage of highly sensitive documents, effectively enhancing the security of the documents; the adjustment unit can flexibly adjust the important index of a document and refine the storage location according to historical data, so that the system has adaptive optimization capabilities, further improving the dynamic nature of the storage strategy.

[0052] On the other hand, the present application also provides an electronic archive splitting and storage method based on distributed storage, which is applied to the above-mentioned electronic archive splitting and storage system based on distributed storage, including:

[0053] Collect all documents to be stored within a preset time period, extract document features of each document to be stored, perform cluster analysis on all documents to be stored according to the document features, and divide the collection according to the clustering results;

[0054] Determine the importance index of each of the documents to be stored according to the size and confidentiality of the documents, and integrate the importance indexes in each set to determine the average index of each set;

[0055] The average index is compared with the index threshold, and the storage location of each of the to-be-stored documents in the set is determined according to the comparison result; when the average index of the set is greater than the index threshold, the to-be-stored documents in the set are encrypted with AES-256 and stored in the first storage unit or the second storage unit;

[0056] When it is determined that the document to be stored in the collection is stored in the first storage unit or the second storage unit, the importance index of each document to be stored in the collection is compared with the historical data, the importance index is adjusted according to the comparison result, the adjusted importance index is obtained, and the storage location of the document to be stored is determined according to the adjusted importance index.

[0057] It is understandable that the above-mentioned electronic archive splitting and storage method and system based on distributed storage have the same beneficial effects, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0059] Figure 1 A functional block diagram of an electronic archive splitting and storage system based on distributed storage provided by an embodiment of the present invention;

[0060] Figure 2 A flowchart of a method for splitting and storing electronic archives based on distributed storage provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0062] In some embodiments of the present application, see Figure 1 As shown, an electronic archive splitting and storage system based on distributed storage includes: a collection unit, a processing unit, a judgment unit, an adjustment unit and a storage unit, wherein:

[0063] The collection unit is configured to collect all documents to be stored within a preset time period, extract document features of each document to be stored, perform cluster analysis on all documents to be stored according to the document features, and divide the collection according to the clustering results.

[0064] The processing unit is configured to determine the importance index of each document to be stored according to the size and confidentiality of the document, and integrate the importance indexes in each set to determine the average index of each set.

[0065] The judgment unit is configured to compare the average index with the index threshold, and determine the storage location of each to-be-stored document in the collection according to the comparison result. When the average index of the collection is greater than the index threshold, the judgment unit determines to perform AES-256 encryption on the to-be-stored document in the collection and store it in the first storage unit or the second storage unit.

[0066] The adjustment unit is configured to compare the importance index of each document to be stored in the collection with the historical data when the judgment unit determines to store the document to be stored in the collection in the first storage unit or the second storage unit, adjust the importance index according to the comparison result, obtain the adjusted importance index, and determine the storage location of the document to be stored according to the adjusted importance index.

[0067] The storage unit includes a first storage unit, a second storage unit, and a third storage unit.

[0068] Specifically, the collection unit obtains all documents to be stored within a preset time period and extracts their document features. These document features may include document size, format, content keywords, etc. Through clustering, the documents are divided into multiple categories or sets according to the document features, so that similar or related documents are classified into the same set, forming the basis for efficient classification management. The processing unit further analyzes the document data in each set, calculates the importance index of the document by comprehensively considering the document size and confidentiality, and integrates the document importance index of each set to obtain the average index of the set. It reflects the overall importance of the documents in the set and provides a basis for subsequent storage decisions. The judgment unit is responsible for comparing the average index of the set with the preset index threshold, and judging the storage location of the documents in the set according to the comparison result. If the average index of the set is higher than the threshold, the AES-256 encryption measure will be activated to enhance storage security, and the document will be stored in the first storage unit or the second storage unit. If the average index of the set is lower than or equal to the threshold, the AES-168 encryption measure will be activated. After determining that the document starts the AES-256 encryption measure, the adjustment unit further introduces a dynamic optimization mechanism. By comparing the importance index of the document with historical data, the importance index of the document is dynamically adjusted, and the storage location of the first storage unit or the second storage unit is further allocated according to the adjusted index. The adaptive adjustment mechanism ensures that the system can continuously optimize the storage strategy when the data distribution, access frequency and document importance change. The storage unit consists of the first storage unit, the second storage unit and the third storage unit, which are respectively suitable for document storage with different levels of importance and security requirements. Among them, the first storage unit and the second storage unit are used for hierarchical storage of highly sensitive documents, while the third storage unit can store general documents.

[0069] It is understandable that the introduction of distributed storage architecture and multi-dimensional dynamic adjustment mechanism has improved the efficiency, security and flexibility of electronic archive management. Specifically, intelligent grouping is achieved through document feature clustering, which improves the accuracy of document classification; storage strategy is dynamically adjusted based on document importance index to optimize storage resource allocation; AES-256 encryption is used to ensure the security of highly sensitive documents; dynamic optimization enhances the adaptability to changing data, and realizes intelligent and secure electronic archive management.

[0070] In some embodiments of the present application, when the acquisition unit extracts the document features of each document to be stored, it includes: analyzing all documents to be stored based on NLP, performing word segmentation, removing stop words, and stemming extraction on the documents to be stored to obtain effective text. Using a dependency parsing model to parse the relationships between words and extract noun phrases, and taking the noun phrases that conform to the syntactic structure as keywords. Comparing the keywords with the set of classified words, if they appear, recording as 1, otherwise recording as 0, and recording the number of occurrences of classified words. Taking the document size, creation time, and the number of occurrences of classified words of each document to be stored as document features.

[0071] Specifically, the first step of text processing is to perform text analysis on all documents to be stored through natural language processing technology, including the following three main steps: Word segmentation: splitting the continuous text in the document into independent words or phrases for convenient subsequent processing. Removing stop words: removing common words that do not affect the actual semantics in the document (such as "of", "is", "in", etc.) to improve the effectiveness of text analysis. Stemming extraction: reducing a word to its root form (such as unifying "storage" and "stored" as "storage") to reduce semantic redundancy. Using a dependency parsing model to parse the grammatical and semantic relationships between words and extract noun phrases from the document. Dependency parsing can accurately identify noun phrases (such as "sensitive information", "confidential document"), which reflect the theme or classified level of the document and are the key basis for keyword extraction. Comparing the extracted keywords with a predefined set of classified words: if a keyword appears in the set of classified words, it is marked as "1", otherwise it is marked as "0". At the same time, counting the number of occurrences of classified words in the document.

[0072] It can be understood that by combining NLP technology and dependency parsing, the accurate extraction of document features and the effective identification of sensitive information are achieved. Keyword extraction is based on syntactic structure, ensuring the semantic value of the extracted content. Through keyword comparison and sensitive word statistics, the classified level of the document can be effectively quantified. The extracted feature data is directly used for the calculation of the importance index, improving the efficiency and security of the overall system.

[0073] In some embodiments of the present application, when the acquisition unit performs clustering analysis on all documents to be stored according to the document features and divides the set according to the clustering results, it includes:

[0074] S101: The acquisition unit normalizes each parameter in the document features to form a document feature vector.

[0075] S102: Combining the document feature vectors of all documents to be stored into a feature matrix.

[0076] S103: Setting the number of clusters k, and calculating the Euclidean distance from each document feature vector to each cluster center.

[0077] ;

[0078] Among them, d represents the Euclidean distance, fjm represents the mth eigenvalue of document j, and cim represents the mth eigenvalue of cluster center i.

[0079] S104: Allocate the document to the set corresponding to the nearest cluster center.

[0080] S105: Recalculate the position of each cluster center.

[0081] S106: Repeat S104 and S105 until the position of the cluster center no longer changes or the preset number of iterations is reached, and all documents to be stored are divided into k sets.

[0082] Specifically, each parameter in the document features (such as size, creation time, and number of occurrences of confidential words) is normalized to ensure that all feature values ​​are in the same magnitude range (such as [0,1]) to avoid distortion of clustering results due to scale differences between features.

[0083] It is understandable that the introduction of the k-means clustering algorithm has achieved high efficiency in document classification management. Based on multidimensional feature vectors and normalization processing, the similarity between document features is fully captured to ensure the accuracy of the division results. Through iterative optimization, the clustering task can be completed in a limited time, which is suitable for the processing needs of large-scale document data. The number of clusters k and the number of iterations can be adjusted according to the specific application scenario, with high adaptability. The collection after document division provides data support for subsequent dynamic storage strategies, improving the allocation efficiency and security of storage resources.

[0084] In some embodiments of the present application, when the processing unit determines the importance index of each document to be stored according to the size of the document and the amount of confidentiality involved, it includes:

[0085] ;

[0086] Where I represents the importance index, Sd represents the normalized result of the document size, Sc represents the normalized result of the number of occurrences of confidential words, Sp represents the frequency of occurrence of confidential words, Tc represents the creation time, a, b, c represent weight coefficients, and a+b+c=1, α represents the sensitivity adjustment coefficient, and the value range of α is [0.01, 10].

[0087] In some embodiments of the present application, when the judgment unit determines the storage location of each to-be-stored document in the collection according to the comparison result, it includes:

[0088] When the average index of the set is greater than the index threshold, the judgment unit determines to perform AES-256 encryption on the documents to be stored in the set and store them in the first storage unit or the second storage unit.

[0089] When the average index of the set is less than or equal to the index threshold, the judgment unit determines to perform AES-168 encryption on all the documents to be stored in the set and store them in the third storage unit.

[0090] Specifically, when the average index of a collection is greater than a threshold, the collection contains highly sensitive documents and needs to be protected. AES-256 is used for encryption and stored in the first storage unit or the second storage unit. When the average index of a collection is less than or equal to the threshold, the collection contains low-sensitivity documents and conventional protection is adopted. AES-16 is used for encryption and stored in the third storage unit.

[0091] It is understandable that by combining the document size, content sensitivity and time characteristics, dynamically calculating the important index and intelligently selecting the storage location, it is possible to comprehensively consider multi-dimensional characteristics and make the evaluation results more accurate. It supports adjustment of the sensitivity adjustment coefficient to meet the needs of multiple scenarios. Highly sensitive documents are strongly encrypted and stored in high-security storage units, which improves data protection capabilities. Storage resources are dynamically allocated according to the collective average index to avoid resource waste.

[0092] In some embodiments of the present application, when the adjustment unit compares the importance index of each document to be stored in the set with the historical data, adjusts the importance index according to the comparison result, and determines the storage location of the document to be stored according to the adjusted importance index, it includes:

[0093] The adjustment unit compares the importance index of each document to be stored with each historical importance index in the historical data to determine the importance index difference ratio. When the importance index difference ratio is less than 10%, the historical data corresponding to the historical importance index is included in the similar set.

[0094] When data exists in the similar set of the document to be stored, the adjustment unit adjusts the importance index according to the similar set to obtain the adjusted importance index.

[0095] When there is no data in the similar set of the document to be stored, the adjustment unit adjusts the importance index according to the difference between the importance index of the document to be stored and the average index of the set to obtain the adjusted importance index.

[0096] The adjustment unit also compares the adjusted importance index of each document to be stored with the index threshold, and when the adjusted importance index is greater than or equal to the index threshold, the adjustment unit stores the document to be stored in the first storage unit. When the adjusted importance index is less than the index threshold, the adjustment unit stores the document to be stored in the second storage unit.

[0097] In some embodiments of the present application, when the adjustment unit determines to adjust the importance index according to the similar set, it includes:

[0098] The adjustment unit extracts the access times of each historical document in the similar set, and obtains the average access times according to the access times.

[0099] The adjustment unit lists historical documents whose access times are greater than the average access times in a first adjustment set, and lists historical documents whose access times are less than the average access times in a second adjustment set.

[0100] The adjustment unit determines an adjustment coefficient according to the average number of accesses and the first adjustment set and / or the second adjustment set to adjust the importance index.

[0101] In some embodiments of the present application, when the adjustment unit determines the adjustment coefficient to adjust the importance index according to the average number of accesses and the first adjustment set and / or the second adjustment set, the method includes:

[0102] When the historical importance indexes in the similar set are all greater than or equal to the importance index of the document to be stored, the adjustment unit determines an adjustment coefficient according to the average number of accesses and the second adjustment set to adjust the importance index.

[0103] ;

[0104] When the historical importance indexes in the similar set are all less than or equal to the importance index of the document to be stored, the adjustment unit determines an adjustment coefficient according to the average number of accesses and the first adjustment set to adjust the importance index.

[0105] ;

[0106] When the existence history importance index in the similar set is greater than the importance index of the document to be stored and the existence history importance index is less than the importance index of the document to be stored, the adjustment unit determines the adjustment coefficient to adjust the importance index according to the average number of accesses and the first adjustment set and the second adjustment set.

[0107] ;

[0108] Among them, P represents the adjustment coefficient, Ct represents the number of visits to the t-th historical document in the first adjustment set, N1 represents the number of documents in the first adjustment set, C0 represents the average number of visits, Cn represents the number of visits to the n-th historical document in the second adjustment set, and N2 represents the number of documents in the second adjustment set.

[0109] Specifically, the adjustment unit compares the importance index of each document to be stored with the historical importance index one by one, and determines the difference ratio, which is the ratio of the absolute value of the difference between the importance index of the document to be stored and the historical importance index to the importance index of the document to be stored. When the difference ratio is less than 10%, it is determined that the historical data belongs to a similar set. The establishment of similar sets is based on the principle of similar document importance, which is helpful for subsequent adjustment operations. If there is a similar set, the importance index of the document to be stored is adjusted according to the historical data in the set. When determining the adjustment coefficient to adjust the importance index, that is, the product of the adjustment coefficient and the importance index, the existing data is fully utilized to optimize the storage strategy. If there is no similar set, independent adjustments are made based on the difference between the importance index of the document to be stored and the average index of the set to ensure the flexibility of the adjustment.

[0110] It is understandable that by introducing historical data comparison and access frequency calculation, the accuracy of the importance index adjustment of the documents to be stored is improved, and the dynamic allocation of storage resources is optimized. The historical data, access frequency and importance index differences are comprehensively considered to achieve accurate assessment and dynamic adjustment of the importance of documents. The document storage strategy is dynamically adjusted according to similar sets. Even if there are no similar sets, the flexibility of the system can be guaranteed through an independent adjustment mechanism. Based on the adjusted importance index, the document is accurately matched to the appropriate storage unit, which effectively improves the utilization rate of storage resources. Different storage locations are used for high-priority documents, and the storage locations of low-priority documents are reasonably allocated, taking into account both data protection and access efficiency.

[0111] In some embodiments of the present application, when the adjustment unit adjusts the importance index according to the difference between the importance index of the document to be stored and the average index of the set, it includes: the adjustment unit determines an adjustment coefficient according to the difference between the importance index of the document to be stored and the average index of the set to adjust the importance index, and the adjustment coefficient is proportional to the difference.

[0112] Specifically, the adjustment unit first calculates the difference between the importance index of the document to be stored and the average index of the collection to which it belongs. When the importance index of the document to be stored is greater than the average index, the larger the importance index is, the larger the adjustment coefficient is, and the value range is [1.1, 1.5]. When the importance index of the document to be stored is less than the average index, the smaller the importance index is, the smaller the adjustment coefficient is, and the value range is [0.7, 0.9].

[0113] It is understandable that in the absence of similar sets, it is still possible to reasonably optimize important indexes based on the average index of the set to avoid rigidity or failure of storage strategies. By designing the adjustment coefficient proportional to the difference, appropriate compensation is made for documents that deviate greatly from the average level of the set, effectively avoiding imbalance in resource allocation.

[0114] In the above embodiments, by introducing a distributed storage architecture, combined with document feature extraction, cluster analysis and important index calculation, intelligent grouping and efficient management of documents are achieved. Cluster analysis based on document features can accurately classify documents, improving the efficiency and pertinence of document management. By calculating the important index of a document based on the size of the document and the amount of confidentiality involved, and dynamically adjusting it based on this, the allocation of storage resources is optimized and the protection of sensitive documents is improved. AES-256 encryption is used to store highly sensitive documents in a hierarchical manner, which effectively enhances the security of the documents. The adjustment unit can flexibly adjust the important index of the document and refine the storage location based on historical data, so that the system has adaptive optimization capabilities, further improving the dynamic nature of the storage strategy.

[0115] In another preferred embodiment based on the above embodiment, refer to Figure 2 As shown, this embodiment provides an electronic archive splitting and storage method based on distributed storage, which is applied to the above-mentioned electronic archive splitting and storage system based on distributed storage, including:

[0116] S100: Collect all documents to be stored within a preset period, extract document features of each document to be stored, perform cluster analysis on all documents to be stored according to the document features, and divide the documents into sets according to the clustering results;

[0117] S200: determining the importance index of each document to be stored according to the size of the document and the amount of confidentiality involved, and integrating the importance indexes in each set to determine the average index of each set;

[0118] S300: Compare the average index with the index threshold, and determine the storage location of each to-be-stored document in the collection according to the comparison result; when the average index of the collection is greater than the index threshold, encrypt the to-be-stored document in the collection with AES-256 and store it in the first storage unit or the second storage unit;

[0119] S400: When it is determined that the document to be stored in the collection is stored in the first storage unit or the second storage unit, the importance index of each document to be stored in the collection is compared with the historical data, the importance index is adjusted according to the comparison result, the adjusted importance index is obtained, and the storage location of the document to be stored is determined according to the adjusted importance index.

[0120] It is understandable that by introducing a distributed storage architecture, combined with document feature extraction, cluster analysis and importance index calculation, intelligent grouping and efficient management of documents are achieved. Cluster analysis based on document features can accurately classify documents, improving the efficiency and pertinence of document management; by calculating the importance index of the document based on the comprehensive document size and confidentiality, and making dynamic adjustments based on this, the allocation of storage resources is optimized and the protection of sensitive documents is improved; AES-256 encryption is used to store highly sensitive documents in a hierarchical manner, effectively enhancing the security of the documents; the adjustment unit can flexibly adjust the importance index of the document and refine the storage location based on historical data, so that the system has adaptive optimization capabilities, further improving the dynamic nature of the storage strategy.

[0121] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An electronic archive splitting and storage system based on distributed storage, characterized in that: include: A collection unit is configured to collect all documents to be stored within a preset time period, extract document features of each document to be stored, perform cluster analysis on all documents to be stored according to the document features, and divide the documents into sets according to the clustering results; A processing unit is configured to determine the importance index of each of the to-be-stored documents according to the size and confidentiality of the documents, and integrate the importance indexes in each set to determine an average index of each set; A judgment unit is configured to compare the average index with the index threshold, and determine the storage location of each of the to-be-stored documents in the set according to the comparison result; when the average index of the set is greater than the index threshold, the judgment unit determines to perform AES-256 encryption on the to-be-stored documents in the set, and store them in the first storage unit or the second storage unit; an adjusting unit, configured to, when the judging unit determines to store the document to be stored in the collection in the first storage unit or the second storage unit, compare the importance index of each document to be stored in the collection with the historical data, adjust the importance index according to the comparison result, obtain the adjusted importance index, and determine the storage location of the document to be stored according to the adjusted importance index; The storage unit includes a first storage unit, a second storage unit and a third storage unit; The adjusting unit compares the importance index of each document to be stored in the collection with the historical data, adjusts the importance index according to the comparison result, and determines the storage location of the document to be stored according to the adjusted importance index, including: The adjustment unit compares the importance index of each document to be stored with each historical importance index in the historical data to determine the importance index difference ratio, and when the importance index difference ratio is less than 10%, the historical data corresponding to the historical importance index is included in the similar set; When there is data in the similar set of the document to be stored, the adjustment unit adjusts the importance index according to the similar set to obtain the adjusted importance index; When there is no data in the similar set of the to-be-stored document, the adjustment unit adjusts the importance index according to the difference between the importance index of the to-be-stored document and the average index of the set to obtain the adjusted importance index; The adjustment unit also compares the adjusted importance index of each document to be stored with the index threshold, and when the adjusted importance index is greater than or equal to the index threshold, the adjustment unit stores the document to be stored in the first storage unit; When the adjusted importance index is less than the index threshold, the adjustment unit stores the document to be stored in the second storage unit.

2. The electronic archive splitting and storage system based on distributed storage according to claim 1 is characterized in that: When the acquisition unit extracts the document features of each of the documents to be stored, it includes: Analyze all documents to be stored based on NLP, perform word segmentation, remove stop words, and extract stems on the documents to be stored to obtain valid text; A dependency syntactic analysis model is used to parse the relationship between words to extract noun phrases, and the noun phrases that conform to the syntactic structure are used as keywords; Compare the keyword with the confidential word set, if it appears, it is recorded as 1, otherwise it is recorded as 0, and the number of occurrences of the confidential word is recorded; The document size, creation time and number of occurrences of confidential words of each document to be stored are used as the document features.

3. The electronic archive splitting and storage system based on distributed storage according to claim 1 is characterized in that: The acquisition unit performs cluster analysis on all the documents to be stored according to the document features, and divides the set according to the clustering results, including: S101: the acquisition unit normalizes each parameter in the document feature to form a document feature vector; S102: combining document feature vectors of all the documents to be stored into a feature matrix; S103: setting the number of clusters k, and calculating the Euclidean distance from each document feature vector to each cluster center; ； Where d represents the Euclidean distance, fjm represents the mth eigenvalue of document j, and cim represents the mth eigenvalue of cluster center i; S104: Allocate the document to the set corresponding to the nearest cluster center; S105: recalculate the position of each cluster center; S106: Repeat S104 and S105 until the position of the cluster center no longer changes or the preset number of iterations is reached, and all documents to be stored are divided into k sets.

4. The electronic archive splitting and storage system based on distributed storage according to claim 1 is characterized in that: When the processing unit determines the importance index of each of the documents to be stored according to the size and confidentiality of the documents, it includes: ； Where I represents the importance index, Sd represents the normalized result of the document size, Sc represents the normalized result of the number of occurrences of confidential words, Sp represents the frequency of occurrence of confidential words, Tc represents the creation time, a, b, c represent weight coefficients, and a+b+c=1, α represents the sensitivity adjustment coefficient, and the value range of α is [0.01, 10].

5. The electronic archive splitting and storage system based on distributed storage according to claim 1 is characterized in that: When the judgment unit determines the storage location of each of the to-be-stored documents in the collection according to the comparison result, it includes: When the average index of the set is greater than the index threshold, the judgment unit determines to perform AES-256 encryption on the documents to be stored in the set and store them in the first storage unit or the second storage unit; When the average index of the set is less than or equal to the index threshold, the judgment unit determines to perform AES-168 encryption on all the documents to be stored in the set and store them in the third storage unit.

6. The electronic archive splitting and storage system based on distributed storage according to claim 1 is characterized in that: When the adjustment unit determines to adjust the importance index according to the similar set, it includes: The adjustment unit extracts the number of accesses of each historical document in the similar set, and obtains an average number of accesses according to the number of accesses; The adjustment unit includes historical documents with access times greater than the average access times in a first adjustment set, and includes historical documents with access times less than the average access times in a second adjustment set; The adjustment unit determines an adjustment coefficient according to the average number of visits and the first adjustment set and / or the second adjustment set to adjust the importance index.

7. The electronic archive splitting and storage system based on distributed storage according to claim 6 is characterized in that: When the adjustment unit determines the adjustment coefficient according to the average number of visits and the first adjustment set and / or the second adjustment set to adjust the importance index, it includes: When the historical importance indexes in the similar set are all greater than or equal to the importance index of the document to be stored, the adjustment unit determines an adjustment coefficient according to the average number of accesses and a second adjustment set to adjust the importance index; ； When the historical importance indexes in the similar set are all less than or equal to the importance index of the document to be stored, the adjustment unit determines an adjustment coefficient according to the average number of accesses and the first adjustment set to adjust the importance index; ； When the existence history importance index in the similar set is greater than the importance index of the document to be stored and the existence history importance index is less than the importance index of the document to be stored, the adjustment unit determines an adjustment coefficient according to the average number of accesses and the first adjustment set and the second adjustment set to adjust the importance index; ； Among them, P represents the adjustment coefficient, Ct represents the number of visits to the t-th historical document in the first adjustment set, N1 represents the number of documents in the first adjustment set, C0 represents the average number of visits, Cn represents the number of visits to the n-th historical document in the second adjustment set, and N2 represents the number of documents in the second adjustment set.

8. The electronic archive splitting and storage system based on distributed storage according to claim 1 is characterized in that: When the adjusting unit adjusts the importance index according to the difference between the importance index of the document to be stored and the set average index, it includes: The adjustment unit determines an adjustment coefficient according to a difference between the importance index of the document to be stored and the set average index to adjust the importance index, and the adjustment coefficient is proportional to the difference.

9. A method for splitting and storing electronic archives based on distributed storage, applied to the electronic archive splitting and storing system based on distributed storage as claimed in any one of claims 1 to 8, characterized in that: include: Collect all documents to be stored within a preset time period, extract document features of each document to be stored, perform cluster analysis on all documents to be stored according to the document features, and divide the collection according to the clustering results; Determine the importance index of each of the documents to be stored according to the size and confidentiality of the documents, and integrate the importance indexes in each set to determine the average index of each set; The average index is compared with the index threshold, and the storage location of each of the to-be-stored documents in the set is determined according to the comparison result; when the average index of the set is greater than the index threshold, the to-be-stored documents in the set are encrypted with AES-256 and stored in the first storage unit or the second storage unit; When it is determined that the document to be stored in the collection is stored in the first storage unit or the second storage unit, the importance index of each document to be stored in the collection is compared with the historical data, the importance index is adjusted according to the comparison result, the adjusted importance index is obtained, and the storage location of the document to be stored is determined according to the adjusted importance index.

Images