A method and system for online management of digital archives based on big data

Abstract

This invention discloses a method and system for online management of digital archives based on big data, belonging to the field of intelligent digital archive management technology. It includes collecting digital archive data and uploading it to a database; automatically generating metadata for corresponding archives using NLP technology; deduplicating and compressing archive data using a redundancy detection algorithm; scoring and classifying compressed archives using time series analysis; dynamically allocating archives using a dynamic K-Means clustering algorithm; implementing hierarchical access control for archives using a zero-trust model; dynamically decompressing user-selected archive content using streaming loading and decompression technology; and jointly determining abnormal user behavior during decompression using the LOF and CFSFDP algorithms. This achieves efficient management and security protection of digital archives, providing efficient resource allocation capabilities while comprehensively improving the security and adaptability of the archive system.

Description

Technical Field

[0001] This invention relates to the field of intelligent management technology for digital archives, and in particular to an online management method and system for digital archives based on big data. Background Technology

[0002] With the rapid development of information technology, especially big data and cloud storage technologies, digital archives management has gradually replaced traditional paper archives management, becoming the mainstream method of archives management in various industries. Through digital storage and management technologies, the efficiency of digital archives in information collection, preservation, indexing, sharing, and long-term storage has been significantly improved, thus meeting the needs of efficient management of massive amounts of data, rapid retrieval, and information security. Currently, the core objective of digital archives management systems is to achieve efficient data storage and analysis, as well as dynamic matching with user needs, when faced with a large number of archives and diverse archive structures. In recent years, the rise of big data technology has brought new technical means to digital archives management. Through in-depth analysis and multi-dimensional analysis of archive data, it is possible to more intelligently optimize archive access, clean up redundant data, and intelligently analyze abnormal user behavior. However, against this backdrop of rapid development, digital archives management still faces many technical bottlenecks, such as the inability to accurately detect data redundancy, low archive access efficiency, lack of dynamic management of the archive lifecycle, and insufficient security of access permissions.

[0003] Existing technologies for digital archive management have several shortcomings, directly limiting the system's intelligence and practical effectiveness. First, current technologies for digital archive management often employ metadata generation schemes based on fixed field matching. However, because they cannot utilize the deep semantic characteristics of archive text to generate high-quality metadata, subsequent issues arise in archive content indexing, semantic retrieval, and hierarchical management, leading to inaccurate retrieval and chaotic classification, thus degrading the user experience. Second, traditional redundancy detection algorithms, when deduplicating and compressing archive data, mostly rely solely on character matching or fixed-format parsing, failing to identify semantically repetitive information. This results in insufficient redundancy detection accuracy and underutilization of storage resources. Third, in the storage and allocation of archives, existing solutions typically employ static hierarchical rules, failing to adjust in real-time based on dynamic access behavior. This not only wastes system resources but also reduces the system's adaptability to changing user needs. In addition, existing systems generally rely on traditional role-based authorization mechanisms for file security, with a relatively simple approach to access control. This makes it difficult to accurately assess the risks of user behavior and poses certain access security vulnerabilities. Furthermore, during the decompression and loading of files, there is a lack of dynamic anomaly analysis of user behavior, making it difficult to detect violations and take corresponding protective measures in a timely manner. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method and system for online management of digital archives based on big data, which solves the problems of removing redundant digital archive data, dynamic lifecycle management, and abnormal user access behavior determination.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides an online digital archive management method based on big data, comprising,

[0008] Collect digital archive data and upload it to the database. Use NLP technology to automatically generate metadata for the corresponding archives. Based on the metadata content, use a redundancy detection algorithm that combines semantic analysis to deduplicatize and compress the archive data.

[0009] The access frequency and timeliness of compressed archives are scored and graded using time series analysis. Based on the scoring and grading of archives, dynamic K-Means clustering algorithm is used to realize dynamic allocation of archives. Based on the dynamic allocation of archives, a life cycle management mechanism for archives is realized. Combined with a zero-trust model, access permissions for archives are controlled in a hierarchical manner.

[0010] The system uses streaming loading and decompression technology to dynamically decompress the file content selected by the user. During the decompression process, the LOF algorithm and CFSFDP algorithm are used to calculate the outlier and deviation values ​​of the user's behavior, and jointly determine the abnormal situation of the user's behavior.

[0011] As a preferred embodiment of the big data-based online digital archive management method of the present invention, the decompression process uses the LOF algorithm and the CFSFDP algorithm to calculate the abnormal value and deviation value of user behavior, and the joint determination of abnormal user behavior refers to binding the user identity and the decompression target file before each decompression task starts.

[0012] During the decompression process, user behavior is monitored in real time to obtain multi-dimensional features of the a-th user's operation behavior, and these multi-dimensional features are transformed into feature points for that user. And generate feature data streams of all user behaviors. ;

[0013] Use the LOF algorithm to calculate outliers for each user action. ;

[0014] The CFSFDP algorithm is used to calculate the density of feature points for all users in the feature data stream. ;

[0015] Calculate the separation degree between each feature point and points with higher density. ;

[0016] Set density threshold With separation threshold , will satisfy and The points are used as high-density center points to generate a high-density center point set, point;

[0017] Calculate feature points Deviation value from the center point ;

[0018] Set outlier threshold and deviation threshold ,like If the current user's operation is suspended and a behavior warning is sent, the user's exception log will be recorded; otherwise, no action will be taken.

[0019] Set a time limit and an upper limit for the number of exceptions. When a user triggers the maximum number of warnings within a time period, freeze the target file to be decompressed and the user's account permissions, and send an emergency warning to the administrator.

[0020] As a preferred embodiment of the online digital archive management method based on big data described in this invention, the method of deduplicating and compressing archive data using a redundancy detection algorithm combined with semantic analysis based on metadata content refers to calculating the TF-IDF value of each word in the title and integrating the TF-IDF values ​​into a title vector. ;

[0021] Calculate the similarity of the title vectors of any two files A and B. ;

[0022] Set title vector similarity threshold ,like Greater than or equal to If so, add file A and file B to the same file similarity group; otherwise, do not add them.

[0023] The text content of archives in similar groups is logically segmented into paragraphs, and the BERT model is used to generate a corresponding semantic vector for each segment. ;

[0024] Calculate the semantic vector similarity of each pair of segments between file A and file B in the file similarity group. ;

[0025] Set semantic vector threshold ,like Greater than or equal to If the nth paragraph in file A is similar to the qth paragraph in file B, then the two paragraphs are not similar.

[0026] For two paragraphs that are deemed similar, select one as the representative content, replace the content of the unselected paragraph with a structure that includes citation marks, and compress and save the processed file.

[0027] As a preferred embodiment of the big data-based online digital archive management method of the present invention, the step of scoring and classifying the access frequency and timeliness of compressed archives through time series analysis refers to collecting the access time data of any archive A and generating the access time series of archive A. ;

[0028] Set the analysis time granularity for time series. Resampling is performed to count the number of visits in each time unit, generating a discrete time series. ;

[0029] Calculate the access frequency score of archives based on discrete time series. and timeliness score ;

[0030] Based on the calculated access frequency score and timeliness score, a comprehensive score is calculated. The files are then categorized based on the overall score.

[0031] The archive-based rating and grading system uses a dynamic K-Means clustering algorithm to dynamically assign archives based on their access frequency. Timeliness score With overall score The standardization process is performed, and the standardized vectors are used as the feature dimensions to initialize the archive's feature vectors. ;

[0032] High-quality archives, ordinary archives, and low-quality archives are defined as the initial classification categories for the clustering algorithm. The centroids of the classification categories are calculated based on the feature vectors of the archives. ;

[0033] For each archive feature vector, calculate the Euclidean distance from the vector to the centroid of each category. ;

[0034] The feature vectors are assigned to the category corresponding to the nearest centroid, and the centroid of each category is recalculated based on the assignment results until the category assignment no longer changes.

[0035] As a preferred embodiment of the big data-based online digital archive management method of the present invention, the dynamic allocation of archives realizes the life cycle management mechanism of archives, and the access control of archives is hierarchically controlled by combining the zero trust model. This means that the life cycle of archives is divided into an active stage, a stable stage and an archiving stage, which correspond to high-priority archives, ordinary archives and low-priority archives of the archive category, respectively. The life cycle stage is determined according to the allocation category of the archives, and specific management actions are performed according to the life cycle stage.

[0036] Calculate a user's trust score using multidimensional features from the zero-trust model. Different access permissions are granted based on the user's credibility score.

[0037] As a preferred embodiment of the online digital archive management method based on big data described in this invention, the method of dynamically decompressing the archive content selected by the user using streaming loading and decompression technology refers to the following: after the user selects the target archive, the block size of each streaming processing is preset to Block_Size, and the actual block size of each streaming processing is calculated. Size of the file remaining after each streaming process ;

[0038] Loop through the file streaming size and the remaining file size until the remaining size is reached. This indicates that the file has been processed.

[0039] As a preferred embodiment of the big data-based online digital archive management method of the present invention, the following steps are included: collecting digital archive data and uploading it to the database, using NLP technology to automatically generate metadata for the corresponding archives, extracting plain text content from the archive data to obtain the text content of each archive, and using the sentence in the first paragraph of the text as a candidate sentence.

[0040] Using the TF-IDF algorithm in NLP technology, calculate words Frequency of occurrence in the text Harmony Frequency of occurrence in candidate sentences ;

[0041] Calculation words Scarcity in text ;

[0042] Calculate words based on word frequency and scarcity. TF-IDF value in text TF- Harmony TF-IDF value in candidate sentences ;

[0043] Text and candidate sentences By integrating the TF-IDF values ​​of the words, text vectors are obtained. and candidate sentence vectors ;

[0044] Calculate text vectors With candidate sentence vectors The cosine similarity is used to determine the similarity, and the sentence with the highest similarity is selected as the title from the candidate sentences.

[0045] Construct co-occurrence relationships for words, treating each word as a word node and setting an initial node score for each word node. If two words appear simultaneously in the same window, then establish an edge between them.

[0046] The TextRank algorithm from NLP is used to calculate the score for each word node. ;

[0047] When the maximum number of iterations is reached, the top m words with the highest scores are selected as keywords;

[0048] The extracted title, keywords, and timestamps are used as the archive's metadata.

[0049] Secondly, this invention provides an online digital archive management system based on big data, comprising:

[0050] The archive data acquisition module is used to collect the original content of archives and upload it to the database;

[0051] The automatic metadata generation module is used to extract structured information from archive content and generate corresponding metadata;

[0052] The data redundancy detection and deduplication compression module is used to identify redundant parts in archive data, and to deduplicat and compress the redundant content to optimize storage space.

[0053] The document access scoring and grading module is used to dynamically analyze the frequency and timeliness of document access and generate scores and grades for the documents.

[0054] The dynamic file allocation module is used to adjust the file classification based on access behavior and timeliness changes;

[0055] The archive lifecycle management module is used to manage the lifecycle of archives based on dynamic allocation results. It achieves full-process control of archive creation, active use, archiving and destruction through the classification and management of different lifecycles.

[0056] The zero-trust access control module is used to strictly control access permissions to files in a hierarchical manner.

[0057] The streaming loading and decompression module is used to dynamically decompress the contents of the file selected by the user, and uses streaming loading and decompression technology to decompress the file data on demand;

[0058] The behavior anomaly detection module is used to calculate the anomaly value and deviation value of user behavior using the LOF algorithm and CFSFDP algorithm during the file decompression process, and jointly determine the abnormal situation of user behavior.

[0059] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the online digital archive management method based on big data as described in the first aspect of the present invention.

[0060] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the online digital archive management method based on big data as described in the first aspect of the present invention.

[0061] The beneficial effects of this invention are as follows: By using natural language processing technology to generate archival metadata, it achieves the automatic extraction and construction of information such as archival titles and keywords; based on the TF-IDF algorithm and TextRank technology, unstructured archival data is transformed into structured metadata, fundamentally improving the efficiency of archival classification, retrieval, and management; employing a redundancy detection algorithm combined with semantic analysis, it effectively identifies content redundancy between archives and achieves efficient compression and storage of similar content through a reference structure, significantly reducing the space occupied by data storage; combining time series analysis and dynamic clustering methods, it realizes the scoring and grading of archival access frequency and timeliness, enabling the management system to flexibly respond to changes in archival access status; and it divides the archival lifecycle... Divided into active, stable, and archived phases, each bound to different resource management strategies, the system achieves efficient resource allocation and storage optimization. It calculates user behavior credibility and dynamically determines user access permissions, preventing unauthorized access to highly sensitive files by low-trust users and comprehensively improving the security of the file system. During file decompression, streaming decompression technology reduces decompression waiting time while avoiding resource overload issues that might occur from decompressing large files at once. Using LOF and CFSFDP algorithms, it accurately identifies abnormal user behavior and potential security risks, and uses a dynamic response mechanism to block malicious behavior and freeze permissions, effectively preventing security vulnerabilities in file operations and providing the system with dynamic and efficient user management capabilities. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of an online digital archive management method based on big data, as shown in Example 1.

[0064] Figure 2 This is a structural diagram of an online digital archive management system based on big data, as shown in Example 1.

[0065] Figure 3 This is a flowchart of an online digital archive management system based on big data, as shown in Example 1. Detailed Implementation

[0066] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0068] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0069] Example 1, referring to Figures 1-3 This is the first embodiment of the present invention, which provides an online digital archive management method based on big data, including the following steps:

[0070] S1: Collect digital archive data and upload it to the database. Use NLP technology to automatically generate metadata for the corresponding archives. Based on the metadata content, use a redundancy detection algorithm combined with semantic analysis to deduplicatize and compress the archive data.

[0071] Specifically, plain text content is extracted from the archive data to obtain the text content of each archive (Text), and the sentence in the first paragraph of the text is used as the candidate sentence (Fist).

[0072] Using the TF-IDF algorithm in NLP technology, calculate words Frequency of occurrence in the text :

[0073] ,

[0074] in, For the j-th word, For the i-th text content, For words In the text The number of times it appears in For text Total number of words;

[0075] Calculation words Frequency of occurrence in candidate sentences :

[0076] ,

[0077] in, For the candidate sentence corresponding to the i-th text content, For words In candidate sentences The number of times it appears in Candidate sentence Total number of words;

[0078] Calculation words Scarcity in text :

[0079] ,

[0080] Where I represents the total number of files, i.e., the total number of texts. For containing words The number of texts;

[0081] Calculate words based on word frequency and scarcity. The TF-IDF values ​​of the words in the text and candidate sentences. TF- in the text The value, specifically expressed as:

[0082] ,

[0083] Among them, TF- For evaluation words The value of importance in the text;

[0084] The word TF- in candidate sentences The value, specifically expressed as:

[0085] ,

[0086] Among them, TF- For evaluation words The value of importance in the candidate sentence;

[0087] Text The TF-IDF values ​​of all words in the text are combined to obtain the text vector. ;

[0088] Candidate sentences By integrating the TF-IDF values ​​of all words in the text, candidate sentence vectors are obtained. ;

[0089] Calculate text vectors With candidate sentence vectors Cosine similarity:

[0090] ,

[0091] in, For vector dot product, , The vector magnitude;

[0092] Select the sentence with the highest similarity from the candidate sentences as the title;

[0093] Construct co-occurrence relationships for words, treating each word as a word node and setting an initial node score for each word node. If two words appear simultaneously in the same window, then establish an edge between them.

[0094] The TextRank algorithm from NLP is used to calculate the score for each word node. :

[0095] ,

[0096] in, For words The rating, The damping factor set for simulating jump behavior. For the k-th word, For word nodes To the vocabulary node edge weights, For pointing words All word nodes, To start from the word node All edges sent out, For the l-th node, For vocabulary nodes The initial score;

[0097] When the maximum number of iterations is reached, the top m words with the highest scores for each word are selected as keywords.

[0098] The extracted title, keywords, and timestamps used to extract the title and keywords are used as the archive's metadata.

[0099] By extracting plain text content from archives and selecting sentences from the first paragraph as a candidate sentence set, the content processing flow is simplified, quickly transforming unstructured archive data into a semantically based content set. The TF-IDF algorithm is used to calculate the importance of words in both the text and candidate sentences, accurately reflecting word frequency and scarcity. Cosine similarity is used to select the candidate sentences with the highest similarity as titles, ensuring semantic relevance between the candidate sentences and the full text. A co-occurrence relationship network of words is constructed based on the TextRank algorithm, and the highest-scoring keywords are extracted after iterative scoring, improving the quality of keyword extraction and making the generated keywords more closely aligned with the core of the archive content. Automated metadata generation replaces the labor of repetitive manual annotation, significantly improving the efficiency of archive management.

[0100] Furthermore, the TF-IDF value of each word in the title is calculated, and the TF-IDF values ​​are integrated into a title vector. ;

[0101] Calculate the similarity of the title vectors of any two files A and B. :

[0102] ,

[0103] in, and Let A and B be the title vectors. and Let the length of the title vector of file A and file B be the modulus.

[0104] Set title vector similarity threshold ,like Greater than or equal to If so, add file A and file B to the same file similarity group; otherwise, do not add them.

[0105] The text content of archives in similar groups is logically segmented into paragraphs, and the BERT model is used to generate a corresponding semantic vector for each segment. :

[0106] ,

[0107] in, For the nth segment of the text content, This is the semantic vector corresponding to the nth segment. This refers to a language model that can generate semantic vectors in existing technologies.

[0108] Calculate the semantic vector similarity of each pair of segments between file A and file B in the file similarity group. ,

[0109] ,

[0110] in, Let be the semantic vector of the nth paragraph in file A. Let be the semantic vector of the q-th paragraph in file B. and This represents the magnitude of the corresponding vector;

[0111] Set semantic vector threshold ,like Greater than or equal to If the nth paragraph in file A is similar to the qth paragraph in file B, then the two paragraphs are not similar.

[0112] For two paragraphs that are deemed similar, one is selected as the representative content. The content of the unselected paragraph is replaced with a structure containing citation marks. The paragraph with citation marks as the representative content is then compressed and saved.

[0113] By using the TF-IDF algorithm to generate title vectors and calculating title vector similarity for initial screening, the scope of archives requiring further in-depth analysis is reduced, thus improving processing speed. For similar archive groups, the BERT model is used to generate semantic vectors for text segments, and segment semantic similarity is calculated, thereby achieving semantic-level archive content redundancy detection and accurately identifying semantically repetitive paragraphs. Only one of the paragraphs judged to be similar is retained as representative content, and the unselected paragraphs are replaced with citation marks, completing compressed storage. This significantly reduces archive storage redundancy and space occupation, while maintaining the integrity of archive access and the clarity of citation relationships.

[0114] S2: The access frequency and timeliness of compressed archives are scored and graded using time series analysis. Based on the scoring and grading of archives, dynamic K-Means clustering algorithm is used to realize dynamic allocation of archives. Based on the dynamic allocation of archives, a life cycle management mechanism for archives is realized. Combined with a zero-trust model, access permissions for archives are controlled in a hierarchical manner.

[0115] Specifically, collect the access time data of any file A, and generate the access time series of file A. ;

[0116] Set the analysis time granularity for time series. Resampling is performed to count the number of visits in each time unit, generating a discrete time series. :

[0117] ,

[0118] in, This refers to a time series that has z time periods.

[0119] Calculate the access frequency score of archives based on discrete time series. :

[0120] ,

[0121] in, Let be the number of times the device is accessed in the s-th time period. The score represents the frequency of access to file A within a given time period, and B represents any other file B belonging to the total number of files I.

[0122] Timeliness score of archives calculated based on discrete time series. :

[0123] ,

[0124] in, For the current time, This is the time at the end of the s-th time interval. A time-sensitivity decay factor set to control the sensitivity of file access over time;

[0125] Based on the calculated access frequency score and timeliness score, a comprehensive score is calculated. :

[0126] ,

[0127] in, The weighting coefficients set to indicate the sensitivity of the importance of the rating results to the frequency of file access. Weighting coefficients are set to indicate the sensitivity of the timeliness of document access to the importance of the scoring results;

[0128] Set a comprehensive score threshold. and ,and ;

[0129] like This indicates that the files are accessed frequently and are very popular, so the files are classified as high-quality files.

[0130] like This indicates that the files are accessed at a moderate frequency and have a stable level of popularity, so the files are classified as ordinary files.

[0131] like This indicates that the files are accessed infrequently and have low popularity, so the files are classified as low-quality files.

[0132] The archive-based rating and grading system uses a dynamic K-Means clustering algorithm to dynamically assign archives based on their access frequency. Timeliness score With overall score The standardization process is performed, and the standardized vectors are used as the feature dimensions to initialize the archive's feature vectors. :

[0133] ,

[0134] High-quality archives, ordinary archives, and low-quality archives are defined as the initial classification categories for the clustering algorithm. The centroids of the classification categories are calculated based on the feature vectors of the archives. :

[0135] ,

[0136] in, For the r-th category, These correspond to high-priority, ordinary, and low-priority archives, respectively. Let r be the centroid of the r-th category. Let r be the number of files in the r-th category;

[0137] For each archive feature vector, calculate the Euclidean distance from the vector to the centroid of each category. :

[0138] ,

[0139] in, For feature vectors to the center of mass The Euclidean distance, where e is the e-th dimension. These correspond to the access frequency score, timeliness score, and overall score, respectively, where E is the total number of dimensions. For feature vectors The value in the e-th dimension Center of mass The value in the e-th dimension;

[0140] The feature vectors are assigned to the category corresponding to the nearest centroid, and the centroid of each category is recalculated based on the assignment results until the category assignment no longer changes.

[0141] By collecting access time data from archives, generating time series data, and resampling, access frequency scores and timeliness scores are calculated for the archives. This effectively quantifies the activity and popularity fluctuations of the archives, making their access characteristics clearer and more intuitive. Based on the scoring results, a comprehensive score for the archives is calculated, and classification rules are set to divide the archives into high-quality archives, ordinary archives, and low-quality archives, achieving the management goal of precise classification of archives at different levels. The standardized scoring results are then used as feature vectors to input into a dynamic K-Means clustering algorithm. The classification results are dynamically adjusted based on the feature vectors, enabling the archive classification to adapt to changes in access conditions in real time, thus improving the flexibility and accuracy of the classification.

[0142] Furthermore, the life cycle of archives is divided into an active stage, a stable stage, and an archiving stage, which correspond to high-priority archives, ordinary archives, and low-priority archives, respectively. The life cycle stage is determined according to the allocation category of the archives, and specific management actions are performed according to the life cycle stage.

[0143] The specific management actions refer to storing active files in high-speed storage devices, prohibiting any form of cleanup operation, storing stable files in ordinary storage devices, periodically deleting redundant files, cold storing files in the archiving stage, and thoroughly deleting files that have exceeded the set retention period.

[0144] Calculate a user's trust score using multidimensional features from the zero-trust model. :

[0145] ,

[0146] in, , , , Feature weights are set to reflect the importance of features in the credibility score. The user's identity credibility is determined by a value of 1 if verification is successful, and 0 otherwise. The device trustworthiness score is 1 for devices on the whitelist and 0 for others. To determine the reliability of access time, an access score of 1 is assigned if the access occurs within a specified time frame, and 0 otherwise. The network trust level is set to 1 if using the internal network, and 0 otherwise.

[0147] Set a confidence score threshold and ;

[0148] like Access permissions are granted to high-priority and ordinary archives, while low-priority archives are allowed to be read-only.

[0149] like Access is granted to high-priority files, read-only access is allowed to ordinary files, and access is denied to low-priority files.

[0150] like No files can be accessed.

[0151] By dividing the archive lifecycle into active, stable, and archiving stages, and combining this with dynamic allocation of archive categories to execute corresponding management actions, the system improves storage resource utilization and response speed. Based on a zero-trust model, the system dynamically calculates user trust scores and manages user access permissions in a tiered manner according to archive categories, effectively preventing unauthorized access to sensitive archives by low-trust users and the risk of data leakage, thereby enhancing the system's security and intelligence.

[0152] S3: The streaming loading and decompression technology is used to dynamically decompress the file content selected by the user. During the decompression process, the LOF algorithm and CFSFDP algorithm are used to calculate the outlier and deviation values ​​of the user's behavior, and the abnormal situation of the user's behavior is jointly determined.

[0153] Specifically, after the user selects the target file, the default block size for each streaming process is Block_Size, and the actual block size for each streaming process is calculated. :

[0154] ,

[0155] in, Let x be the block size for the x-th streaming process. This represents the remaining size of the file after the (x-1)th streaming process.

[0156] Calculate the remaining file size after each streaming process. :

[0157] ,

[0158] Loop through the file streaming size and the remaining file size until the remaining size is reached. This indicates that the file has been processed.

[0159] By dynamically loading and decompressing the target file selected by the user in chunks, calculating the decompressed size of the current chunk and the remaining size of the file in real time, and looping until the process is complete, this method of loading and decompressing simultaneously achieves dynamic and segmented decompression. It avoids memory overflow or system performance degradation problems that may be caused by loading the entire file at once. Furthermore, streaming decompression allows users to preview and operate on parts of the content after decompression is complete, significantly reducing the overall operation waiting time and improving the user experience.

[0160] Furthermore, before each decompression task begins, the user identity performing the decompression operation and the target file for decompression are bound together;

[0161] During the decompression process, user behavior is monitored in real time to obtain multi-dimensional features of the a-th user's operation behavior, and these multi-dimensional features are transformed into feature points for that user. And generate feature data streams of all user behaviors. :

[0162] ,

[0163] in, This refers to a total of w users;

[0164] Use the LOF algorithm to calculate outliers for each user action. :

[0165] ,

[0166] in, For feature points The set of nearest neighbor feature points, For feature points Feature points in the nearest neighbor feature point set Let the feature point of user a be the Euclidean average distance from its nearest neighbor, i.e., the feature point... Local density, For feature points Local density, For feature points Average density of nearest neighbors;

[0167] The CFSFDP algorithm is used to calculate the density of feature points for all users in the feature data stream. :

[0168] ,

[0169] in, Let a be the density of the a-th user in the feature data stream. For point With point The Euclidean distance between them These are the distance stage coefficients obtained through distance feature extraction, used to measure the relative distance between the target point and the reference point;

[0170] Calculate the separation degree between each feature point and points with higher density. :

[0171] ,

[0172] Set density threshold With separation threshold , will satisfy and The points are used as high-density center points to generate a high-density center point set, point;

[0173] Calculate feature points Deviation value from the center point :

[0174] ,

[0175] in, Let h be the h-th center point in the set of center points, and t be the number of center points in the set of center points. For feature points The Euclidean distance between the h-th center point;

[0176] Set outlier threshold and deviation threshold ,like If the current user's operation is suspended and a behavior warning is sent, the user's exception log will be recorded; otherwise, no action will be taken.

[0177] Set a time limit and an upper limit for the number of exceptions. When a user triggers the maximum number of warnings within a time period, freeze the target file to be decompressed and the user's account permissions, and send an emergency warning to the administrator.

[0178] By binding the decompression operation to the user's identity and the target file, user behavior can be directly linked to the operation target, ensuring the traceability of file operations. A user behavior feature data stream is constructed to provide continuous and dynamic feature descriptions for subsequent user behavior analysis. The LOF algorithm is used to calculate the local outlier degree of user behavior features in the feature data stream, and the CFSFDP algorithm is used to calculate the global deviation of user behavior features. Joint judgment of local and global factors avoids misjudgments and omissions. Thresholds are set to assess the threat level of user operations at multiple levels and issue warnings, achieving security monitoring and real-time protection during the decompression process. When the number of warnings triggered by a user reaches the upper limit within a unit of time, the target file and user account are directly frozen, effectively solving the problem of dealing with high-frequency threats.

[0179] This embodiment also provides an online digital archive management system based on big data, including:

[0180] The archive data acquisition module is used to collect the original content of archives and upload it to the database;

[0181] The automatic metadata generation module is used to extract structured information from archive content and generate corresponding metadata;

[0182] The data redundancy detection and deduplication compression module is used to identify redundant parts in archive data, and to deduplicat and compress the redundant content to optimize storage space.

[0183] The document access scoring and grading module is used to dynamically analyze the frequency and timeliness of document access and generate scores and grades for the documents.

[0184] The dynamic file allocation module is used to adjust the file classification based on access behavior and timeliness changes;

[0185] The archive lifecycle management module is used to manage the lifecycle of archives based on dynamic allocation results. It achieves full-process control of archive creation, active use, archiving and destruction through the classification and management of different lifecycles.

[0186] The zero-trust access control module is used to strictly control access permissions to files in a hierarchical manner.

[0187] The streaming loading and decompression module is used to dynamically decompress the contents of the file selected by the user, and uses streaming loading and decompression technology to decompress the file data on demand;

[0188] The behavior anomaly detection module is used to calculate the anomaly value and deviation value of user behavior using the LOF algorithm and CFSFDP algorithm during the file decompression process, and jointly determine the abnormal situation of user behavior.

[0189] This embodiment also provides a computer device applicable to the online management method of digital archives based on big data, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the online management method of digital archives based on big data as proposed in the above embodiment.

[0190] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0191] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the online digital archive management method based on big data as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0192] In summary, this invention generates archival metadata using natural language processing technology, enabling the automatic extraction and construction of information such as archival titles and keywords; it transforms unstructured archival data into structured metadata based on the TF-IDF algorithm and TextRank technology, fundamentally improving the efficiency of archival classification, retrieval, and management; it employs a redundancy detection algorithm combined with semantic analysis to effectively identify content redundancy among archives, and achieves efficient compression and storage of similar content through a reference structure, significantly reducing data storage space usage; it combines time series analysis and dynamic clustering methods to achieve scoring and grading of archival access frequency and timeliness, enabling the management system to flexibly respond to changes in archival access status; and it divides the archival lifecycle. Different resource management strategies are applied to the active, stable, and archiving phases, achieving efficient resource allocation and storage optimization. User behavior credibility is calculated to dynamically determine user access permissions, preventing unauthorized access to highly sensitive files by low-trust users and comprehensively improving the security of the archive system. During file decompression, streaming decompression technology reduces decompression waiting time while avoiding resource overload issues that may occur when decompressing large files at once. LOF and CFSFDP algorithms are used to accurately identify abnormal user behavior and potential security risks, and a dynamic response mechanism blocks malicious behavior and freezes permissions, effectively preventing security vulnerabilities in file operations and providing the system with dynamic and efficient user management capabilities.

[0193] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for online management of digital archives based on big data, characterized in that: include, Collect digital archive data and upload it to the database. Use NLP technology to automatically generate metadata for the corresponding archives. Based on the metadata content, use a redundancy detection algorithm that combines semantic analysis to deduplicatize and compress the archive data. The access frequency and timeliness of compressed archives are scored and graded using time series analysis. Based on the scoring and grading of archives, dynamic K-Means clustering algorithm is used to realize dynamic allocation of archives. Based on the dynamic allocation of archives, a life cycle management mechanism for archives is realized. Combined with a zero-trust model, access permissions for archives are controlled in a hierarchical manner. The system uses streaming loading and decompression technology to dynamically decompress the file content selected by the user. During the decompression process, the LOF algorithm and CFSFDP algorithm are used to calculate the outlier and deviation values ​​of the user's behavior, and jointly determine the abnormal situation of the user's behavior. During the decompression process, the LOF algorithm and CFSFDP algorithm are used to calculate the outlier and deviation values ​​of user behavior. The joint determination of abnormal user behavior refers to binding the user identity and the target file to be decompressed before each decompression task begins. During the decompression process, user behavior is monitored in real time to obtain multi-dimensional features of the a-th user's operation behavior, and these multi-dimensional features are transformed into feature points for that user. And generate feature data streams of all user behaviors. ; Use the LOF algorithm to calculate outliers for each user action. ; The CFSFDP algorithm is used to calculate the density of feature points for all users in the feature data stream. ; Calculate the separation degree between each feature point and points with higher density. ; Set density threshold With separation threshold , will satisfy and The points are used as high-density center points to generate a high-density center point set, point; Calculate feature points Deviation value from the center point ; Set outlier threshold and deviation threshold ,like If the current user's operation is suspended and a behavior warning is sent, the user's exception log will be recorded; otherwise, no action will be taken. Set a time limit and an upper limit for the number of exceptions. When a user triggers the maximum number of warnings within a time period, freeze the target file to be decompressed and the user's account permissions, and send an emergency warning to the administrator.

2. The online digital archive management method based on big data as described in claim 1, characterized in that: The deduplication and compression of archive data based on metadata content using a redundancy detection algorithm combined with semantic analysis involves calculating the TF-IDF value of each word in the title and integrating the TF-IDF values ​​into a title vector. ; Calculate the similarity of the title vectors of any two files A and B. ; Set title vector similarity threshold ,like Greater than or equal to If so, add file A and file B to the same file similarity group; otherwise, do not add them. The text content of archives in similar groups is logically segmented into paragraphs, and the BERT model is used to generate a corresponding semantic vector for each segment. ; Calculate the semantic vector similarity of each pair of segments between file A and file B in the file similarity group. ; Set semantic vector threshold ,like Greater than or equal to If the nth paragraph in file A is similar to the qth paragraph in file B, then the two paragraphs are not similar. For two paragraphs that are deemed similar, select one as the representative content, replace the content of the unselected paragraph with a structure that includes citation marks, and compress and save the processed file.

3. The online digital archive management method based on big data as described in claim 2, characterized in that: The method of scoring and classifying the access frequency and timeliness of compressed archives through time series analysis refers to collecting access time data of any archive A and generating an access time series of archive A. ; Set the analysis time granularity for time series. Resampling is performed to count the number of visits in each time unit, generating a discrete time series. ; Calculate the access frequency score of archives based on discrete time series. and timeliness score ; Based on the calculated access frequency score and timeliness score, a comprehensive score is calculated. The files are then categorized based on the overall score. The archive-based rating and grading system uses a dynamic K-Means clustering algorithm to dynamically assign archives based on their access frequency. Timeliness score With overall score The standardization process is performed, and the standardized vectors are used as the feature dimensions to initialize the archive's feature vectors. ; High-quality archives, ordinary archives, and low-quality archives are defined as the initial classification categories for the clustering algorithm. The centroids of the classification categories are calculated based on the feature vectors of the archives. ; For each archive feature vector, calculate the Euclidean distance from the vector to the centroid of each category. ; The feature vectors are assigned to the category corresponding to the nearest centroid, and the centroid of each category is recalculated based on the assignment results until the category assignment no longer changes.

4. The online digital archive management method based on big data as described in claim 3, characterized in that: The dynamic allocation of archives enables the lifecycle management mechanism of archives. Combined with the zero-trust model, the access rights of archives are controlled hierarchically. The lifecycle of archives is divided into an active stage, a stable stage, and an archiving stage, which correspond to high-priority archives, ordinary archives, and low-priority archives, respectively. The lifecycle stage is determined according to the allocation category of the archives, and specific management actions are performed according to the lifecycle stage. Calculate a user's trust score using multidimensional features from the zero-trust model. Different access permissions are granted based on the user's credibility score.

5. The online digital archive management method based on big data as described in claim 4, characterized in that: The use of streaming loading and decompression technology to dynamically decompress the content of the file selected by the user refers to the process where, after the user selects the target file, the block size for each streaming process is preset to Block_Size, and the actual block size for each streaming process is calculated. Size of the file remaining after each streaming process ; Loop through the file streaming size and the remaining file size until the remaining size is reached. This indicates that the file has been processed.

6. The online digital archive management method based on big data as described in claim 5, characterized in that: The process of collecting digital archive data and uploading it to the database, and using NLP technology to automatically generate metadata for the corresponding archives, refers to extracting plain text content from the archive data to obtain the text content of each archive, and using the sentence in the first paragraph of the text as a candidate sentence. Using the TF-IDF algorithm in NLP technology, calculate words Frequency of occurrence in the text Harmony Frequency of occurrence in candidate sentences ; Calculation words Scarcity in text ; Calculate words based on word frequency and scarcity. TF-IDF value in text TF- Harmony TF-IDF value in candidate sentences ; Text and candidate sentences By integrating the TF-IDF values ​​of the words, text vectors are obtained. and candidate sentence vectors ; Calculate text vectors With candidate sentence vectors The cosine similarity is used to determine the similarity, and the sentence with the highest similarity is selected as the title from the candidate sentences. Construct co-occurrence relationships for words, treating each word as a word node and setting an initial node score for each word node. If two words appear simultaneously in the same window, then establish an edge between them. The TextRank algorithm from NLP is used to calculate the score for each word node. ; When the maximum number of iterations is reached, the top m words with the highest scores are selected as keywords; The extracted title, keywords, and timestamps are used as the archive's metadata.

7. A big data-based online digital archive management system, based on the big data-based online digital archive management method according to any one of claims 1 to 6, characterized in that: include, The archive data acquisition module is used to collect the original content of archives and upload it to the database; The automatic metadata generation module is used to extract structured information from archive content and generate corresponding metadata; The data redundancy detection and deduplication compression module is used to identify redundant parts in archive data, and to deduplicat and compress the redundant content to optimize storage space. The document access scoring and grading module is used to dynamically analyze the frequency and timeliness of document access and generate scores and grades for the documents. The dynamic file allocation module is used to adjust the file classification based on access behavior and timeliness changes; The archive lifecycle management module is used to manage the lifecycle of archives based on dynamic allocation results. It achieves full-process control of archive creation, active use, archiving and destruction through the classification and management of different lifecycles. The zero-trust access control module is used to strictly control access permissions to files in a hierarchical manner. The streaming loading and decompression module is used to dynamically decompress the contents of the file selected by the user, and uses streaming loading and decompression technology to decompress the file data on demand; The behavior anomaly detection module is used to calculate the anomaly value and deviation value of user behavior using the LOF algorithm and CFSFDP algorithm during the file decompression process, and jointly determine the abnormal situation of user behavior.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the online digital archive management method based on big data as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the online digital archive management method based on big data as described in any one of claims 1 to 6.

Images