A digital archive management method and system based on big data

Abstract

The present invention relates to the field of archive management technology, and in particular to a digital archive management method and system based on big data. A digital archive management method based on big data, comprising the following steps: S1: obtaining the query keyword of the target user, querying the archive according to the query keyword, and taking the relevant archive as a sequence of archives to be processed; S2: obtaining the target user's required format and permission data, and dividing the sequence of archives to be processed into a first query archive sequence and a second query archive sequence according to the target user's required format. The present invention realizes the backup of each archive format, and uses a forgetting adjustment formula to forget the uncommon part of the archive content in each archive format; by analyzing the similarity of the content in different archive formats and the number of archive formats, the storage space can be saved while retaining the content of different archive formats.

Description

Technical Field

[0001] The present invention relates to the technical field of archive management, and in particular to a digital archive management method and system based on big data. Background Art

[0002] With the development of the information society, archive management is gradually transforming towards digitalization and intelligence. Traditional archive management methods usually rely on paper documents or centralized databases for storage and management. Although they meet basic needs, query efficiency and storage space management have become major issues when faced with large amounts of data and diverse archive formats. The current digital archive management system has the following problems: due to the redundant storage of archive data, especially when distributed storage has backups, it is difficult for the system to effectively manage the content of multiple backup data, resulting in a waste of storage space; in a big data environment, traditional archive query methods are difficult to cope with the rapid retrieval of massive information, and the user experience deteriorates. Summary of the invention

[0003] In order to overcome the disadvantage of wasting storage space, the present invention provides a digital archive management method and system based on big data.

[0004] The technical solution of the present invention is: a digital archive management method based on big data, comprising the following steps:

[0005] S1: Obtain the query keyword of the target user, query the archives according to the query keyword, and take the relevant archives as the archive sequence to be processed;

[0006] S2: Obtain the target user's demand format and permission data, and divide the to-be-processed archive sequence into a first query archive sequence and a second query archive sequence according to the target user's demand format;

[0007] S3: adjusting the query quantity of the first query archive sequence or the second query archive sequence based on the authority data of the target user to obtain the adjusted query archive quantity, and converting the summary content or the archive content into a format and distributing the formatted content based on the summary content selected by the target user for query;

[0008] S4: Based on the converted archive data and the original archive data in distributed storage, use the forgetting adjustment formula to make adjustments, and return the recommended query results to the target user based on the adjusted archive data.

[0009] Preferably, the method of obtaining the target user's requirement format and permission data and dividing the to-be-processed archive sequence into a first query archive sequence and a second query archive sequence according to the target user's requirement format includes: taking the archives in the to-be-processed archive sequence that are different from the target user's requirement format as the first query archive sequence, and taking the archives in the to-be-processed archive sequence that are the same as the target user's requirement format as the second query archive sequence.

[0010] Preferably, the step of adjusting the query quantity of the first query archive sequence or the second query archive sequence based on the authority data of the target user to obtain the adjusted query archive quantity comprises: using a quantity adjustment formula according to the authority data of the target user to obtain the adjusted query archive quantity; when the adjusted query archive quantity is less than the quantity of the second query archive sequence, selecting archives of the adjusted query archive quantity from the second query archive sequence; when the adjusted query archive quantity is greater than or equal to the quantity of the second query archive sequence, supplementing the archives from the first query archive sequence, wherein the quantity adjustment formula is:

[0011] Q total =Q origina (1+μM);

[0012] In the formula, Q total is the number of query files after adjustment, Q origina is the number of initial query files, μ is the adjustment coefficient, and M is the adjustment factor related to the target user's permissions.

[0013] Preferably, the summary content or archive content of the archive is format-converted and then distributedly stored based on the summary content selected by the target user for query, including: after the summary content in the first query archive sequence is format-converted, according to the adjustment of the query archive quantity, a query result with a maximum number of summary contents returned to the target user is obtained, the archive content corresponding to the summary content selected by the target user for query is format-converted, and the converted summary content and archive content and the unconverted summary content and archive content are distributedly stored.

[0014] Preferably, the conversion archive data and the original archive data stored in distributed manner are adjusted using a forgetting adjustment formula, and a recommended query result is returned to the target user based on the adjusted archive data, including: using the forgetting adjustment formula to obtain a forgetting adjustment value of the conversion archive content or the original archive content, forgetting a portion of the archive content whose forgetting adjustment value is greater than or equal to a first preset threshold, and distributively storing the remaining portion of the archive content, wherein the forgetting adjustment formula is:

[0015] S = αw + βt;

[0016] Where S is the forgetting adjustment value, α and β are adjustment parameters, w is the similarity between the corresponding parts of the transformed file and the original file, and t is the number of views of the corresponding parts of the transformed file or the original file.

[0017] Preferably, before using the forgetting adjustment formula to obtain the forgetting adjustment value of the converted archive data or the original archive data, the method includes: using the FIFO algorithm to obtain the historical query archive sequence of each converted archive format and the original archive format, and performing weighted summation on each archive format in the historical query archive sequence to obtain the converted archive format weight and the original archive format weight, and using the forgetting adjustment formula to forget the archive format with the smallest weight; when the converted archive format weight is the same as the original archive format weight, forgetting the archive format that entered the historical query archive sequence earliest; wherein at least one complete archive is retained for each archive.

[0018] Preferably, before using the forgetting adjustment formula to obtain the forgetting adjustment value of the converted archive data or the original archive data, the method includes: decomposing the converted archive or the original archive to obtain partial content of each archive, and inputting the partial content of each archive into a similarity acquisition model to obtain the similarity between the converted archive and the corresponding partial content of the original archive, wherein the similarity acquisition model uses a BERT model, and before inputting the partial content of each archive into the similarity acquisition model, the text content is preprocessed by removing punctuation marks, stop words, and converting it to lowercase, and the preprocessed text content is converted into an embedding vector, and the cosine similarity formula is used to obtain the similarity between the converted archive and the corresponding partial content of the original archive.

[0019] Preferably, returning the recommended query result to the target user based on the adjusted archive data includes: using the final search formula to obtain the search value of the archive data, and returning the recommended query result to the target user for the archive data whose search value is greater than a second preset threshold.

[0020] Preferably, the method of using the final search formula to obtain the search value of the archive data includes: the final search formula is:

[0021] R = γe n +δln(εb+1);

[0022] Where R is the retrieval value of the archive data, γ, δ, and ε are the retrieval value adjustment parameters, n is the number of archive formats corresponding to each archive, and b is the number of keywords that the summary content of each archive meets the search requirements of the target user.

[0023] Preferably, a digital archive management system based on big data includes:

[0024] The user query module is used to obtain the query keywords of the target user and query the archives to form a sequence of archives to be processed;

[0025] The requirement format and authority management module is used to adjust the archive sequence to be processed according to the requirement format and authority data of the target user, and divide it into a first query archive sequence and a second query archive sequence;

[0026] The query adjustment and file format conversion module is used to adjust the query quantity according to the target user's permission data, convert the summary content or file content of the file, and perform distributed storage;

[0027] A distributed storage and management module is used to store the converted archive data and the original archive data in a distributed manner;

[0028] A forgetting adjustment module, used for performing forgetting adjustment on the stored archive data using a forgetting adjustment formula;

[0029] A similarity calculation module is used to calculate the similarity of the archive content using the similarity acquisition model, based on the similarity between the user's query history and the archive content;

[0030] The query recommendation module is used to adjust the order of archive recommendation and recommend archives with a large number of archive formats.

[0031] The beneficial effect is that the present invention determines which parts of the content should be forgotten by analyzing the similarity of the content in different file formats and the number of file formats. In the file management system, while retaining the complete content of one file format, delete the parts with high similarity or large number of views in other file formats, ensuring that the content of different file formats is retained while also greatly saving storage space;

[0032] Through the careful management of user authority data, the present invention ensures that users can only access the archive content that they have the authority to view, thus avoiding the risk of sensitive information leakage;

[0033] By using the final search formula, by allowing users to click on archives with multiple archive formats, the number of archive formats retained is reduced, and at least one completed archive content is retained while saving storage space;

[0034] The present invention uses distributed storage and intelligent adjustment mechanisms to efficiently store archive contents in different formats and ensure efficient response speed when inquiring. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a digital archive management method based on big data of the present invention;

[0036] Figure 2 This is a flow chart of a digital archive management system based on big data of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Embodiment 1: A digital archive management method based on big data, such as Figure 1-2 As shown, the following steps are included:

[0039] S1: Obtain the query keyword of the target user, query the archives according to the query keyword, and take the relevant archives as the archive sequence to be processed;

[0040] The files in the to-be-processed file sequence that are in a different format from the target user's required format are used as a first query file sequence, and the files in the to-be-processed file sequence that are in the same format as the target user's required format are used as a second query file sequence.

[0041] It should be noted that in a digital archive management system, archives are usually stored in multiple formats (for example, PDF, Word, Excel). Different users have different requirements for archive formats. For example, one user needs archives in PDF format, while another user needs archives in Excel format. In order to meet these requirements, the system will first classify the archives to be processed, the first query archive sequence: the format of these archives is different from the format required by the user. For example, if the user needs PDF format, but some archives are in Word format, then these Word format archives will be classified as the first query archive sequence; the second query archive sequence: the format of these archives is consistent with the user's requirements. For example, if the user needs PDF format and some archives are already in PDF format, then these archives will be classified as the second query archive sequence.

[0042] S2: Obtain the target user's demand format and permission data, and divide the to-be-processed archive sequence into a first query archive sequence and a second query archive sequence according to the target user's demand format;

[0043] According to the target user's authority data, the number of query files is adjusted using a quantity adjustment formula. When the number of adjusted query files is less than the number of the second query file sequence, files with the adjusted query file number are selected from the second query file sequence; when the number of adjusted query files is greater than or equal to the number of the second query file sequence, they are supplemented from the first query file sequence, wherein the quantity adjustment formula is:

[0044] Q total =Qorigina (1+μM);

[0045] In the formula, Q total is the number of query files after adjustment, Q origina is the number of initial query files, μ is the adjustment coefficient, and M is the adjustment factor related to the target user's permissions.

[0046] It should be noted that the system determines the number of files to be queried by using the user's authority data. Assuming that the user has high authority (such as administrator authority), the system allows the user to view more files, while users with low authority can only view fewer files. The number of files that should be returned is calculated through the quantity adjustment formula. The adjustment coefficient and authority factor are used in the formula to adjust the number of query results according to the user's authority. When the adjusted query number is less than the number of the second query file sequence: select enough files from the second query file sequence (files whose format matches the user's requirements); when the adjusted query number is greater than or equal to the number of the second query file sequence: supplement from the first query file sequence (files whose format does not match the user's requirements) until the adjusted query number is reached.

[0047] S3: adjusting the query quantity of the first query archive sequence or the second query archive sequence based on the authority data of the target user to obtain the adjusted query archive quantity, and converting the summary content or the archive content into a format and distributing the formatted content based on the summary content selected by the target user for query;

[0048] After the summary contents in the first query archive sequence are formatted, a query result with the maximum number of summary contents returned to the target user is obtained according to the number of adjusted query archives, and the archive contents corresponding to the summary contents selected for query by the target user are formatted, and the converted summary contents, archive contents and unconverted summary contents and archive contents are distributedly stored.

[0049] It should be noted that when a user searches for archives, the system first displays the summary content of the relevant archives, and the user chooses to view the detailed archive corresponding to a certain summary content. Based on the user's choice, the system will convert the summary content and archive content into the format required by the user (for example, PDF to Word, or other formats), and then return the archive results required by the user based on the adjusted query quantity. In addition, in order to facilitate large-scale storage, all converted summaries and archive contents will be stored in a distributed manner.

[0050] S4: Based on the converted archive data and the original archive data in distributed storage, use the forgetting adjustment formula to make adjustments, and return the recommended query results to the target user based on the adjusted archive data.

[0051] The forgetting adjustment formula is used to obtain the forgetting adjustment value of the converted archive content or the original archive content, and the archive content whose forgetting adjustment value is greater than or equal to the first preset threshold is forgotten, and the remaining archive content is distributedly stored, wherein the forgetting adjustment formula is:

[0052] S = αw + βt;

[0053] Where S is the forgetting adjustment value, α and β are adjustment parameters, w is the similarity between the corresponding parts of the transformed file and the original file, and t is the number of views of the corresponding parts of the transformed file or the original file.

[0054] It should be noted that if the calculated forgetting adjustment value is greater than or equal to the first preset threshold, the part of the content will be forgotten, thereby reducing the storage space occupied; by analyzing the similarity and viewing times of the archive content, it is decided which content to retain and which content to forget. The archive parts with high similarity and frequent viewing will be marked as redundant and deleted; the remaining important content will be retained and distributedly stored. This method can effectively save storage space and ensure that only the archive content most needed by users is retained, while retaining those archive parts with poor similarity, so that relevant personnel have the ability to correct erroneous archives.

[0055] A FIFO algorithm is used to obtain a historical query archive sequence of each converted archive format and the original archive format, and a weighted sum is performed on each archive format in the historical query archive sequence to obtain the converted archive format weight and the original archive format weight. The archive format with the smallest weight is forgotten using a forgetting adjustment formula; when the converted archive format weight is the same as the original archive format weight, the archive format that enters the historical query archive sequence earliest is forgotten; and at least one complete archive is retained for each archive.

[0056] It should be noted that the system will record the access history of each archive format when the user queries the archives, and sort these archives using the FIFO algorithm. The system will determine which archive was queried first (the oldest archive) and which archive was queried most recently (the newest archive) based on the order in which the archives were queried; the system will perform a weighted summation on each archive format in the historical query archive sequence and calculate the weight of each format. In order to ensure the integrity of the archives, each archive will retain at least one complete archive. Even if the content of some formats is forgotten, the core data will still be retained.

[0057] The conversion archive or the original archive is decomposed to obtain partial contents of each archive, and the partial contents of each archive are input into a similarity acquisition model to obtain the similarity between the conversion archive and the corresponding partial contents of the original archive, wherein the similarity acquisition model uses a BERT model, and before the partial contents of each archive are input into the similarity acquisition model, the text content is preprocessed by removing punctuation marks, stop words, and converting to lowercase, and the preprocessed text content is converted into an embedding vector, and the cosine similarity formula is used to obtain the similarity between the conversion archive and the corresponding partial contents of the original archive.

[0058] It should be noted that the system decomposes the content of the archive and extracts various parts; for example, the division of summary, theme, ending, or division by chapters, paragraphs, sentences in paragraphs, and other logical methods. The system uses the BERT model to calculate the similarity between the converted archive and the contents of each part of the original archive. The BERT model is a deep learning model used for natural language processing tasks, especially the calculation of text similarity.

[0059] The final search formula is used to obtain the search value of the archive data, and the archive data with the search value greater than the second preset threshold is returned to the target user as a recommended query result.

[0060] It should be noted that after the content of the archive has undergone format conversion, similarity calculation and forgetting adjustment, the final adjusted archive data will be the data stored by the system with optimized archive content; based on these adjusted archive data, the system uses the final retrieval formula to calculate the retrieval value of each archive, and the retrieval value reflects the relevance of the archive to the user's query needs. When the retrieval value is greater than the second preset threshold, the system marks the archive as a relevant archive and returns it to the target user as a recommended query result.

[0061] The final search formula is:

[0062] R = γe n +δln(εb+1);

[0063] Where R is the retrieval value of the archive data, γ, δ, and ε are the retrieval value adjustment parameters, n is the number of archive formats corresponding to each archive, and b is the number of keywords that the summary content of each archive meets the search requirements of the target user.

[0064] It should be noted that the more keywords the user searches for that match in the archive's abstract, or the archive has multiple formats, the higher its retrieval value will be, and the higher it will be ranked in the query results. By recommending archives with multiple formats, the forgetting of multiple formats can be accelerated; while retaining the complete archive format, storage space can be saved.

[0065] Embodiment 2: Based on Embodiment 1, a digital archive management system based on big data includes:

[0066] The user query module is used to obtain the query keywords of the target user and query the archives to form a sequence of archives to be processed;

[0067] The requirement format and authority management module is used to adjust the to-be-processed archive sequence according to the requirement format and authority data of the target user, and divide it into a first query archive sequence and a second query archive sequence;

[0068] The query adjustment and file format conversion module is used to adjust the query quantity according to the target user's permission data, convert the summary content or file content of the file, and perform distributed storage;

[0069] A distributed storage and management module is used to store the converted archive data and the original archive data in a distributed manner;

[0070] A forgetting adjustment module, used for performing forgetting adjustment on the stored archive data using a forgetting adjustment formula;

[0071] A similarity calculation module is used to calculate the similarity of the archive content using the similarity acquisition model, based on the similarity between the user's query history and the archive content;

[0072] The query recommendation module is used to adjust the order of archive recommendation and recommend archives with a large number of archive formats.

[0073] The above description is only an example of the present invention and is not intended to limit the present invention. Any equivalent substitutions made within the principles of the present invention should be included in the protection scope of the present invention. The contents not elaborated in detail in the present invention belong to the existing technologies known to those skilled in the art.

Claims

1. A digital archive management method based on big data, characterized in that: S1: Obtain the query keyword of the target user, query the archives according to the query keyword, and take the relevant archives as the archive sequence to be processed; S2: Obtain the target user's demand format and permission data, and divide the to-be-processed archive sequence into a first query archive sequence and a second query archive sequence according to the target user's demand format; S3: adjusting the query quantity of the first query archive sequence or the second query archive sequence based on the authority data of the target user to obtain the adjusted query archive quantity, and converting the summary content or the archive content into a format and distributing the formatted content based on the summary content selected by the target user for query; S4: Based on the converted archive data and the original archive data in distributed storage, use the forgetting adjustment formula to make adjustments, and return the recommended query results to the target user based on the adjusted archive data.

2. A digital archive management method based on big data as claimed in claim 1, characterized in that: The method of obtaining the target user's requirement format and permission data and dividing the to-be-processed archive sequence into a first query archive sequence and a second query archive sequence according to the target user's requirement format includes: taking archives in the to-be-processed archive sequence that are in a format different from that of the target user as the first query archive sequence, and taking archives in the to-be-processed archive sequence that are in a format the same as that of the target user as the second query archive sequence.

3. A digital archive management method based on big data as claimed in claim 1, characterized in that: The step of adjusting the query quantity of the first query archive sequence or the second query archive sequence based on the authority data of the target user to obtain the adjusted query archive quantity includes: using a quantity adjustment formula according to the authority data of the target user to obtain the adjusted query archive quantity; when the adjusted query archive quantity is less than the quantity of the second query archive sequence, selecting archives of the adjusted query archive quantity from the second query archive sequence; when the adjusted query archive quantity is greater than or equal to the quantity of the second query archive sequence, supplementing the archives from the first query archive sequence, wherein the quantity adjustment formula is: Q total =Q origina (1+μM); In the formula, Q total is the number of query files after adjustment, Q origina is the number of initial query files, μ is the adjustment coefficient, and M is the adjustment factor related to the target user's permissions.

4. A digital archive management method based on big data as claimed in claim 3, characterized in that: The method converts the summary content or the archive content of the archive into a format and then performs distributed storage based on the summary content selected by the target user for query, including: converting the summary content in the first query archive sequence, obtaining a query result with a maximum number of summary contents returned to the target user according to the adjustment of the query archive quantity, converting the archive content corresponding to the summary content selected by the target user for query, and distributing the converted summary content and archive content and the unconverted summary content and archive content.

5. A digital archive management method based on big data as claimed in claim 1, characterized in that: The method uses a forgetting adjustment formula to adjust the converted archive data and the original archive data in distributed storage, and returns a recommended query result to the target user based on the adjusted archive data, including: using the forgetting adjustment formula to obtain a forgetting adjustment value of the converted archive content or the original archive content, forgetting the archive content whose forgetting adjustment value is greater than or equal to a first preset threshold, and distributing the remaining archive content, wherein the forgetting adjustment formula is: S = αw + βt; Where S is the forgetting adjustment value, α and β are adjustment parameters, w is the similarity between the corresponding parts of the transformed file and the original file, and t is the number of views of the corresponding parts of the transformed file or the original file.

6. A digital archive management method based on big data as claimed in claim 5, characterized in that: Before using the forgetting adjustment formula to obtain the forgetting adjustment value of the converted archive data or the original archive data, the method includes: using the FIFO algorithm to obtain the historical query archive sequence of each converted archive format and the original archive format, and performing weighted summation on each archive format in the historical query archive sequence to obtain the converted archive format weight and the original archive format weight, and using the forgetting adjustment formula to forget the archive format with the smallest weight; when the converted archive format weight is the same as the original archive format weight, forgetting the archive format that entered the historical query archive sequence earliest; wherein at least one complete archive is retained for each archive.

7. A digital archive management method based on big data as claimed in claim 5, characterized in that: Before using the forgetting adjustment formula to obtain the forgetting adjustment value of the converted archive data or the original archive data, the method includes: decomposing the converted archive or the original archive to obtain partial content of each archive, and inputting the partial content of each archive into a similarity acquisition model to obtain the similarity between the converted archive and the corresponding partial content of the original archive, wherein the similarity acquisition model uses a BERT model, and before inputting the partial content of each archive into the similarity acquisition model, the text content is preprocessed by removing punctuation marks, stop words, and converting it to lowercase, and the preprocessed text content is converted into an embedding vector, and the cosine similarity formula is used to obtain the similarity between the converted archive and the corresponding partial content of the original archive.

8. A digital archive management method based on big data as claimed in claim 7, characterized in that: The method of returning the recommended query result to the target user based on the adjusted archive data includes: using the final search formula to obtain the search value of the archive data, and returning the recommended query result to the target user for the archive data whose search value is greater than a second preset threshold.

9. A digital archive management method based on big data as claimed in claim 8, characterized in that: The method of using the final search formula to obtain the search value of the archive data includes: the final search formula is: R=γe n +δln(εb+1); Where R is the retrieval value of the archive data, γ, δ, and ε are the retrieval value adjustment parameters, n is the number of archive formats corresponding to each archive, and b is the number of keywords that the summary content of each archive meets the search requirements of the target user.

10. A digital archive management method based on big data, according to a digital archive management system based on big data according to any one of claims 1 to 9, characterized in that: include: The user query module is used to obtain the query keywords of the target user and query the archives to form a sequence of archives to be processed; The requirement format and authority management module is used to adjust the to-be-processed archive sequence according to the requirement format and authority data of the target user, and divide it into a first query archive sequence and a second query archive sequence; The query adjustment and file format conversion module is used to adjust the query quantity according to the target user's permission data, convert the summary content or file content of the file, and perform distributed storage; A distributed storage and management module is used to store the converted archive data and the original archive data in a distributed manner; A forgetting adjustment module, used for performing forgetting adjustment on the stored archive data using a forgetting adjustment formula; A similarity calculation module is used to calculate the similarity of the archive content using the similarity acquisition model, based on the similarity between the user's query history and the archive content; The query recommendation module is used to adjust the order of archive recommendation and recommend archives with a large number of archive formats.

Images