Archive classification management system based on big data

Abstract

The invention discloses an archive classification management system based on big data, and relates to the technical field of archive management, the archive classification management system comprises an archive data acquisition module, a data storage module, an archive classification module, a retrieval recommendation module and a visualization module, the retrieval recommendation module comprises a result recommendation unit and a correlation analysis unit, the result recommendation unit is used for providing specific archives according to the content input by the user, and the correlation analysis unit compares the archives provided by the result recommendation unit with other archives. Based on historical access frequency, timeliness, similarity and other factors, archives most meeting requirements are recommended for users, related archives are provided for reference, retrieval precision is improved, efficiency of an archive classification management system is improved, and the effects of effectively improving access speed and response time, reducing storage cost and reducing system construction cost are achieved through a hierarchical storage technology.

Description

Technical Field

[0001] The present invention relates to the technical field of archive management, and in particular to an archive classification management system based on big data. Background Art

[0002] With the development of modern civilized society, "archives" have long become something that people are familiar with. They appear in people's lives, studies, and work, and run through various aspects such as scientific research, medical treatment, and litigation. In order to ensure the integrity and originality of archives, archive classification management has come into being. With the development of big data technology, information has gradually been integrated through Internet technology in various technical fields to share information resources. Archival information has the characteristics of large data volume, complex information, and a wide variety of types. Traditional archival information is mainly paper archives. The collection, classification, and storage of a large amount of archival information have brought huge workload and pressure to the staff.

[0003] With the development of science and technology, digitalization of archives has become a trend. However, in the current process of archive digitization, archives are usually scanned and identified and stored together. Due to the huge amount of archive data and the high cost of high-performance storage devices, the cost of system construction increases. In addition, when retrieving archive data, there is a lack of comprehensive analysis of the archive data and the inability to adjust the search results, resulting in low accuracy of the retrieval results and reduced effectiveness of the archive classification management system. Summary of the invention

[0004] The purpose of the present invention is to provide an archive classification management system based on big data, which solves the problems raised in the above-mentioned background technology.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an archive classification management system based on big data, comprising an archive data collection module, a data storage module, an archive classification module, a retrieval recommendation module and a visualization module; The archive data acquisition module is used to obtain archive data from a data source; The data storage module is used to store the data collected by the archive data collection module, and the data storage module includes a hierarchical storage unit, and the hierarchical storage unit is used to divide the archive data into cold archives and hot archives, and use different storage media for storage; The archive classification module uses a metadata collection tool to obtain metadata of archive data and generates a label for each archive data. The label includes time, keyword, data source and category, and classifies the archive data according to the label; The search recommendation module includes a result recommendation unit and a correlation analysis unit. The result recommendation unit is used to provide specific archives according to the content input by the user. The correlation analysis unit compares the archives provided by the result recommendation unit with other archives and provides relevant archives to the user according to the comparison results.

[0006] Optionally, the archival data includes paper archives and electronic archives, the electronic archives include audio and video data, and the archival data acquisition module includes an optical recognition unit and an audio conversion unit. The optical recognition unit is used to convert paper archives into image format and extract text content in the image format, and the audio conversion unit is used to convert audio and video data, extract text data, and perform digital processing in combination with video transcoding technology.

[0007] Optionally, the hierarchical storage unit archive data division process is as follows: ; Where A is the heat score; F is the file access frequency, which indicates the number of accesses per unit time in the past; α is the access frequency influence coefficient, ranging from 0 to 1; T is the time interval since the last visit; β is the access time interval influence coefficient, ranging from 0 to 1; Z is the archive importance weight, ranging from 0 to 1; The higher the heat score A is, the more frequently the archive data is used. Conversely, the lower the heat score A is, the less frequently the archive data is used. The classification threshold of the heat score A is set to Y1. When A is greater than Y1, it is a hot archive. When A is less than Y1, it is a cold archive. For hot archives, move them to high-performance storage media to improve access speed and response time. For cold archives, move them to low-cost storage media to reduce storage costs.

[0008] Optionally, the result recommendation unit process is as follows: ; Where G represents the user query content, and S represents the candidate score; S(D1,G) represents the candidate score of content profile 1 according to the user query; R(D1,G) represents the similarity score between the user query content and profile 1; W1 represents the influence coefficient of R(D1,G), ranging from 0 to 1; H(D1) represents the historical access frequency of file 1; W2 represents the influence coefficient of H(D1), ranging from 0 to 1; C(D1) indicates the timeliness of file 1; W3 represents the influence coefficient of C(D1), ranging from 0 to 1; The historical access frequency H(D1) of file 1 is obtained as follows: ; Where U represents the user set; M p (D1) represents the number of times user p accesses file 1; M p It represents the influence coefficient of user p, ranging from 0 to 1, representing the importance of different users; The larger S(D1,G) is, the stronger the correlation between file 1 and the user's query content is. When the user uses the system, after entering the user query content G, the search result bar will automatically recommend the file with the highest score to ensure the accuracy of the file query. The user can adjust W1, W2 and W3 according to needs to improve the flexibility of file query.

[0009] Optionally, the correlation analysis unit performs an analysis process as follows: ; Where R(D1,D2) represents the similarity score between file 1 and file 2; F i (D1) represents the frequency of occurrence of the i-th keyword in file 1; F i (D2) represents the frequency of occurrence of the i-th keyword in file 2; n represents the number of keywords; It represents the sum of the minimum values ​​of the shared feature frequencies between profile 1 and profile 2, which is used to calculate how much overlap the two profiles have on each keyword and find the minimum value for each keyword; It represents the maximum value of the total number of features in file 1 and file 2; By calculating the degree of overlap of shared features between two archives and normalizing them with the maximum value of their total number of features, the correlation between the two archives is measured to obtain a similarity score. The correlation analysis unit can determine the similarity of the two archives in terms of keywords, help find similar archives and compare archive contents. After the result recommendation unit recommends the archive with the highest candidate score S to the user, the correlation analysis unit will calculate the similarity score between the archive with the highest candidate score S and other archives, and set the score threshold of R(D1, D2) to Y2. When R(D1, D2) is greater than Y2, it means that the similarity between the two archives is high, and the archive compared with the archive with the highest score is added to the search result column. When R(D1, D2) is less than Y2, it is not added.

[0010] Optionally, the visualization module provides the administrator with monitoring data of system operation, the monitoring data including archive storage capacity, access frequency and system hardware operating parameters.

[0011] Optionally, the high-performance storage medium includes a solid-state drive, and the low-cost storage medium includes a mechanical hard drive.

[0012] Optionally, the metadata collection tool is Apache Tika.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. After the present invention classifies the archival data, when the user searches, the result recommendation unit in the search recommendation module recommends the most suitable archives to the user according to the user input content. The result recommendation unit includes influencing factors such as historical access frequency, timeliness, and similarity with the user input content, so as to achieve a comprehensive analysis of the archival data. In actual operation, the user can make adjustments as needed, thereby improving the archival retrieval accuracy and the use quality of the archival classification management system. The archives recommended by the result recommendation unit are then compared with other archives through the correlation analysis unit, and relevant archives are provided to the user according to the comparison results for reference, so that the user can view the relevant archives without repeated searches, thereby improving the intelligence level and use experience of the archival classification management system.

[0014] 2. When storing archival data, the present invention classifies the archival data according to factors such as access frequency and importance of the archival data through the hierarchical storage unit in the data storage module, and divides the archival data into hot archives and cold archives. For hot archives, they are moved to high-performance storage media to improve access speed and response time, and for cold archives, they are moved to low-cost storage media to reduce storage costs. Therefore, the use of storage devices can be determined according to the actual situation of the archival data, and the cost of storage devices can be reduced while ensuring the efficiency of system use, thereby reducing the construction cost of the archive classification management system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a block diagram of the system modules of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Example: See Figure 1,This implementation provides a file classification management system based on big data, including a file data collection module, a data storage module, a file classification module, a retrieval recommendation module and a visualization module; The archive data acquisition module is used to obtain archive data from the data source; The data storage module is used to store the data collected by the archive data collection module. The data storage module includes a hierarchical storage unit. The hierarchical storage unit is used to divide the archive data into cold archives and hot archives, and use different storage media for storage; The archive classification module uses metadata collection tools to obtain metadata of archive data and generates labels for each archive data. The labels include time, keywords, data source and category, and classifies the archive data according to the labels. The search recommendation module includes a result recommendation unit and a relevance analysis unit. The result recommendation unit is used to provide specific archives according to the content input by the user. The relevance analysis unit compares the archives provided by the result recommendation unit with other archives and provides relevant archives to the user according to the comparison results.

[0018] More specifically, in this embodiment: the archive data acquisition module is used to obtain archive data from the data source, and then when storing it through the data storage module, the archive data is classified according to factors such as access frequency and importance of the archive data through the hierarchical storage unit, and the archive data is divided into hot archives and cold archives, and different storage media are used for storage, so that the use of storage devices can be determined according to the actual situation of the archive data, and the cost of storage devices can be reduced while ensuring the efficiency of system use, thereby reducing the construction cost of the archive classification management system.

[0019] Then, the metadata of the archival data is obtained by using the metadata collection tool through the archive classification module, and a label is generated for each archival data, and the archival data is classified according to the label. Later, when the user searches, he can directly enter the content in the search bar to search, and can directly enter the archive label information. At the same time, the result recommendation unit in the retrieval recommendation module recommends the most suitable archives for the user based on the user input content. The result recommendation unit includes influencing factors such as historical access frequency, timeliness, and similarity with the user input content, so as to achieve a comprehensive analysis of the archival data, and in actual operation, the user can make adjustments as needed, thereby improving the accuracy of archive retrieval, ensuring that it meets user needs, and improving the quality of use of the archive classification management system. Afterwards, the archives recommended by the result recommendation unit are compared with other archives through the correlation analysis unit, and relevant archives are provided to the user based on the comparison results for reference, so that the user can view the relevant archives without repeated searches, thereby improving the intelligence level and usage experience of the archive classification management system.

[0020] Furthermore, the archival data includes paper archives and electronic archives, the electronic archives include audio and video data, and the archival data acquisition module includes an optical recognition unit and an audio conversion unit. The optical recognition unit is used to convert the paper archives into an image format and extract the text content in the image format. The audio conversion unit is used to convert the audio and video data, extract the text data, and perform digital processing in combination with video transcoding technology.

[0021] Specifically, electronic archives also include PDF, Word, Excel and other files. The audio in audio and video files includes lectures, meeting minutes, interviews, etc., and the video files include meeting videos, lecture videos, surveillance videos, etc. Specifically, ASR speech recognition technology can be used to convert the speech in audio files into text, and video transcoding can be used to convert video files into a data format that can be analyzed. The audio in the video is combined for speech recognition processing, and all audio and video data are stored in text, which can facilitate subsequent archive classification management.

[0022] Furthermore, the hierarchical storage unit archive data division process is as follows: ; Where A is the heat score; F is the file access frequency, which indicates the number of accesses per unit time in the past; α is the access frequency influence coefficient, ranging from 0 to 1; T is the time interval since the last visit; β is the access time interval influence coefficient, ranging from 0 to 1; Z is the archive importance weight, ranging from 0 to 1; Specifically, the higher the heat score A, the more frequently the archive data is used, and vice versa, the less frequently the archive data is used. The classification threshold of the heat score A is set to Y1. When A is greater than Y1, it is a hot archive, and when A is less than Y1, it is a cold archive. For hot archives, they are moved to high-performance storage media to improve access speed and response time. For cold archives, they are moved to low-cost storage media to reduce storage costs, but the access speed and response time will be reduced accordingly. High-performance storage media include solid-state drives, and low-cost storage media include mechanical hard drives. In actual operations, managers can adjust the archive importance weight, access frequency influence coefficient, and access time interval influence coefficient according to the actual situation of the archive to prevent misclassification. For example, long-term important archives are classified as cold archives because they are accessed too little in the short term, thereby improving the flexibility of this archive classification management system.

[0023] Furthermore, the result recommendation unit process is as follows: ; Where G represents the user query content, and S represents the candidate score; S(D1,G) represents the candidate score of content profile 1 according to the user query; R(D1,G) represents the similarity score between the user query content and profile 1; W1 represents the influence coefficient of R(D1,G), ranging from 0 to 1; H(D1) represents the historical access frequency of file 1; W2 represents the influence coefficient of H(D1), ranging from 0 to 1; C(D1) indicates the timeliness of file 1; W3 represents the influence coefficient of C(D1), ranging from 0 to 1; The historical access frequency H(D1) of file 1 is obtained as follows: ; Where U represents the user set; M p (D1) represents the number of times user p accesses file 1; M p It represents the influence coefficient of user p, ranging from 0 to 1, representing the importance of different users; Specifically, the larger the S(D1,G), the stronger the correlation between file 1 and the user's query content. When the user uses the system, after entering the user query content G, the search result bar will automatically recommend the file with the highest score to ensure the accuracy of the file query. In actual operation, the system adds adjustment options for different influence coefficients in the search bar settings. Users can adjust W1, W2 and W3 as needed to ensure that the search results meet their needs and improve the flexibility of file queries and the accuracy of results.

[0024] Furthermore, the correlation analysis unit analysis process is as follows: ; Where R(D1,D2) represents the similarity score between file 1 and file 2; F i (D1) represents the frequency of occurrence of the i-th keyword in file 1; F i (D2) represents the frequency of occurrence of the i-th keyword in file 2; n represents the number of keywords; It represents the sum of the minimum values ​​of the shared feature frequencies between profile 1 and profile 2, which is used to calculate how much overlap the two profiles have on each keyword and find the minimum value for each keyword; It represents the maximum value of the total number of features in file 1 and file 2; Specifically, the correlation between the two archives is measured by calculating the degree of overlap of shared features between the two archives and normalizing them with the maximum value of their total number of features, and the similarity score R(D1, D2) of archive one and archive two is obtained. R(D1, D2) can determine the similarity between the two archives in terms of keywords or other features, and help find similar archives and compare archive contents. After the result recommendation unit recommends the highest-scoring archive to the user, the correlation analysis unit will calculate the similarity score between the highest-scoring archive and other archives, and set the score threshold of R(D1, D2) to Y2. When R(D1, D2) is greater than Y2, it means that the two archives are highly similar, and the archive compared with the highest-scoring archive is added to the search result bar. When R(D1, D2) is less than Y2, it is not added. In actual application, in order to avoid too many archives in the search result bar, the number of displays in the search result bar can be set to improve the system usage experience.

[0025] Furthermore, the visualization module provides administrators with monitoring data of system operation, including file storage capacity, access frequency and system hardware operating parameters.

[0026] Specifically, during system operation, managers can view system usage in real time, including system storage, graphics card and CPU usage, to avoid system overload, and convert data into charts through the visualization module, so that managers can quickly and clearly understand system usage and improve system management quality.

[0027] Furthermore, the metadata collection tool is Apache Tika.

[0028] Specifically, Apache Tika is an open source content analysis tool that can automatically detect and extract metadata and content of files, and supports multiple file formats, including documents, PDFs, images, audio, video, etc. It can generate structured metadata for archival data and support batch processing of large numbers of files, and is suitable for large-scale archival data classification scenarios.

[0029] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A file classification management system based on big data, characterized in that: It includes archive data collection module, data storage module, archive classification module, retrieval recommendation module and visualization module; The archive data acquisition module is used to obtain archive data from a data source; The data storage module is used to store the data collected by the archive data collection module, and the data storage module includes a hierarchical storage unit, and the hierarchical storage unit is used to divide the archive data into cold archives and hot archives, and use different storage media for storage; The archive classification module uses a metadata collection tool to obtain metadata of archive data and generates a label for each archive data. The label includes time, keyword, data source and category, and classifies the archive data according to the label; The search recommendation module includes a result recommendation unit and a correlation analysis unit. The result recommendation unit is used to provide specific archives according to the content input by the user. The correlation analysis unit compares the archives provided by the result recommendation unit with other archives and provides relevant archives to the user according to the comparison results.

2. The file classification management system based on big data according to claim 1 is characterized by: The archival data includes paper archives and electronic archives, the electronic archives include audio and video data, and the archival data acquisition module includes an optical recognition unit and an audio conversion unit. The optical recognition unit is used to convert paper archives into image format and extract text content in the image format. The audio conversion unit is used to convert audio and video data, extract text data, and perform digital processing in combination with video transcoding technology.

3. The file classification management system based on big data according to claim 2 is characterized by: The hierarchical storage unit archive data division process is as follows: ； Where A is the heat score; F is the file access frequency, which indicates the number of accesses per unit time in the past; α is the access frequency influence coefficient, ranging from 0 to 1; T is the time interval since the last visit; β is the access time interval influence coefficient, ranging from 0 to 1; Z is the archive importance weight, ranging from 0 to 1; The higher the heat score A is, the more frequently the archive data is used. Conversely, the lower the heat score A is, the less frequently the archive data is used. The classification threshold of the heat score A is set to Y1. When A is greater than Y1, it is a hot archive. When A is less than Y1, it is a cold archive. For hot archives, move them to high-performance storage media to improve access speed and response time. For cold archives, move them to low-cost storage media to reduce storage costs.

4. The file classification management system based on big data according to claim 3 is characterized by: The result recommendation unit process is as follows: ； Where G represents the user query content, and S represents the candidate score; S(D1,G) represents the candidate score of content profile 1 according to the user query; R(D1,G) represents the similarity score between the user query content and profile 1; W1 represents the influence coefficient of R(D1,G), ranging from 0 to 1; H(D1) represents the historical access frequency of file 1; W2 represents the influence coefficient of H(D1), ranging from 0 to 1; C(D1) indicates the timeliness of file 1; W3 represents the influence coefficient of C(D1), ranging from 0 to 1; The historical access frequency H(D1) of file 1 is obtained as follows: ； Where U represents the user set; M p (D1) represents the number of times user p accesses file 1; M p It represents the influence coefficient of user p, ranging from 0 to 1, representing the importance of different users; The larger S(D1,G) is, the stronger the correlation between file 1 and the user's query content is. When the user uses the system, after entering the user query content G, the search result bar will automatically recommend the file with the highest score to ensure the accuracy of the file query. The user can adjust W1, W2 and W3 according to needs to improve the flexibility of file query.

5. The file classification management system based on big data according to claim 4 is characterized by: The analysis process of the correlation analysis unit is as follows: ； Where R(D1,D2) represents the similarity score between file 1 and file 2; F i (D1) represents the frequency of occurrence of the i-th keyword in file 1; F i (D2) represents the frequency of occurrence of the i-th keyword in file 2; n represents the number of keywords; It represents the sum of the minimum values ​​of the shared feature frequencies between profile 1 and profile 2, which is used to calculate how much overlap the two profiles have on each keyword and find the minimum value for each keyword; It represents the maximum value of the total number of features in file 1 and file 2; By calculating the degree of overlap of shared features between two archives and normalizing them with the maximum value of their total number of features, the correlation between the two archives is measured to obtain a similarity score. The correlation analysis unit can determine the similarity of the two archives in terms of keywords, help find similar archives and compare archive contents. After the result recommendation unit recommends the archive with the highest candidate score S to the user, the correlation analysis unit will calculate the similarity score between the archive with the highest candidate score S and other archives, and set the score threshold of R(D1, D2) to Y2. When R(D1, D2) is greater than Y2, it means that the similarity between the two archives is high, and the archive compared with the archive with the highest score is added to the search result column. When R(D1, D2) is less than Y2, it is not added.

6. The file classification management system based on big data according to claim 1 is characterized by: The visualization module provides the administrator with monitoring data of system operation, including file storage capacity, access frequency and system hardware operation parameters.

7. The file classification management system based on big data according to claim 3 is characterized by: The high-performance storage medium includes a solid-state hard disk, and the low-cost storage medium includes a mechanical hard disk.

8. The file classification management system based on big data according to claim 1 is characterized by: The metadata collection tool is Apache Tika.

Images