Electronic archive automatic classification management method and system based on big data
Through modal decomposition and preprocessing methods based on big data, the quality scores of text, image and video data of electronic files are solved, and the evaluation inconsistency problem in multimodal electronic files management is realized, efficient and automated classification management is achieved, and data quality and availability are improved.
Patent Information
- Application Number
- CN202510659158.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-21
- Publication Date
- 2025-08-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art is difficult to conduct comprehensive, efficient and automated quality evaluation and classification management of multimodal electronic files, resulting in strong subjectivity and high inconsistency in the evaluation results, and it is impossible to comprehensively consider the quality of various data such as text, images, audio and video.
Using a big data-based method, through modal decomposition and preprocessing, the electronic archives are scored in quality of text, image and video data, and quantitative evaluation indicators are constructed to realize automatic classification management, including missing value processing of text data, image clarity and uniformity evaluation, and audio signal-to-noise ratio calculation of videos, and automatic classification of electronic archives is combined with the comprehensive score of multimodal data.
It realizes comprehensive and in-depth processing and evaluation of electronic archive data, improves data quality and availability, ensures quantitative standards and uniformity of evaluation, supports precise classification, differentiated storage, dynamic monitoring and repair, and improves management efficiency and security.
Smart Images
Figure CN120508691A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of automatic classification management, and specifically relates to a method and system for automatic classification management of electronic archives based on big data. Background Art
[0002] In the digital age, the number of electronic archives has exploded, encompassing a variety of modalities, including text, images, structured data, audio, and video. These electronic archives are widely used across government agencies, businesses, scientific research institutions, and other sectors, serving as a vital resource for information storage, sharing, and utilization. However, as the scale of electronic archives continues to expand, managing them efficiently and in a standardized manner has become a pressing challenge. Traditional electronic archive management methods have exposed numerous limitations when faced with massive, multimodal electronic archives, making them difficult to meet practical needs.
[0003] Conventional methods for evaluating the quality of electronic archives rely primarily on manual judgment and lack objective, quantitative evaluation metrics. Different evaluators may reach different conclusions, leading to subjective and inconsistent evaluation results. For multimodal electronic archives, conventional methods struggle to comprehensively consider the quality of multiple data types, including text, images, audio, and video, and therefore are unable to provide comprehensive quality assessment results.
[0004] Therefore, there is an urgent need for an automatic classification and management method for electronic archives based on big data, which can carry out comprehensive, efficient and automated processing, evaluation and classification management of multimodal electronic archives. Summary of the Invention
[0005] The purpose of the present invention is to provide a method and system for automatic classification and management of electronic archives based on big data, which is used to solve the technical problem in the existing technology that it is difficult to comprehensively consider the quality of various data such as text, images, audio and video.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions: The automatic classification and management method of electronic archives based on big data includes: Step 1: Clarify the data content covered by the electronic archives and perform modal decomposition and preprocessing on the original electronic archives; Step 2: Based on the multimodal data of the electronic document, construct a quality score for the text data, image data, and video data in the electronic document; Step 3: Based on the quality scores of text data, image data and video data in electronic documents, automatic classification management of electronic archives is achieved.
[0007] Furthermore, the data content of the electronic archives is clarified, and the original electronic archives are subjected to modal decomposition and preprocessing. The specific methods are as follows: Clarify the data content covered by electronic archives. According to different data types, the data content covered by electronic archives is divided into text data, image data, structured data and video data for classification and processing. For text data and structured data, use NLP to convert text information of different formats into a unified format. Use regular expressions to identify and remove unit characters in the text. Fill or delete missing values according to the missing value ratio. Delete stop words that appear in the text content by constructing a stop word list. Use OCR technology to extract text content from images, process the text content according to the text data processing method, determine the resolution of each image in the electronic archive through the image processing library, use the Sobel operator to determine the image clarity, calculate the standard deviation of the pixel grayscale values in the electronic archive image data, and obtain the image uniformity; According to the preset frame rate a, video frames are extracted from the video data, the extracted video frames are processed according to the image data processing method, independent audio data is separated from the video data, the audio signal in the audio data is sampled at the preset sampling rate, the sampled audio signal is Fourier transformed, the energy distribution of the spectrum is calculated to obtain the signal power, and in the spectrum, the frequency band with a frequency greater than A is selected as the noise frequency band, the energy of the noise frequency band is selected as the noise power, and the ratio of the audio signal power to the noise power is used as the audio data quality score.
[0008] Furthermore, the Sobel operator is used to determine the clarity of the image. The specific method is as follows: The Sobel operator is used to calculate the gradient amplitude of the electronic archive image data. The average value of the gradient amplitude of all pixels in the image represents the clarity of the image. The formula is used Represents the clarity of the image, where k represents the kth image, qx(k) represents the clarity of the kth image, Indicates the height of the k-th image in pixels. Indicates the width of the k-th image in pixels, s1 indicates the clarity constant factor, i indicates the i-th pixel coordinate in the vertical direction of the image, and j indicates the j-th pixel coordinate in the horizontal direction of the image. Represents the gradient magnitude of pixel (i, j).
[0009] Furthermore, the standard deviation of the pixel grayscale values in the electronic archive image data is calculated to obtain the uniformity of the image. The specific method is as follows: Using the formula Represents the uniformity of the image, where k represents the kth image, represents the uniformity of the k-th image, Indicates the height of the k-th image in pixels. Indicates the width of the k-th image in pixels, s2 indicates the uniformity constant factor, i indicates the i-th pixel coordinate in the vertical direction of the image, j indicates the j-th pixel coordinate in the horizontal direction of the image, represents the grayscale value of pixel (i, j), Represents the mean grayscale value of all pixels in the kth image.
[0010] Furthermore, a quality score of text data in electronic documents is constructed. The specific method is as follows: Using the formula represents the quality score of the electronic archive text data, where x represents the xth electronic archive, represents the text data quality index of the x-th electronic file, ys(x) represents the total number of characters in the text of the x-th electronic file, tc(x) represents the number of characters deleted due to missing values in the x-th electronic file, sj(x) represents the number of characters added due to missing values in the x-th electronic file, rc(x) represents the number of unconventional characters removed due to regular cleaning in the x-th electronic file, b is a constant, It represents the number of stop word characters removed from the xth electronic file, e is a natural constant, k1 represents the missing value deletion weight coefficient, k2 represents the missing value addition weight coefficient, and k3 represents the unconventional weight coefficient.
[0011] Furthermore, a quality score of image data in electronic documents is constructed, and the specific method is as follows: Preset the specific requirements for the resolution of electronic archive images, determine the resolution, clarity and uniformity of each image in the electronic archive image data, and use the formula represents the quality score of electronic archive image data, where x represents the xth electronic archive, represents the image data quality score of the x-th electronic file, k represents the k-th image, N(x) represents that the x-th electronic file contains a total of N(x) images, represents the resolution of image k, f is the specific requirement for the resolution of electronic archive images, I[*] is the conditional function, when condition * is met, I[*] is equal to 1, when condition * is not met, I[*] is equal to 0, qx(k) represents the clarity of the kth image, represents the uniformity of the k-th image.
[0012] Furthermore, a quality score of the video data in the electronic document is constructed, and the specific method is as follows: Using the formula represents the quality score of the electronic archive video data, where x represents the xth electronic archive, represents the video data quality score of the x-th electronic file, v represents the v-th video, M(x) represents that the x-th electronic file contains a total of M(x) videos, g represents the g-th video frame extracted from the video, Indicates that the vth video has extracted a total of video frames, represents the resolution of the g-th video frame, f is the specific requirement for the electronic archive image resolution, I[*] is the conditional function, when condition * is met, I[*] is equal to 1, when condition * is not met, I[*] is equal to 0, qx(g) represents the clarity of the g-th video frame, Indicates the uniformity of the g-th video frame.
[0013] Furthermore, based on the quality scores of text data, image data and video data in electronic documents, automatic classification management of electronic archives is achieved. The specific method is as follows: Presetting electronic document text data quality score thresholds, image data quality score thresholds, and video data quality score thresholds, determining the quality scores of the electronic document text data, image data, and video data to be classified and managed, and when the text data, image data, and video data quality scores of the electronic document are all greater than or equal to the corresponding thresholds, storing the electronic document in a distributed database that supports millisecond-level retrieval and automatically backing it up to multiple data centers on a regular basis; When the quality score of any one of the text data, image data, and video data of the electronic document is lower than the corresponding threshold, the electronic document is stored in the standard cloud storage, and access frequency monitoring is configured to automatically recheck the quality score of the electronic document every T1 period. When the quality scores of any two of the electronic text data, image data, and video data are less than the corresponding threshold, the electronic document is marked as a document to be repaired, pushed to the repair queue, stored in low-cost storage, with restricted access rights, and retained for T2 time. The quality score of the electronic document is automatically rechecked, and the document is automatically cleared if the quality score does not improve; When the quality scores of the electronic text data, image data and video data are all less than the corresponding thresholds, the document will be immediately compressed and stored in cold storage, direct access will be prohibited, and it will be automatically deleted after T3 time. A summary report will be generated for audit before deletion.
[0014] The present invention also proposes an automatic classification and management system for electronic archives based on big data, which is applied to the automatic classification and management method for electronic archives based on big data, comprising: The electronic archive data content processing module is used to clarify the data content covered by the electronic archive and perform modal decomposition and preprocessing on the original electronic archive; The electronic archive quality score setting module is used to construct the quality score of the text data, image data and video data in the electronic document based on the multimodal data of the electronic document; The electronic archive automatic classification management module realizes automatic classification management of electronic archives based on the quality scores of text data, image data and video data in electronic documents.
[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention clearly divides the data content of electronic archives into text data, image data, structured data, and video data. It adopts specialized processing methods for different types of data, and conducts comprehensive and in-depth processing and evaluation of different types of data covered by electronic archives. This improves the quality and usability of electronic archive data, and provides strong support for the long-term preservation, effective utilization, and value mining of electronic archives. In combination with image clarity and uniformity, the quality assessment process has clear quantitative standards and is operational. 2. The present invention constructs a detailed text data quality assessment indicator system that can accurately reflect problems in text data in terms of integrity, standardization, semantic interference, etc. By presetting specific requirements for electronic archive image resolution and comprehensively setting image data quality scores based on image resolution, clarity, and uniformity, the uniformity and standardization of image data quality assessment are ensured, avoiding assessment deviations caused by inconsistent standards such as resolution. By comprehensively combining the image data quality score and audio data quality score of video frame data, the electronic archive video data quality score is set, fully considering the two important components of video data, namely image and audio, making video data quality assessment more comprehensive and accurate; 3. The present invention realizes automatic classification management of electronic archives based on the quality scores of text data, image data and video data in electronic documents. It has many beneficial effects such as accurate classification, differentiated storage, dynamic monitoring and repair, auditing and compliance, and improved user experience and decision support. It helps to improve the efficiency and quality of electronic document management and ensure data security and compliance. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0017] Figure 1 A step diagram of an automatic classification management method for electronic archives based on big data is shown; Figure 2Shows a module diagram of an automatic classification management system for electronic files based on big data; Figure 3 Shows a flowchart of the calculation method for the quality score of electronic file video data. Specific implementation manner
[0018] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
[0019] Embodiment 1. As Figure 1 、 Figure 3 shown, the automatic classification management method for electronic files based on big data specifically includes the following steps: Step 1: Define the data content covered by the electronic files, and perform modal decomposition and preprocessing on the original electronic files.
[0020] Define the data content covered by the electronic files. According to different data types, the data content covered by the electronic files is divided into text data, image data, structured data, and video data for classification processing. Among them, structured data refers to database tables and Excel report data in the electronic files. For text data and structured data, NLP is used to convert text information in different formats into a unified format. The Regular Expression technology is used to define patterns by setting specific syntax rules in advance, identify and remove irrelevant characters, special symbols, and garbled parts in the text, and record the number of removed unit characters. For numerical text data with fewer missing values, that is, when the missing value ratio is less than or equal to h, the mean value is used for filling. For text data with fewer missing values, the mode is used for filling. If the missing value ratio is higher than h, the deletion technology is adopted to directly delete the specific row where the missing value is located, and record the number of deleted unit characters and the number of unit characters filled with missing values. A stop word list is constructed, and common stop words (such as "de", "shi", "zai") are stored in a list or set. During the text processing, traverse the tokenized vocabulary list. If the vocabulary is in the stop word list, it is deleted, and the number of characters of the stop word list vocabulary deleted is recorded; Use OCR technology to extract text content from images. Process the text content according to the text data processing method. Use image processing libraries (such as OpenCV and PIL) to extract the number of pixels per unit length in the electronic archive image data. This will determine the resolution of each image in the electronic archive. Use the Sobel operator to calculate the gradient amplitude of the electronic archive image data. The average value of the gradient amplitude of all pixels in the image represents the clarity of the image. Calculate the standard deviation of the pixel grayscale values in the electronic archive image data and record it as the uniformity of the image in the electronic archive. Furthermore, the specific calculation formula for image clarity is as follows: ; Among them, k represents the k-th image, qx(k) represents the clarity of the k-th image, Indicates the height of the k-th image in pixels. Indicates the width of the k-th image in pixels, s1 indicates the clarity constant factor, i indicates the i-th pixel coordinate in the vertical direction of the image, and j indicates the j-th pixel coordinate in the horizontal direction of the image. Represents the gradient magnitude of pixel (i, j).
[0021] Furthermore, the specific calculation formula for image uniformity is as follows: ; Among them, k represents the kth image, represents the uniformity of the k-th image, Indicates the height of the k-th image in pixels. Indicates the width of the k-th image in pixels, s2 indicates the uniformity constant factor, i indicates the i-th pixel coordinate in the vertical direction of the image, j indicates the j-th pixel coordinate in the horizontal direction of the image, represents the grayscale value of pixel (i, j), Represents the mean grayscale value of all pixels in the kth image.
[0022] According to the preset frame rate a, video frames are extracted from the video data, the extracted video frames are processed according to the image data processing method, independent audio data is separated from the video data, the audio signal in the audio data is sampled at the preset sampling rate, the sampled audio signal is Fourier transformed, the energy distribution of the spectrum is calculated to obtain the signal power, and in the spectrum, the frequency band with a frequency greater than A is selected as the noise frequency band, the energy of the noise frequency band is selected as the noise power, and the ratio of the audio signal power to the noise power is used as the audio data quality score.
[0023] Step 2: Based on the multimodal data of the electronic document, construct a quality score for the text data, image data, and video data in the electronic document.
[0024] Clarify the text content of the electronic document, determine the number of characters deleted or added due to missing values in the text data and structured data, as well as the number of characters deleted due to the use of unconventional characters or incorrectly formatted characters in the text data and structured data. Based on the number of characters deleted due to stop words and repeated words in the text data and structured data, comprehensively determine the quality score of the electronic archive text data. The specific formula is as follows: ; Among them, x represents the xth electronic file, represents the text data quality index of the x-th electronic file, ys(x) represents the total number of characters in the text of the x-th electronic file, tc(x) represents the number of characters deleted due to missing values in the x-th electronic file, sj(x) represents the number of characters added due to missing values in the x-th electronic file, rc(x) represents the number of unconventional characters removed due to regular cleaning in the x-th electronic file, b is a constant, It represents the number of stop word characters removed from the xth electronic file, e is a natural constant, k1 represents the missing value deletion weight coefficient, k2 represents the missing value addition weight coefficient, and k3 represents the unconventional weight coefficient.
[0025] Preset specific requirements for the electronic archive image resolution, for example, requiring the electronic archive image resolution to be 1080×1080 or 1680×1680, determine the resolution, clarity, and uniformity of each image in the electronic archive image data, and comprehensively set the electronic archive image data quality score. The specific formula is as follows: ; Among them, x represents the xth electronic file, represents the image data quality score of the x-th electronic file, k represents the k-th image, N(x) represents that the x-th electronic file contains a total of N(x) images, represents the resolution of image k, f is the specific requirement for the resolution of electronic archive images, I[*] is the conditional function, when condition * is met, I[*] is equal to 1, when condition * is not met, I[*] is equal to 0, qx(k) represents the clarity of the kth image, represents the uniformity of the k-th image.
[0026] Determine the video frame data and audio data extracted from the electronic archive video data, integrate the image data quality score and the audio data quality score of the video frame data, and set the electronic archive video data quality score. The specific calculation formula is as follows: ; Among them, x represents the xth electronic file, represents the video data quality score of the x-th electronic file, v represents the v-th video, M(x) represents that the x-th electronic file contains a total of M(x) videos, g represents the g-th video frame extracted from the video, Indicates that the vth video has extracted a total of video frames, represents the resolution of the g-th video frame, f is the specific requirement for the electronic archive image resolution, I[*] is the conditional function, when condition * is met, I[*] is equal to 1, when condition * is not met, I[*] is equal to 0, qx(g) represents the clarity of the g-th video frame, Indicates the uniformity of the g-th video frame.
[0027] Step 3: Based on the quality scores of text data, image data and video data in electronic documents, automatic classification management of electronic archives is achieved.
[0028] Preset thresholds for electronic document text data quality scores, image data quality scores, and video data quality scores to determine the quality scores of the electronic document text data, image data, and video data to be classified and managed. When the quality scores of the electronic document text data, image data, and video data are all greater than or equal to the corresponding thresholds, the electronic document is judged to have high text standardization, meet image resolution standards, have clear videos, and have excellent audio signal-to-noise ratios. The electronic document is then stored in a distributed database (such as MongoDB Atlas) that supports millisecond-level retrieval and is automatically backed up regularly to multiple data centers. If the quality score of any one of the text, image, and video data of the electronic document is lower than the corresponding threshold, the electronic document is judged to have text missing, image blur, or insufficient video frame rate problems. The electronic document is stored in standard cloud storage (such as AWS S3) and access frequency monitoring is configured to automatically recheck the quality score of the electronic document every T1 period. When the quality scores of any two of the electronic text data, image data, and video data are lower than the corresponding threshold, the electronic document is judged to have garbled text, extremely low image resolution, or substandard video quality. The electronic document is marked as a document to be repaired, pushed to the repair queue, and stored in low-cost storage (such as Alibaba Cloud Storage) with restricted access rights and retained for T2. The electronic document quality score is automatically rechecked, and if the quality score does not improve, the document is automatically cleared. When the quality scores of the electronic text data, image data and video data are all less than the corresponding thresholds, the electronic document is judged to not meet the storage standards and the document is immediately compressed and stored in cold storage (such as Google Coldline). Direct access is prohibited and the document is automatically deleted after T3 time. A summary report is generated for audit before deletion.
[0029] Example 2: Figure 2 The shown automatic classification management system for electronic archives based on big data specifically includes the following: An electronic archive data content processing module that clarifies the data content covered by the electronic archives and performs modal decomposition and preprocessing on the original electronic archives.
[0030] Clarify the data content covered by the electronic archives. According to different data types, the data content covered by the electronic archives is classified into text data, image data, structured data, and video data for classification processing. Among them, structured data refers to database tables and Excel report data in the electronic archives. For text data and structured data, NLP is used to convert text information in different formats into a unified format. Using Regular Expression technology, patterns are defined by setting specific syntax rules in advance to identify and remove irrelevant characters, special symbols, and garbled parts in the text, and record the number of removed unit characters. For numerical text data with fewer missing values, that is, when the missing value ratio is less than or equal to h, the mean value is used for filling. For text data with fewer missing values, the mode is used for filling. If the missing value ratio is higher than h, deletion technology is adopted to directly delete the specific row where the missing value is located, and record the number of deleted unit characters and the number of unit characters filled for missing values. A stop word list is constructed, and common stop words (such as "of", "is", "in") are stored in a list or set. During the text processing, traverse the tokenized vocabulary list. If the vocabulary is in the stop word list, delete it and record the number of characters of the deleted stop word list vocabulary; Use OCR technology to extract the text content in the pictures, and process this text content according to the processing method of text data. Extract the number of pixels contained in each unit length of the electronic archive image data through an image processing library (such as OpenCV, PIL), thereby obtaining the resolution size of each image in the electronic archive. Use the Sobel operator to calculate the gradient amplitude of the electronic archive image data, and represent the clarity of the image by the average value of the gradient amplitudes of all pixels in the image. Calculate the standard deviation of the pixel gray values in the electronic archive image data, which is recorded as the uniformity of the images in the electronic archive; Furthermore, the specific calculation formula for the clarity of the image is as follows: ; Among them, k represents the k-th image, qx(k) represents the clarity of the k-th image, represents the height of the k-th image, with the unit of pixel, represents the width of the k-th image, with the unit of pixel, s1 represents the clarity constant factor, i represents the i-th pixel coordinate in the vertical direction of the image, j represents the j-th pixel coordinate in the horizontal direction of the image, represents the gradient amplitude of the pixel (i, j).
[0031] Furthermore, the specific calculation formula for image uniformity is as follows: ; Among them, k represents the kth image, represents the uniformity of the k-th image, Indicates the height of the k-th image in pixels. Indicates the width of the k-th image in pixels, s2 indicates the uniformity constant factor, i indicates the i-th pixel coordinate in the vertical direction of the image, j indicates the j-th pixel coordinate in the horizontal direction of the image, represents the grayscale value of pixel (i, j), Represents the mean grayscale value of all pixels in the kth image.
[0032] According to the preset frame rate a, video frames are extracted from the video data, the extracted video frames are processed according to the image data processing method, independent audio data is separated from the video data, the audio signal in the audio data is sampled at the preset sampling rate, the sampled audio signal is Fourier transformed, the energy distribution of the spectrum is calculated to obtain the signal power, and in the spectrum, the frequency band with a frequency greater than A is selected as the noise frequency band, the energy of the noise frequency band is selected as the noise power, and the ratio of the audio signal power to the noise power is used as the audio data quality score.
[0033] The electronic archive quality scoring setting module constructs the quality scores of text data, image data and video data in electronic documents based on the multimodal data of electronic documents.
[0034] Clarify the text content of the electronic document, determine the number of characters deleted or added due to missing values in the text data and structured data, as well as the number of characters deleted due to the use of unconventional characters or incorrectly formatted characters in the text data and structured data. Based on the number of characters deleted due to stop words and repeated words in the text data and structured data, comprehensively determine the quality score of the electronic archive text data. The specific formula is as follows: ; Among them, x represents the xth electronic file, represents the text data quality index of the x-th electronic file, ys(x) represents the total number of characters in the text of the x-th electronic file, tc(x) represents the number of characters deleted due to missing values in the x-th electronic file, sj(x) represents the number of characters added due to missing values in the x-th electronic file, rc(x) represents the number of unconventional characters removed due to regular cleaning in the x-th electronic file, b is a constant, It represents the number of stop word characters removed from the xth electronic file, e is a natural constant, k1 represents the missing value deletion weight coefficient, k2 represents the missing value addition weight coefficient, and k3 represents the unconventional weight coefficient.
[0035] Preset specific requirements for the electronic archive image resolution, for example, requiring the electronic archive image resolution to be 1080×1080 or 1680×1680, determine the resolution, clarity, and uniformity of each image in the electronic archive image data, and comprehensively set the electronic archive image data quality score. The specific formula is as follows: ; Among them, x represents the xth electronic file, represents the image data quality score of the x-th electronic file, k represents the k-th image, N(x) represents that the x-th electronic file contains a total of N(x) images, represents the resolution of image k, f is the specific requirement for the resolution of electronic archive images, I[*] is the conditional function, when condition * is met, I[*] is equal to 1, when condition * is not met, I[*] is equal to 0, qx(k) represents the clarity of the kth image, represents the uniformity of the k-th image.
[0036] Determine the video frame data and audio data extracted from the electronic archive video data, integrate the image data quality score and the audio data quality score of the video frame data, and set the electronic archive video data quality score. The specific calculation formula is as follows: ; Among them, x represents the xth electronic file, represents the video data quality score of the x-th electronic file, v represents the v-th video, M(x) represents that the x-th electronic file contains a total of M(x) videos, g represents the g-th video frame extracted from the video, Indicates that the vth video has extracted a total of video frames, represents the resolution of the g-th video frame, f is the specific requirement for the electronic archive image resolution, I[*] is the conditional function, when condition * is met, I[*] is equal to 1, when condition * is not met, I[*] is equal to 0, qx(g) represents the clarity of the g-th video frame, Indicates the uniformity of the g-th video frame.
[0037] The electronic archive automatic classification management module realizes automatic classification management of electronic archives based on the quality scores of text data, image data and video data in electronic documents.
[0038] Preset thresholds for electronic document text data quality scores, image data quality scores, and video data quality scores to determine the quality scores of the electronic document text data, image data, and video data to be classified and managed. When the quality scores of the electronic document text data, image data, and video data are all greater than or equal to the corresponding thresholds, the electronic document is judged to have high text standardization, meet image resolution standards, have clear videos, and have excellent audio signal-to-noise ratios. The electronic document is then stored in a distributed database (such as MongoDB Atlas) that supports millisecond-level retrieval and is automatically backed up regularly to multiple data centers. If the quality score of any one of the text, image, and video data of the electronic document is lower than the corresponding threshold, the electronic document is judged to have text missing, image blur, or insufficient video frame rate problems. The electronic document is stored in standard cloud storage (such as AWS S3) and access frequency monitoring is configured to automatically recheck the quality score of the electronic document every T1 period. When the quality scores of any two of the electronic text data, image data, and video data are lower than the corresponding threshold, the electronic document is judged to have garbled text, extremely low image resolution, or substandard video quality. The electronic document is marked as a document to be repaired, pushed to the repair queue, and stored in low-cost storage (such as Alibaba Cloud Storage) with restricted access rights and retained for T2. The electronic document quality score is automatically rechecked, and if the quality score does not improve, the document is automatically cleared. When the quality scores of the electronic text data, image data and video data are all less than the corresponding thresholds, the electronic document is judged to not meet the storage standards and the document is immediately compressed and stored in cold storage (such as Google Coldline). Direct access is prohibited and the document is automatically deleted after T3 time. A summary report is generated for audit before deletion.
[0039] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.
[0040] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. The automatic classification and management method of electronic archives based on big data is characterized by: include: Step 1: Clarify the data content covered by the electronic archives and perform modal decomposition and preprocessing on the original electronic archives; Step 2: Based on the multimodal data of the electronic document, construct a quality score for the text data, image data, and video data in the electronic document; Step 3: Based on the quality scores of text data, image data and video data in electronic documents, automatic classification management of electronic archives is achieved.
2. The method for automatic classification and management of electronic archives based on big data according to claim 1 is characterized in that: Clarify the data content covered by the electronic archives and perform modal decomposition and preprocessing on the original electronic archives. The specific methods are as follows: Clarify the data content covered by electronic archives. According to different data types, the data content covered by electronic archives is divided into text data, image data, structured data and video data for classification and processing. For text data and structured data, use NLP to convert text information of different formats into a unified format. Use regular expressions to identify and remove unit characters in the text. Fill or delete missing values according to the missing value ratio. Delete stop words that appear in the text content by constructing a stop word list. Use OCR technology to extract text content from images, process the text content according to the text data processing method, determine the resolution of each image in the electronic archive through the image processing library, use the Sobel operator to determine the image clarity, calculate the standard deviation of the pixel grayscale values in the electronic archive image data, and obtain the image uniformity; According to the preset frame rate a, video frames are extracted from the video data, the extracted video frames are processed according to the image data processing method, independent audio data is separated from the video data, the audio signal in the audio data is sampled at the preset sampling rate, the sampled audio signal is Fourier transformed, the energy distribution of the spectrum is calculated to obtain the signal power, and in the spectrum, the frequency band with a frequency greater than A is selected as the noise frequency band, the energy of the noise frequency band is selected as the noise power, and the ratio of the audio signal power to the noise power is used as the audio data quality score.
3. The method for automatic classification and management of electronic archives based on big data according to claim 2, characterized in that: Use the Sobel operator to determine the clarity of the image. The specific method is: The Sobel operator is used to calculate the gradient amplitude of the electronic archive image data. The average value of the gradient amplitude of all pixels in the image represents the clarity of the image. The formula is used Represents the clarity of the image, where k represents the kth image, qx(k) represents the clarity of the kth image, Indicates the height of the k-th image in pixels. Indicates the width of the k-th image in pixels, s1 indicates the clarity constant factor, i indicates the i-th pixel coordinate in the vertical direction of the image, and j indicates the j-th pixel coordinate in the horizontal direction of the image. Represents the gradient magnitude of pixel (i, j).
4. The method for automatic classification and management of electronic archives based on big data according to claim 2, characterized in that: Calculate the standard deviation of the pixel grayscale values in the electronic archive image data to obtain the uniformity of the image. The specific method is: Using the formula Represents the uniformity of the image, where k represents the kth image, represents the uniformity of the k-th image, Indicates the height of the k-th image in pixels. Indicates the width of the k-th image in pixels, s2 indicates the uniformity constant factor, i indicates the i-th pixel coordinate in the vertical direction of the image, j indicates the j-th pixel coordinate in the horizontal direction of the image, represents the grayscale value of pixel (i, j), Represents the mean grayscale value of all pixels in the kth image.
5. The method for automatic classification and management of electronic archives based on big data according to claim 1 is characterized in that: Construct a quality score for text data in electronic documents. The specific method is as follows: Using the formula represents the quality score of the electronic archive text data, where x represents the xth electronic archive, represents the text data quality index of the x-th electronic file, ys(x) represents the total number of characters in the text of the x-th electronic file, tc(x) represents the number of characters deleted due to missing values in the x-th electronic file, sj(x) represents the number of characters added due to missing values in the x-th electronic file, rc(x) represents the number of unconventional characters removed due to regular cleaning in the x-th electronic file, b is a constant, It represents the number of stop word characters removed from the xth electronic file, e is a natural constant, k1 represents the missing value deletion weight coefficient, k2 represents the missing value addition weight coefficient, and k3 represents the unconventional weight coefficient.
6. The method for automatic classification and management of electronic archives based on big data according to claim 1 is characterized in that: Construct a quality score for image data in an electronic document. The specific method is as follows: Preset the specific requirements for the resolution of electronic archive images, determine the resolution, clarity and uniformity of each image in the electronic archive image data, and use the formula represents the quality score of electronic archive image data, where x represents the xth electronic archive, represents the image data quality score of the x-th electronic file, k represents the k-th image, N(x) represents that the x-th electronic file contains a total of N(x) images, represents the resolution of image k, f is the specific requirement for the resolution of electronic archive images, I[*] is the conditional function, when condition * is met, I[*] is equal to 1, when condition * is not met, I[*] is equal to 0, qx(k) represents the clarity of the kth image, represents the uniformity of the k-th image.
7. The method for automatic classification and management of electronic archives based on big data according to claim 1, characterized in that: Construct a quality score for video data in an electronic document. The specific method is as follows: Using the formula represents the quality score of the electronic archive video data, where x represents the xth electronic archive, represents the video data quality score of the x-th electronic file, v represents the v-th video, M(x) represents that the x-th electronic file contains a total of M(x) videos, g represents the g-th video frame extracted from the video, Indicates that the vth video has extracted a total of video frames, represents the resolution of the g-th video frame, f is the specific requirement for the electronic archive image resolution, I[*] is the conditional function, when condition * is met, I[*] is equal to 1, when condition * is not met, I[*] is equal to 0, qx(g) represents the clarity of the g-th video frame, Indicates the uniformity of the g-th video frame.
8. The method for automatic classification and management of electronic archives based on big data according to claim 1 is characterized in that: Based on the quality scores of text data, image data, and video data in electronic documents, automatic classification and management of electronic archives are achieved. The specific method is as follows: Presetting electronic document text data quality score thresholds, image data quality score thresholds, and video data quality score thresholds, determining the quality scores of the electronic document text data, image data, and video data to be classified and managed, and when the text data, image data, and video data quality scores of the electronic document are all greater than or equal to the corresponding thresholds, storing the electronic document in a distributed database that supports millisecond-level retrieval and automatically backing it up to multiple data centers on a regular basis; When the quality score of any one of the text data, image data, and video data of the electronic document is lower than the corresponding threshold, the electronic document is stored in the standard cloud storage, and access frequency monitoring is configured to automatically recheck the quality score of the electronic document every T1 period. When the quality scores of any two of the electronic text data, image data, and video data are less than the corresponding threshold, the electronic document is marked as a document to be repaired, pushed to the repair queue, stored in low-cost storage, with restricted access rights, and retained for T2 time. The quality score of the electronic document is automatically rechecked, and the document is automatically cleared if the quality score does not improve; When the quality scores of the electronic text data, image data and video data are all less than the corresponding thresholds, the document will be immediately compressed and stored in cold storage, direct access will be prohibited, and it will be automatically deleted after T3 time. A summary report will be generated for audit before deletion.
9. An automatic classification and management system for electronic archives based on big data, applied to the automatic classification and management method for electronic archives based on big data according to any one of claims 1 to 8, characterized in that: include: The electronic archive data content processing module is used to clarify the data content covered by the electronic archive and perform modal decomposition and preprocessing on the original electronic archive; The electronic archive quality score setting module is used to construct the quality score of the text data, image data and video data in the electronic document based on the multimodal data of the electronic document; The electronic archive automatic classification management module realizes automatic classification management of electronic archives based on the quality scores of text data, image data and video data in electronic documents.