Digital archives intelligent information mining method

Abstract

The application discloses a kind of digital archives intelligent information mining method, it is related to file information processing technical field, the present application is evaluated data quality by a series of quantitative indicators including consistency index, integrity index, importance index and comprehensive importance index, which effectively reduces subjective judgment when manual screening, make data screening process more objective, reliable and scientific, simultaneously, by combining quantitative indicators with comprehensive weight, to automatically generate weighted threshold based on data-driven, the application of this weighted threshold not only can quickly process large-scale data set, to reduce the demand for a large number of manual annotation of manpower, and can effectively filter out high-quality data set, and be applied to subsequent deep learning model training, to significantly improve the output accuracy and stability of model, ensure that more accurate prediction results can be obtained, and based on this more accurate prediction results, more scientific and reasonable business decisions can be made.

Description

Technical Field

[0001] This invention relates to the field of archival information processing technology, and in particular to a method for intelligent information mining of digital archives. Background Technology

[0002] Intelligent information mining of digital archives is a process that uses artificial intelligence and data mining techniques to automatically extract valuable information from large amounts of digital archives. This approach can help organizations and individuals manage and utilize massive datasets more effectively, thereby improving work efficiency, facilitating decision-making, and supporting research activities. Existing intelligent information mining of digital archives typically follows a series of steps to process and analyze data to extract valuable information. The following are the specific implementation steps and their target results:

[0003] First, raw data is collected from various sources such as databases and file systems and cleaned, including noise removal, missing value imputation, and format conversion, to ensure data quality. Next, relevant features are defined and extracted according to the problem requirements. For example, in text mining, this may include TF-IDF weights and word frequency statistics; in image processing, it may include color histograms and edge detection. Then, a suitable machine learning or deep learning model is selected based on the nature of the task, and the model is trained using a labeled training set. This step usually involves parameter tuning. Next, the trained model is applied to new data to generate prediction results or complete specific tasks such as classification and clustering. At the same time, the model performance is evaluated through methods such as cross-validation. Finally, the results, i.e., the prediction results, are used to support decision-making.

[0004] Although existing methods for intelligent information mining from digital archives have achieved certain results, they still face a series of challenges in practical applications, particularly in terms of data quality:

[0005] Data quality issues: To ensure the accuracy of subsequent model outputs, existing technologies, besides iterative training, also employ data screening to ensure that high-quality data can further improve the accuracy of model outputs, thus bringing them closer to the expected goals. However, current data screening methods often rely on manual annotation. While domain experts can assist in data screening to some extent, this manual screening approach is clearly inefficient due to the significant human resource required. Furthermore, screening based on subjective human factors cannot effectively guarantee data quality. Therefore, the current manual data screening method is not only inefficient but also fails to effectively ensure data quality, potentially impacting the accuracy of subsequent model outputs and consequently affecting the accuracy and scientific rigor of final decision-making.

[0006] Therefore, there is an urgent need for a technical solution for intelligent information mining of digital archives. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a method for intelligent information mining from digital archives, specifically comprising the following steps:

[0008] Step S1: Obtain the digital archive dataset to be processed, and preprocess the digital archive dataset to generate a preprocessed digital archive dataset.

[0009] Step S2: Extract all feature columns from the preprocessed digital archive dataset. Each feature column contains at least two data points. Obtain the consistency index, integrity index, and importance index of all data points in each feature column. Based on the importance index of each data point, obtain the comprehensive importance index of all data points. Based on the consistency index, integrity index, and comprehensive importance index, obtain the comprehensive weight of all data points in each feature column.

[0010] Step S2a1: Calculate the mean of all data points contained in each feature column, and calculate the squared deviation of each data point from the mean based on the mean;

[0011] Step S2a2: Obtain the mean of the squared deviations of each data point from the mean;

[0012] Step S2a3: Take the square root of the mean of the squared deviations to obtain the standard deviation of all data points contained in each feature column;

[0013] Step S2a4: Convert the standard deviation of all data points in each feature column into a consistency score, and perform standardization on the consistency score to obtain the consistency index of all data points in each feature column.

[0014] Step S2b1: Count the number of all data points contained in each feature column;

[0015] Step S2b2: Detect the missing values ​​of each data point in each feature column and count the number of data points with missing values ​​in each feature column;

[0016] Step S2b3: Based on the number of all data points in each feature column and the number of data points with missing values ​​in each feature column, obtain the proportion of missing values ​​for all data points in each feature column;

[0017] Step S2b4: Convert the proportion of missing values ​​of all data points in each feature column into an integrity score, and perform standardization on the integrity score to obtain the integrity index of all data points in each feature column.

[0018] Step S2c1: Analyze the importance of each data point contained in each feature column, and assign an importance score to each data point based on the analysis results;

[0019] Step S2c2: Perform standardization on the importance scores of all data points in each feature column to obtain the importance index of each data point in each feature column;

[0020] Step S2c3: Combine the importance index of each data point in each feature column and perform mean processing to obtain the comprehensive importance index of all data points in each feature column;

[0021] Step S2d1: Assign initial weights to all data points in each feature column based on the comprehensive importance index;

[0022] Step S2d2: Based on the consistency index and the integrity index of all data points in each feature column, obtain the secondary weight of all data points in each feature column.

[0023] Step S2d3: Adjust the initial weights according to the secondary weights to obtain the comprehensive weights of all data points contained in each feature column;

[0024] Step S3: Based on the consistency index, completeness index, importance index, comprehensive weight, and comprehensive importance index, obtain the weighted threshold of all data points contained in each feature column;

[0025] The formula for calculating the weighted threshold of all data points in each feature column is as follows:

[0026] ;

[0027] Among them, T i The weighted threshold representing all data points contained in the i-th feature column; This represents the consistency index of all data points contained in the i-th feature column; E represents the completeness index of all data points contained in the i-th feature column; i O represents the overall importance index of all data points contained in the i-th feature column; ij W represents the importance index of the j-th data point contained in the i-th feature column; i N represents the combined weight of all data points contained in the i-th feature column;i represents the number of data points in the i-th feature column; α, β, γ, and δ represent the weight coefficients of the consistency index, the completeness index, the comprehensive importance index, and the importance index, respectively.

[0028] Step S4: Determine the importance index of each data point in each feature column using a weighted threshold of all data points in each feature column to obtain a high-quality dataset, including:

[0029] Step S41: Normalize the weighted threshold of all data points in each feature column and the importance index of each data point in each feature column;

[0030] Step S42: Determine the importance index of each data point using the weighted threshold after normalization.

[0031] Step S43: Determine the importance index of the current data point in the current feature column according to the weighted threshold. If the importance index of the current data point in the current feature column is greater than or equal to the weighted threshold, the current data point is retained; if the importance index of the current data point in the current feature column is less than the weighted threshold, the current data point is removed; until every data point in each feature column has been determined by the weighted threshold.

[0032] Step S44: Define the current feature column after the judgment as a new feature column, and integrate all the acquired new feature columns to form a high-quality dataset;

[0033] Step S5: Train the deep learning model using a high-quality dataset, and make predictions on the digital archive data to be processed based on the trained deep learning model to obtain the prediction results. Make business decisions based on the prediction results.

[0034] The embodiments of the present invention have the following technical effects:

[0035] This invention evaluates data quality using a series of quantitative indicators, including consistency indicators, integrity indicators, importance indices, and comprehensive importance indices. This effectively reduces subjective judgment during manual screening, making the data screening process more objective, reliable, and scientific. Furthermore, by combining quantitative indicators with comprehensive weights, it automatically generates data-driven weighted thresholds. The application of these weighted thresholds not only enables rapid processing of large-scale datasets, significantly improving the speed and efficiency of data screening and reducing the need for extensive manual annotation, but also effectively enhances the scientific rigor and accuracy of data screening. This allows for the selection of high-quality datasets that can be applied to the training of subsequent deep learning models, significantly improving the model's output accuracy and stability. This ensures more accurate prediction results and, based on these more accurate predictions, enables the formulation of more scientific and rational business decisions. Attached Figure Description

[0036] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a digital archive intelligent information mining method provided by an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0039] Example 1: As Figure 1 As shown, the present invention provides a method for intelligent information mining of digital archives, comprising the following steps:

[0040] Step S1: Obtain the digital archive dataset to be processed, and preprocess the digital archive dataset to generate a preprocessed digital archive dataset.

[0041] Regarding the sources of digital archive datasets, the following methods are used: Databases: Extracting digital archive data from relational databases such as NoSQL databases within the organization; File systems: Reading stored digital archive files from local file systems or network file systems such as NAS and SAN, which can be in formats such as CSV, Excel, PDF, and images; Cloud storage: Downloading digital archive data from cloud storage services such as Amazon S3 and Google Cloud Storage; API interfaces: Obtaining real-time or historical digital archive data from external systems through RESTful APIs or other web service interfaces.

[0042] Regarding methods for acquiring digital archive datasets, there are several approaches: Batch export: Use database management tools such as SQL Developer to export data tables to files in batches; ETL tools: Use ETL (Extract, Transform, Load) tools such as Talend to extract data from multiple data sources and integrate them into a unified data warehouse; Manual upload: Users can upload local files to a specified server directory through a graphical interface; Automated scripts: Write shell scripts or use task scheduling tools such as Cron to periodically download data files from cloud storage.

[0043] The steps for preprocessing digital archival datasets include: data cleaning, data integration, and data transformation; data cleaning includes: noise removal, handling missing values, and format conversion; data integration includes: data merging and data deduplication; and data transformation includes: feature engineering.

[0044] Regarding noise removal, for text data, extra spaces, special characters, HTML tags, etc. are removed; for numerical data, outliers are detected and corrected, such as values ​​that are outside the reasonable range; for image data, noise in the image is removed to improve image quality.

[0045] Regarding handling missing values, this includes identifying missing values: checking each field for null values ​​or default values ​​such as "N / A" or "NULL"; filling missing values: for numerical data, the mean, median, or interpolation method can be used to fill missing values; for categorical data, the mode can be used to fill missing values; deleting records with severe missing values: if the proportion of missing values ​​in some records is too high, these records can be deleted.

[0046] Regarding format conversion, this includes unifying all text to lowercase, removing punctuation, and performing stemming or word form restoration; unifying date and time fields to a standard format such as YYYY-MM-DD HH:MM:SS; ensuring that all numeric fields have consistent data types such as integers and floating-point numbers; and converting all images to a uniform size and format such as 224x224 pixel JPEG format.

[0047] Regarding data merging, this includes table joining, which connects data from different tables using a common key such as customer ID or order number to generate a comprehensive table; and file merging, which combines data from multiple files into one file, such as merging multiple CSV files into a large CSV file.

[0048] Regarding data deduplication, this includes duplicate record detection: identifying and deleting completely identical records; and handling near-duplicate records: for records with similar but not completely identical content, fuzzy matching technology can be used for deduplication.

[0049] Feature engineering includes feature selection: selecting relevant feature columns based on problem requirements and removing irrelevant or redundant features for subsequent analysis; and feature construction: constructing new features based on existing features, such as calculating derived features like age and transaction frequency.

[0050] Step S2: Extract all feature columns from the preprocessed digital archive dataset. Each feature column contains at least two data points. Obtain the consistency index, integrity index, and importance index of all data points in each feature column. Based on the importance index of each data point, obtain the comprehensive importance index of all data points. Based on the consistency index, integrity index, and comprehensive importance index, obtain the comprehensive weight of all data points in each feature column.

[0051] Steps S2a1 to S2a4 mainly explain how to obtain the consistency index of all data points contained in each feature column;

[0052] Step S2a1: Calculate the mean of all data points contained in each feature column, and calculate the squared deviation of each data point from the mean based on the mean;

[0053] Calculate the mean of all data points in each feature column using the following formula:

[0054] ;

[0055] in, This represents the mean of all data points contained in the i-th feature column; This represents the number of data points in the i-th feature column; This represents the value of the j-th data point in the i-th feature column;

[0056] Calculate the squared deviation of each data point from the mean using the following formula:

[0057] ;

[0058] in, This represents the squared deviation of each data point from the mean.

[0059] Step S2a2: Obtain the mean of the squared deviations of each data point from the mean;

[0060] The mean of the squared deviations is calculated using the following formula:

[0061] ;

[0062] in, This represents the average of the squared deviations of all data points;

[0063] Step S2a3: Take the square root of the mean of the squared deviations to obtain the standard deviation of all data points in each feature column. ;

[0064] Step S2a4: Convert the standard deviation of all data points in each feature column into a consistency score, and perform standardization on the consistency score to obtain the consistency index of all data points in each feature column. ;

[0065] When performing consistency score transformation, the standard deviation is converted into a consistency score. Generally, the smaller the standard deviation, the higher the consistency. The purpose of standardization is mainly to normalize the consistency score to the [0,1] interval. The formula used is as follows:

[0066] ;

[0067] in, Represents the standard deviation of the i-th feature column. and These represent the minimum and maximum values ​​of the standard deviation, respectively.

[0068] Steps S2b1 to S2b4 mainly explain how to obtain the integrity index of all data points contained in each feature column;

[0069] Step S2b1: Count the number of all data points contained in each feature column;

[0070] Step S2b2: Detect missing values ​​for each data point in each feature column and count the number M of data points with missing values ​​in each feature column. i ;

[0071] Step S2b3: Based on the total number of data points in each feature column and the number of data points with missing values ​​in each feature column, obtain the proportion of missing values ​​for all data points in each feature column; the calculation formula is as follows:

[0072] ;

[0073] Among them, P i This represents the proportion of missing values ​​in the i-th feature column;

[0074] It is worth noting that when calculating the proportion of missing values, the missing value information recorded in the missing value processing step of the preprocessing process should be used first to count the number of missing values ​​in each feature column.

[0075] Step S2b4: Convert the proportion of missing values ​​of all data points in each feature column into an integrity score, and perform standardization on the integrity score to obtain the integrity index of all data points in each feature column.

[0076] The proportion of missing values ​​is converted into a completeness score. Generally, the lower the proportion of missing values, the higher the completeness. The completeness score needs to be normalized to the [0,1] interval. The calculation formula is as follows:

[0077] ;

[0078] in, I represents the integrity index of all data points contained in the i-th feature column. i I represents the completeness score of all data points contained in the i-th feature column. min and I max These represent the minimum and maximum values ​​of the integrity score, respectively.

[0079] Steps S2c1 to S2c3 mainly explain how to obtain the comprehensive importance index of all data points contained in each feature column;

[0080] Step S2c1: Analyze the importance of each data point in each feature column, and assign an importance score L to each data point based on the analysis results. ij ;

[0081] It is worth noting that when performing importance analysis on each data point contained in each feature column, a feature selection algorithm such as random forest is used to analyze the importance of each data point and assign an importance score. This analysis process is more advanced than existing analysis methods and will not be described in detail here.

[0082] Step S2c2: Perform standardization on the importance scores of all data points in each feature column to obtain the importance index of each data point in each feature column;

[0083] The importance score for each data point is normalized to the [0, 1] interval, and the calculation formula is as follows:

[0084] ;

[0085] Among them, O ij L represents the importance index of the j-th data point contained in the i-th feature column. max L min These represent the maximum and minimum importance scores for all data points, respectively.

[0086] Step S2c3: Combine the importance index of each data point in each feature column and perform mean processing to obtain the comprehensive importance index of all data points in each feature column;

[0087] Steps S2d1 to S2d3 mainly explain how to obtain the comprehensive weight of all data points contained in each feature column based on the consistency index, completeness index, and comprehensive importance index.

[0088] Step S2d1: Based on the comprehensive importance index, assign initial weights to all data points in the i-th feature column contained in each feature column. ;

[0089] Step S2d2: Based on the consistency index and integrity index of all data points in each feature column, obtain the secondary weight of all data points in each feature column; the calculation formula is as follows:

[0090] ;

[0091] in, This represents the secondary weight of all data points in the i-th feature column;

[0092] Step S2d3: Adjust the initial weights according to the secondary weights to obtain the comprehensive weights of all data points contained in each feature column. The calculation formula is as follows:

[0093] ;

[0094] Step S3: Based on the consistency index, completeness index, importance index, comprehensive weight, and comprehensive importance index, obtain the weighted threshold of all data points contained in each feature column;

[0095] The formula for calculating the weighted threshold of all data points in each feature column is as follows:

[0096] ;

[0097] Among them, T i C represents the weighted threshold for all data points contained in the i-th feature column; iThis represents the consistency index of all data points contained in the i-th feature column; E represents the completeness index of all data points contained in the i-th feature column; i O represents the overall importance index of all data points contained in the i-th feature column; ij W represents the importance index of the j-th data point contained in the i-th feature column; i N represents the combined weight of all data points contained in the i-th feature column; i The number of data points in the i-th feature column is represented by α, β, γ, and δ, which represent the weight coefficients of the consistency index, the completeness index, the comprehensive importance index, and the importance index, respectively.

[0098] Step S4: Determine the importance index of each data point in each feature column using a weighted threshold of all data points in each feature column to obtain a high-quality dataset, including:

[0099] Step S41: Normalize the weighted threshold of all data points in each feature column and the importance index of each data point in each feature column;

[0100] Step S42: Determine the importance index of each data point using the weighted threshold after normalization.

[0101] Step S43: Determine the importance index of the current data point in the current feature column according to the weighted threshold. If the importance index of the current data point in the current feature column is greater than or equal to the weighted threshold, the current data point is retained; if the importance index of the current data point in the current feature column is less than the weighted threshold, the current data point is removed; until every data point in each feature column has been determined by the weighted threshold.

[0102] Step S44: Define the current feature column after the judgment as a new feature column, and integrate all the acquired new feature columns to form a high-quality dataset;

[0103] Step S5: Train the deep learning model using a high-quality dataset, and make predictions on the digital archive data to be processed based on the trained deep learning model to obtain the prediction results, and make business decisions based on the prediction results.

[0104] The prediction process is existing technology and will not be elaborated upon here. The main focus is on explaining how to make business decisions based on the prediction results. Specifically, the prediction results should first be explained to ensure interpretability. For example, in anomaly detection tasks, it is necessary to explain which features led to the anomaly judgment, and the meaning of the prediction results must be deeply understood and linked to business objectives. Then, business decisions are made based on the explained prediction results, including: Customer segmentation: segmenting customers based on the prediction results to identify high-value customers, potential churn customers, etc.; Personalized services: providing personalized services and marketing strategies for different customer groups; for example, providing exclusive offers and services for high-value customers; Customer retention: taking targeted retention measures for customers predicted to churn, such as offering discounts, improving service quality, etc.; and even developing operational optimization strategies based on the prediction results, etc., which will not be elaborated upon here.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent information mining from digital archives, characterized in that, Includes the following steps: Step S1: Obtain the digital archive dataset to be processed, and preprocess the digital archive dataset to generate a preprocessed digital archive dataset. Step S2: Extract all feature columns from the preprocessed digital archive dataset, obtain the consistency index, integrity index and importance index of all data points in each feature column, and obtain the comprehensive importance index of all data points based on the importance index of each data point. Obtain the comprehensive weight of all data points in each feature column based on the consistency index, integrity index and comprehensive importance index. Step S3: Based on the consistency index, completeness index, importance index, comprehensive weight, and comprehensive importance index, obtain the weighted threshold of all data points contained in each feature column; Step S4: Use the weighted threshold of all data points in each feature column to determine the importance index of each data point in each feature column, and obtain a high-quality dataset; Step S5: Train the deep learning model using a high-quality dataset, and make predictions on the digital archive data to be processed based on the trained deep learning model to obtain the prediction results, and make business decisions based on the prediction results. The process of obtaining the consistency index of all data points contained in each feature column includes: Step S2a1: Calculate the mean of all data points contained in each feature column, and calculate the squared deviation of each data point from the mean based on the mean; Step S2a2: Obtain the mean of the squared deviations of each data point from the mean; Step S2a3: Take the square root of the mean of the squared deviations to obtain the standard deviation of all data points contained in each feature column; Step S2a4: Convert the standard deviation of all data points in each feature column into a consistency score, and perform standardization on the consistency score to obtain the consistency index of all data points in each feature column. The process of obtaining the comprehensive weight of all data points in each feature column based on consistency index, completeness index, and comprehensive importance index includes: Step S2d1: Assign initial weights to all data points in each feature column based on the comprehensive importance index; Step S2d2: Based on the consistency index and the integrity index of all data points in each feature column, obtain the secondary weight of all data points in each feature column. Step S2d3: Adjust the initial weights according to the secondary weights to obtain the comprehensive weights of all data points contained in each feature column; The formula for calculating the weighted threshold is: ； Among them, T i C represents the weighted threshold for all data points contained in the i-th feature column; i This represents the consistency index of all data points contained in the i-th feature column; E represents the completeness index of all data points contained in the i-th feature column; i O represents the overall importance index of all data points contained in the i-th feature column; ij W represents the importance index of the j-th data point contained in the i-th feature column; i N represents the combined weight of all data points contained in the i-th feature column; i α represents the number of data points in the i-th feature column; α, β, γ, and δ represent the weight coefficients of the consistency index, the completeness index, the comprehensive importance index, and the importance index, respectively.

2. The method for intelligent information mining of digital archives according to claim 1, characterized in that, The process of obtaining the integrity index of all data points contained in each feature column includes: Step S2b1: Count the number of all data points contained in each feature column; Step S2b2: Detect the missing values ​​of each data point in each feature column and count the number of data points with missing values ​​in each feature column; Step S2b3: Based on the number of all data points in each feature column and the number of data points with missing values ​​in each feature column, obtain the proportion of missing values ​​for all data points in each feature column; Step S2b4: Convert the proportion of missing values ​​of all data points in each feature column into an integrity score, and perform standardization on the integrity score to obtain the integrity index of all data points in each feature column.

3. The method for intelligent information mining of digital archives according to claim 1, characterized in that, The importance index of each data point in each feature column, calculated using a weighted threshold of all data points in each feature column, includes: Step S2c1: Analyze the importance of each data point contained in each feature column, and assign an importance score to each data point based on the analysis results; Step S2c2: Perform standardization on the importance scores of all data points in each feature column to obtain the importance index of each data point in each feature column; Step S2c3: Combine the importance index of each data point in each feature column and perform mean averaging to obtain the comprehensive importance index of all data points in each feature column.

4. The method for intelligent information mining of digital archives according to claim 1, characterized in that, The importance index of each data point in each feature column is determined by using a weighted threshold of all data points in each feature column to obtain a high-quality dataset, including: Step S41: Normalize the weighted threshold of all data points in each feature column and the importance index of each data point in each feature column; Step S42: Determine the importance index of each data point using the weighted threshold after normalization. Step S43: Determine the importance index of the current data point in the current feature column according to the weighted threshold. If the importance index of the current data point in the current feature column is greater than or equal to the weighted threshold, the current data point is retained; if the importance index of the current data point in the current feature column is less than the weighted threshold, the current data point is removed; until every data point in each feature column has been determined by the weighted threshold. Step S44: Define the current feature column after the judgment as a new feature column, and integrate all the acquired new feature columns to form a high-quality dataset.

Images