Optimization method and system for automatically identifying and sorting data and storage medium

Abstract

The invention relates to the technical field of data identification and arrangement, in particular to an optimization method and system for automatically identifying and arranging data and a storage medium. Through the automatic data arrangement process, the requirement for manual intervention is remarkably reduced, the data processing efficiency is improved, the time cost is saved, meanwhile, by adopting a machine learning algorithm and a natural language processing technology, recognition and arrangement errors caused by manual operation are reduced, the data quality is improved, the method can adapt to data of different types and formats, and the method is suitable for large-scale popularization and application. The method has good flexibility, can effectively work in a complex data environment, generates a high-quality structured data set through effective preprocessing and quality evaluation, meets service requirements and enhances the reliability of subsequent analysis, and meanwhile, a finally generated arrangement result is optimized to provide a solid foundation for subsequent data analysis and application. And the decision support capability is improved.

Description

Technical Field

[0001] This invention relates to the field of data identification and organization technology, and more specifically, to an optimized method, system, and storage medium for automatically identifying and organizing data. Background Technology

[0002] With the continuous development of information technology, various types of data are emerging in an explosive growth. Especially in the era of big data, the amount of data acquired and processed by various industries is constantly increasing, and the complexity of data organization and management is also rising accordingly. Traditional data organization methods mainly rely on manual processing, which is not only inefficient but also prone to errors and difficult to adapt to the dynamically changing data environment. Therefore, improving the automation and intelligence level of data processing and developing an effective method and system for automatically identifying and organizing data is of great significance for improving work efficiency and data quality.

[0003] While some data processing tools exist on the market, most lack the ability to intelligently identify and optimize data processing. Furthermore, existing systems often fail to perform fast, efficient, and automatic data processing when handling complex data, and they also lack flexibility and scalability.

[0004] Therefore, there is an urgent need for a new optimization method and system to improve the ability and efficiency of automatic data processing. Summary of the Invention

[0005] The purpose of this invention is to provide an optimized method, system, and storage medium for automatically identifying and organizing data, in order to solve the problems mentioned in the background art, where existing systems often fail to achieve fast and efficient automatic organization when processing complex data, and lack flexibility and scalability.

[0006] In a first aspect, to solve the above-mentioned technical problems, embodiments of the present invention provide an optimization method for automatically identifying and organizing data, comprising the following steps:

[0007] S1. Data Collection: Obtain data to be processed from multiple data sources and construct an initial dataset;

[0008] S2. Data preprocessing: Cleaning the initial dataset, including deduplication, filling in missing values, and formatting;

[0009] S3. Feature Extraction: Extract key features from the cleaned data using natural language processing techniques and data mining algorithms;

[0010] S4. Automatic Recognition: Apply machine learning algorithms to analyze the extracted features to achieve automatic recognition of data types and content;

[0011] S5. Data Classification and Organization: Based on the identification results, classify and organize the data to construct a structured dataset;

[0012] S6. Result Optimization: Further optimize the processed data to generate a usable final dataset, and provide a corresponding quality assessment report based on data quality assessment indicators, which satisfy the following formula:

[0013] Q = w1·A + w2·C + w3·U

[0014] Where A is the accuracy score, C is the completeness score, U is the uniqueness score, and w1, w2, and w3 are weighting coefficients.

[0015] In one optional implementation, the specific steps for data collection in step S1 are as follows:

[0016] S11. Determine the types and sources of data to be collected;

[0017] S12. Write or use existing data acquisition programs to connect to various data sources;

[0018] S13. Run the data acquisition program and extract data;

[0019] S14. Store the acquired data in a temporary database or file to form an initial dataset.

[0020] In one optional embodiment, the specific steps of data preprocessing in step S2 are as follows:

[0021] S21. Import the initial dataset;

[0022] S22. Use the appropriate library or tool to remove duplicate data rows;

[0023] S23. Identify missing values ​​and fill or delete them according to a preset strategy;

[0024] S24. Format the data columns to ensure data consistency;

[0025] S25. Generate the cleaned dataset and save it as a new file or database.

[0026] In one optional implementation, the specific steps for feature extraction in step S3 are as follows:

[0027] S31. Import the cleaned dataset;

[0028] S32. For text data, perform word segmentation, stop word removal, and word form restoration.

[0029] S33. Calculate the numerical characteristics, including the mean and standard deviation;

[0030] S34. Extract text features using the TF-IDF formula, whereby the TF-IDF formula is:

[0031]

[0032] Where t is a term, representing a specific word in a document; d is a document, representing a text unit containing multiple terms; N is the total number of documents, referring to the number of documents in the entire text set; DF(t) is the number of documents containing the term, i.e., how many documents the term t has appeared in; and TF(t,d) is the term frequency of term t in document d, representing the number of times term t appears in document d.

[0033] S35. Summarize all extracted features and construct a feature matrix.

[0034] In one optional embodiment, the specific steps for automatic identification in step S4 are as follows:

[0035] S41. Prepare the labeled training dataset;

[0036] S42. Select an appropriate machine learning algorithm;

[0037] S43. Divide the dataset into a training set and a test set;

[0038] S44. Train the model using the training set and evaluate it on the test set using the F1 score formula, wherein the F1 score formula is:

[0039]

[0040] Where Precision is the percentage of samples predicted as positive that are actually positive, and Recall is the percentage of samples that are actually positive that are correctly predicted as positive.

[0041] S45. Use the trained model to identify the features extracted from the cleaned data and generate classification results.

[0042] In one optional implementation, the specific steps for data classification and organization in step S5 are as follows:

[0043] S51. Associate the classification information output by the automatic identification module with the cleaned data;

[0044] S52. Create a new data frame or database table and store data by category;

[0045] S53. Apply sorting, grouping, and summarizing operations to build a structured dataset;

[0046] S54. Confirm the integrity and consistency of the data for each category, and save it as a new dataset format.

[0047] In one optional implementation, the specific steps for optimizing the result in step S6 are as follows:

[0048] S61. Define data quality standards;

[0049] S62. Perform quality checks on the prepared dataset to identify potential problems;

[0050] S63. Optimize the dataset based on the evaluation results;

[0051] S64. Perform integrity scoring and uniqueness scoring, and generate a data quality assessment report, recording various parameters and optimization processes. The integrity scoring formula is as follows: The uniqueness scoring formula is as follows:

[0052] S65. Output the final, usable optimized dataset and corresponding report documents.

[0053] Secondly, embodiments of the present invention also provide an optimization system for automatically identifying and organizing data, comprising:

[0054] The data acquisition module is used to collect data to be processed from multiple data sources in real time or periodically, and to build an initial dataset;

[0055] The data processing module is used to preprocess the initial dataset and perform data cleaning operations, including removing duplicate data, filling in missing values, and formatting.

[0056] The feature extraction module is used to extract key features from cleaned data using natural language processing techniques and data mining algorithms.

[0057] The automatic identification module is used to analyze the extracted features using machine learning algorithms to automatically identify data types, content, and structure, and to label the classification information of the data.

[0058] The data processing module is used to systematically classify and organize the data based on the results of the automatic identification module, and to build a structured dataset.

[0059] The system also includes a results optimization module, which performs quality assessment and further optimization on the processed data and generates a quality assessment report. The results optimization module uses the following formula when performing the quality assessment:

[0060]

[0061] In one optional embodiment, the output of the data acquisition module is connected to the data processing module for transmitting the initial dataset; the output of the data processing module is connected to the feature extraction module for transmitting the cleaned data; the output of the feature extraction module is connected to the automatic recognition module for transmitting the extracted features; the output of the automatic recognition module is connected to the data processing module for transmitting the automatic recognition results; and the output of the data processing module is connected to the result optimization module for transmitting the processed data.

[0062] Thirdly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps included in any of the embodiments of the first aspect.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0064] 1. This invention significantly reduces the need for manual intervention, improves data processing efficiency, and saves time costs through an automated data processing process. At the same time, by adopting machine learning algorithms and natural language processing technology, it reduces identification and processing errors caused by human operation and improves data quality.

[0065] 2. This invention can adapt to different types and formats of data, has good flexibility, and can work effectively in complex data environments. Through effective preprocessing and quality assessment, it generates high-quality structured datasets, meets business needs, and enhances the reliability of subsequent analysis. At the same time, the final processed results are optimized, providing a solid foundation for subsequent data analysis and application, and improving decision support capabilities.

[0066] 3. In this invention, the system can effectively manage and control sensitive data through automatic identification and organization of data, ensuring that data processing complies with relevant laws and regulations, strengthening data security and compliance, and thus effectively reducing the risk of data leakage. Attached Figure Description

[0067] Figure 1 This is a schematic diagram illustrating the steps of the optimized method for automatically identifying and organizing data according to the present invention.

[0068] Figure 2 This is a schematic diagram of the structure of the optimized system for automatically identifying and organizing data according to the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] like Figure 1 As shown, this embodiment of the invention provides an optimization method for automatically identifying and organizing data, including the following steps:

[0071] Step 1: Data collection: Obtain data to be processed from multiple data sources and build an initial dataset.

[0072] 1. Determine the types of data and sources that need to be collected.

[0073] Data type identification: Determine business requirements and analyze the nature of the required data. This includes structured data (such as tabular data in a database, CSV files), semi-structured data (such as XML, JSON files, or text data containing tags), and unstructured data (such as plain text, images, and videos).

[0074] For example, if it is market analysis, it may be necessary to collect sales data, customer feedback, market trends, etc.

[0075] Data source identification: Select appropriate data sources based on data type. For example, API interfaces—data interfaces provided by social media, open-source data platforms, etc.; databases—internal enterprise databases (such as MySQL, PostgreSQL, etc.); documents—document formats such as PDF and Excel.

[0076] For each data source, its access method and connection information must be recorded.

[0077] 2. Write or use existing data acquisition programs to connect to various data sources.

[0078] Program Selection and Preparation: If suitable data acquisition tools (such as web crawlers or ETL tools) are available, these tools should be evaluated to ensure they meet the data acquisition requirements. If no suitable tools are available, a custom data acquisition program can be written using common programming languages ​​(such as Python or Java).

[0079] Connect to various data sources: Write connection code based on the characteristics of each data source.

[0080] For example, an API connection example—using an HTTP request to call the API and retrieve data—can yield data using a RESTful interface according to the following formula:

[0081] [\text{Response}=\text{GET}(\text{API_Endpoint})]

[0082] Database connection example – connecting to a database using a database connection library (such as pymysql or psycopg2 for Python):

[0083] [\text{Connection}=\text{Database_Connector}(\text{host},\text{user},\text{password},\text{database_name})].

[0084] 3. Run the data acquisition program to extract data.

[0085] Program execution: The pre-written data acquisition program is launched to perform data extraction. The extraction frequency (real-time, periodic) and range (full, incremental) can be set according to the characteristics of different data sources.

[0086] Data extraction: For each data source, call the appropriate interface or query statement to read the data.

[0087] API Example – Obtaining the required data may require data filtering or parameter passing, for example:

[0088] [\text{Response}=\text{GET}(\text{API_Endpoint},\text{params}={\text{"date"}:\text{"2023-01-01"}})]

[0089] SQL query example – using SQL to select the desired data from the database:

[0090] [\text{data}=\text{SELECT},\text{*},\text{FROM},\text{sales_data},\text{WHERE},\text{date}>'2023-01-01';].

[0091] 4. Store the acquired data in a temporary database or file to form the initial dataset.

[0092] Temporary storage option: Determine the storage method. You can choose to use temporary files (such as CSV, JSON) or temporary databases (such as SQLite) to store the retrieved data.

[0093] Store data: Save the acquired data to the selected storage medium.

[0094] File storage example – writing data records to a CSV file:

[0095] [\text{dataframe}.to_csv(\text{"initial_dataset.csv"},

[0096] \text{index}=\text{False})]

[0097] Database storage example – storing data in a temporary database:

[0098] [\text{INSERT INTO temp_table(columns)VALUES(data_values)}]

[0099] Ensure that the data can be quickly accessed and queried after it is stored, in order to prepare for subsequent data processing steps.

[0100] By following the steps above, we can better complete the collection work from multiple data sources.

[0101] The second step is data preprocessing: cleaning the initial dataset, including deduplication, filling in missing values, and formatting.

[0102] 1. Import the initial dataset.

[0103] Choose appropriate tools and libraries: Select libraries suitable for your data format, such as the Pandas library in Python, data.table in R, or data processing software (such as Excel or Tableau).

[0104] Importing Datasets: Import data using the appropriate method based on its storage format. Common formats include CSV, Excel, and JSON. Ensure the imported dataset has a clear structure and field definitions for easy subsequent processing.

[0105] 2. Use the appropriate library or tool to remove duplicate data rows.

[0106] Identifying duplicate rows—use appropriate library functions to check the data, such as Pandas' `duplicated()` method. You can determine which columns are used for duplicate checking by setting parameters.

[0107] Perform deduplication: First, use the drop_duplicates() method to remove duplicate data. Then, select the duplicate rows to keep (e.g., keep the first occurrence of the row). Finally, evaluate the state of the dataset and record the total number of rows after deduplication for subsequent comparison.

[0108] 3. Identify missing values ​​and fill or delete them according to preset strategies.

[0109] Missing value identification: Use the isnull() or isna() function to count missing values ​​and output the statistical results to view the missing value situation and decide on the subsequent processing strategy.

[0110] Missing value handling strategy: Select a strategy for filling or deleting missing values ​​based on business needs and data logic.

[0111] Imputation strategy – using mean, median, mode, etc. to impute (numerical features).

[0112] For categorical features, use the mode to fill in the gaps.

[0113] Deletion strategy – Delete rows containing missing values.

[0114] After handling missing values, perform missing value statistics to confirm that the missing values ​​have been processed.

[0115] 4. Format the data columns to ensure data consistency.

[0116] Standardize data format: Perform format conversions for different column data types. For example, format a date column and convert numeric columns to appropriate floating-point types.

[0117] Processing text data: Performing operations such as string trimming and case conversion.

[0118] Ensure consistency in categorized data: Replace inconsistent category labels, for example, unify "male" and "male" into "male".

[0119] 5. Generate the cleaned dataset and save it as a new file or database.

[0120] Determine the save format: Select a suitable data storage format (such as CSV, Excel, database, etc.) based on subsequent needs.

[0121] Save the cleaned dataset:

[0122] Save as a CSV file;

[0123] Save as an Excel file;

[0124] If you choose to store data in a database, you can use SQLAlchemy or related libraries to execute it.

[0125] Confirmation after data saving: Confirm that the data has been successfully saved and perform integrity checks (such as the number of rows, the number of fields, etc.) if necessary.

[0126] Through the above steps, the quality and consistency of the data can be ensured.

[0127] Step 3: Feature extraction: Extract key features from the cleaned data through natural language processing techniques and data mining algorithms.

[0128] 1. Import the cleaned dataset.

[0129] Select suitable tools and libraries: Usually use the Pandas library in Python or similar tools for convenient data manipulation and processing.

[0130] Import the cleaned dataset: Import the data using the corresponding method according to the file format. Then perform a basic view of the data to confirm whether the imported data structure and data type meet the expectations.

[0131] 2. For text data, perform word segmentation, stop word removal, and word form reduction.

[0132] Word segmentation: Split the text into individual words (Tokens). The processing method can choose a suitable NLP library, such as NLTK, spaCy, or jieba.

[0133] Stop word removal: Stop words refer to words with little meaning in text analysis (such as "de", "shi", "zai", etc.), and they need to be removed.

[0134] Word form reduction: For most NLP tasks, different forms of words need to be unified into the standard form.

[0135] 3. Calculate numerical features, including mean and standard deviation.

[0136] Select numerical columns: Determine the numerical feature columns to be calculated, such as sales amount, page views, etc.

[0137] Calculate the mean and standard deviation: Use the mean() and std() methods in the Pandas library to calculate the mean and standard deviation of each numerical column.

[0138] 4. Extract text features.

[0139] Extract text features using the TF-IDF formula, and the TF-IDF formula is:

[0140]

[0141] Where t is a term, representing a specific word in a document; d is a document, representing a text unit containing multiple terms; N is the total number of documents, referring to the number of documents in the entire text set; DF(t) is the number of documents containing the term, i.e., how many documents the term t has appeared in; and TF(t,d) is the term frequency of term t in document d, representing the number of times term t appears in document d.

[0142] Extracting n-gram features: N-gram refers to n consecutive words in a text, which can be achieved by setting the ngram_range parameter.

[0143] 5. Summarize all extracted features and construct a feature matrix.

[0144] Merging numerical and textual features: Use the functions in scipy.sparse or the concat method in Pandas to merge numerical and textual features.

[0145] Transform the matrix to a DataFrame (if needed): If the final feature matrix needs to be converted to PandasDataFrame format, this process must be performed.

[0146] Output feature matrix information: Print the shape of the feature matrix and the first few rows to verify the feature extraction results.

[0147] By following the steps above, we can ensure that the extracted features are used effectively and accurately when training machine learning models.

[0148] Step 4: Automatic Recognition: Apply machine learning algorithms to analyze the extracted features to achieve automatic recognition of data types and content.

[0149] 1. Prepare the labeled training dataset.

[0150] Before training a machine learning model, a well-labeled training dataset needs to be prepared to ensure that the dataset has good quality and representativeness.

[0151] Data collection and processing: Collect relevant data from various sources, such as databases, CSV files, APIs, etc. Ensure the data is relevant to the task at hand and perform appropriate cleaning.

[0152] Data labeling: Label the dataset according to the needs of the task. For example, for a text classification task, it may be necessary to label the text as relevant categories (such as positive, negative, etc.). Labeling can be done manually or using a pre-trained model for initial labeling, followed by manual review.

[0153] Format the dataset: Format the dataset into the input format required by the model. For example, ensure that features and labels are appropriate arrays or DataFrames, typically with labels placed in one column and features in other columns.

[0154] 2. Select appropriate machine learning algorithms for model training based on data characteristics and task requirements.

[0155] Task type determination: Determine whether it is a classification task, a regression task, or a clustering task. For example, text sentiment analysis is a classification task, while house price prediction is a regression task.

[0156] Algorithm selection: Choose a suitable machine learning algorithm based on the task type and data characteristics. Commonly used algorithms include:

[0157] Classification tasks: Decision trees, random forests, support vector machines (SVM), logistic regression, k-nearest neighbors (KNN), neural networks, etc.

[0158] Regression tasks include linear regression, ridge regression, LASSO regression, and support vector regression.

[0159] Importing libraries and instantiating algorithms: Use the appropriate machine learning libraries (such as Scikit-learn, TensorFlow, PyTorch, etc.) and instantiate the model based on the chosen algorithm.

[0160] 3. Divide the dataset into training and test sets to evaluate model performance.

[0161] Choice of split ratio: The dataset is usually randomly divided into training and test sets, with common ratios being 70%-30% or 80%-20%. Cross-validation can be used to increase the model's generalization ability.

[0162] Perform data splitting: Use the train_test_split function in Scikit-learn to perform random splitting.

[0163] 4. Train the model using the training set and evaluate it using the test set.

[0164] Use the training set to train the selected machine learning model and evaluate its performance on the test set.

[0165] Model training: Call the model's fit method to train the model using the training data.

[0166] Model prediction: Use the trained model to make predictions on the test set.

[0167] Performance evaluation: The test set was evaluated using the F1 score formula, which is as follows:

[0168]

[0169] Precision refers to the proportion of samples predicted as positive that are actually positive, while Recall refers to the proportion of samples that are actually positive that are correctly predicted as positive.

[0170] 5. Use the trained model to identify the features extracted from the cleaned data and generate classification results.

[0171] Prepare the data to be predicted: Ensure that the features extracted from the cleaned data are in the same format as the features used during training.

[0172] Make predictions: Use the trained model to predict new inputs.

[0173] Output prediction results: Organize the output prediction results into an easy-to-interpret format.

[0174] By following the steps above, automatic identification of data types and content can be achieved.

[0175] Step 5: Data Classification and Organization: Based on the identification results, classify and organize the data to construct a structured dataset.

[0176] 1. Associate the classification information output by the automatic identification module with the cleaned data.

[0177] Obtain classification information: Assume the classification results output by the previous automatic recognition module are stored in array or DataFrame format. Associate the labels (predicted categories) with the cleaned dataset.

[0178] Adding categorical information to the dataset: Adding categorical information to the cleaned dataset can usually be done using Pandas' assign() function or by directly specifying the columns.

[0179] Check the association results: Confirm the accuracy of the data association and check the contents of the new column.

[0180] 2. Create new data frames or database tables to store data by category.

[0181] Create new data frames: Use Pandas' groupby method or DataFrame's to_csv method to create new data frames based on labels and store data in groups by category.

[0182] Output data for each category: For each category, you can create a new DataFrame and save it, or you can output it directly to a CSV file.

[0183] Create a database table (optional): If you want to store data in a database, you can use the SQLAlchemy library in conjunction with Pandas to write the data into a database table.

[0184] 3. Apply sorting, grouping, and summarizing operations to build a structured dataset.

[0185] Apply sorting operations: For data that has already been categorized, sorting operations can be performed based on a certain feature, such as date, rating, etc.

[0186] Apply grouping and summarizing operations: Use Pandas to perform grouping and summarizing calculations, calculating some statistical characteristics (such as count, mean, etc.) within each category.

[0187] Output summary results: Output the summarized statistical results as a new DataFrame.

[0188] 4. Confirm the integrity and consistency of the data for each category, and save it as a new dataset format.

[0189] Check data integrity: Confirm that the data for each category is complete, such as that there are no missing values.

[0190] Check data consistency: Perform a reasonableness check on the features to ensure that the data type and range of each category are as expected.

[0191] Save data in a new format: Save the cleaned and organized data in a new dataset format, such as CSV or Excel, for easy access and analysis later.

[0192] Documentation steps and results: Record the characteristics of each category of data and the filename to save it for future reference.

[0193] By following the steps above, we can ensure that the implementation proceeds smoothly and generate a structured and high-quality dataset.

[0194] Step 6: Result Optimization: Further optimize the processed data to generate a usable final dataset, and provide a corresponding quality assessment report based on data quality assessment indicators, which satisfy the following formula:

[0195] Q = w1·A + w2·C + w3·U

[0196] Where A is the accuracy score, C is the completeness score, U is the uniqueness score, and w1, w2, and w3 are weighting coefficients.

[0197] 1. Define data quality standards.

[0198] Before optimizing the results, it is necessary to define data quality standards to ensure the scientific validity and effectiveness of subsequent testing and evaluation.

[0199] Determine data quality dimensions: Identify the data quality dimensions that need to be considered.

[0200] For example, accuracy—whether the data is true and accurate; completeness—whether the data is missing important information; consistency—whether the data is consistent across multiple records or systems; timeliness—whether the data is collected and updated within an appropriate time; and uniqueness—whether there are duplicate data records.

[0201] Set standard values: For each quality dimension, set specific standard values. For example, completeness—the proportion of missing values ​​in the dataset should be less than 5%; accuracy—the data accuracy rate should reach 90% or higher; duplication rate—the data duplication rate should be less than 2%.

[0202] Documented standards: The defined standards are recorded in detail in a document for future reference and tracking.

[0203] Data quality standards: Accuracy ≥ 90%; Completeness ≤ 5% null values; Consistency 100%; Timeliness – meets update time requirements; Uniqueness ≤ 2% duplicates.

[0204] 2. Perform quality checks on the prepared dataset to identify potential problems.

[0205] Establish quality inspection tools: Develop or use existing tool scripts to automate data quality inspections.

[0206] Test each quality indicator: check each item against the defined standards.

[0207] Integrity check – Calculate the proportion of missing values.

[0208] Accuracy testing – Accuracy can be tested by comparing with external data sources (such as verifying IDs and values).

[0209] Consistency check – Checking the consistency of the same field across different records.

[0210] Duplicate detection – checking whether there are duplicate records in a dataset.

[0211] Record potential problems: Based on the detection results, record the detection results for each dimension and identify potential problems.

[0212] Completeness: Number of null values; Accuracy: Degree of deviation; Consistency: Number of inconsistent records.

[0213] 3. Optimize the dataset based on the evaluation results.

[0214] After identifying the problems in the data, measures need to be taken to optimize the dataset in order to improve data quality.

[0215] Handling missing values: Take appropriate measures to handle missing data, such as completing, interpolating, or deleting records.

[0216] Improve accuracy: Perform data correction, especially for critical fields, by verifying and correcting them, for example by referencing external data sources.

[0217] Remove duplicates: Delete duplicate records to ensure data uniqueness.

[0218] Record the optimization process: Record every step and result taken during the optimization process for subsequent analysis and auditing.

[0219] 4. Perform integrity and uniqueness scoring, generate a data quality assessment report, and record all parameters and optimization processes. The integrity scoring formula is as follows: The uniqueness scoring formula is as follows:

[0220] Summary evaluation indicators: Integrate the quality inspection and optimization results to summarize the current status of each dimension.

[0221] Describe the optimization process in detail: Record the steps taken during the optimization process, such as methods for handling missing values ​​and data correction approaches, and form a complete optimization report document.

[0222] Save the report document: You can choose to export the evaluation report as a Markdown, PDF, or Word file for later use and sharing.

[0223] 5. Output the final, usable optimized dataset and corresponding report documents.

[0224] Save the optimized dataset: Save the data that has undergone quality testing and optimization as a new CSV, Excel, or other format for later use.

[0225] Generate the final report document: Integrate the data quality assessment report with the optimization process document to form a complete report for relevant stakeholders.

[0226] Clean up your work environment: After completing all operations, you can choose to clean up temporary files and variables to ensure a clean work environment.

[0227] By following the steps above, the results can be effectively optimized, the overall quality of the dataset can be improved, and a complete data quality assessment report can be provided for reference.

[0228] Based on the same inventive concept, embodiments of the present invention also provide an optimization system for automatically identifying and organizing data. This optimization system can achieve the functions corresponding to the aforementioned optimization method for automatically identifying and organizing data. The optimization system can be a hardware structure, a software module, or a hardware structure plus a software module. This optimization system can be implemented by a chip system, which can consist of chips or include chips and other discrete devices. Please refer to [link to previous text]. Figure 2 The diagram below illustrates the structure of an optimization system for automatically identifying and organizing data, as provided in an embodiment of the invention. This system includes a data acquisition module, a data processing module, a feature extraction module, an automatic identification module, a data organization module, and a result optimization module.

[0229] The data acquisition module's output is connected to the data processing module for transmitting the initial dataset. The data processing module's output is connected to the feature extraction module for transmitting cleaned data. The feature extraction module's output is connected to the automatic recognition module for transmitting the extracted features. The automatic recognition module's output is connected to the data processing module for transmitting the automatic recognition results. The data processing module's output is connected to the result optimization module for transmitting the processed data.

[0230] In practical applications, such as Figure 2 As shown, the process begins with a data acquisition module that collects data from multiple data sources in real-time or periodically to build an initial dataset. Then, a data processing module preprocesses the initial dataset, performing data cleaning operations, including removing duplicate data, filling missing values, and formatting. Next, a feature extraction module uses natural language processing techniques and data mining algorithms to extract key features from the cleaned data. An automatic identification module then uses machine learning algorithms to analyze the extracted features, automatically identifying data type, content, and structure, and labeling the data's classification information. Following this, a data processing module systematically classifies and processes the data based on the results of the automatic identification module, constructing a structured dataset. Finally, a result optimization module performs quality assessment and further optimization on the processed data, generating a quality assessment report. The result optimization module uses the following formula for quality assessment:

[0231]

[0232] Finally, the generated quality assessment report is delivered to subsequent applications or users.

[0233] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the functions corresponding to the aforementioned optimization method for automatically identifying and organizing data.

[0234] In summary, this invention significantly reduces the need for manual intervention, improves data processing efficiency, and saves time costs through an automated data preparation process. Furthermore, by employing machine learning algorithms and natural language processing technology, it reduces identification and organization errors caused by human intervention, improves data quality, and is adaptable to different types and formats of data. It exhibits excellent flexibility and can work effectively in complex data environments. Through effective preprocessing and quality assessment, it generates high-quality structured datasets that meet business needs and enhance the reliability of subsequent analysis. The final optimized data preparation results provide a solid foundation for subsequent data analysis and applications, improving decision support capabilities.

[0235] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An optimized method for automatically identifying and organizing data, characterized in that, Includes the following steps: S1. Data Collection: Obtain data to be processed from multiple data sources and construct an initial dataset; S2. Data preprocessing: Cleaning the initial dataset, including deduplication, filling in missing values, and formatting; S3. Feature Extraction: Extract key features from the cleaned data using natural language processing techniques and data mining algorithms; S4. Automatic Recognition: Apply machine learning algorithms to analyze the extracted features to achieve automatic recognition of data types and content; S5. Data Classification and Organization: Based on the identification results, classify and organize the data to construct a structured dataset; S6. Result Optimization: Further optimize the processed data to generate a usable final dataset, and provide a corresponding quality assessment report based on data quality assessment indicators, which satisfy the following formula: Q = w1·A + w2·C + w3·U Where A is the accuracy score, C is the completeness score, U is the uniqueness score, and w1, w2, and w3 are weighting coefficients.

2. The optimized method for automatically identifying and organizing data according to claim 1, characterized in that, The specific steps for data collection in step S1 are as follows: S11. Determine the types and sources of data to be collected; S12. Write or use existing data acquisition programs to connect to various data sources; S13. Run the data acquisition program and extract data; S14. Store the acquired data in a temporary database or file to form an initial dataset.

3. The optimized method for automatically identifying and organizing data according to claim 1, characterized in that, The specific steps for data preprocessing in step S2 are as follows: S21. Import the initial dataset; S22. Use the appropriate library or tool to remove duplicate data rows; S23. Identify missing values ​​and fill or delete them according to a preset strategy; S24. Format the data columns to ensure data consistency; S25. Generate the cleaned dataset and save it as a new file or database.

4. The optimized method for automatically identifying and organizing data according to claim 1, characterized in that, The specific steps for feature extraction in step S3 are as follows: S31. Import the cleaned dataset; S32. For text data, perform word segmentation, stop word removal, and word form restoration. S33. Calculate the numerical characteristics, including the mean and standard deviation; S34. Extract text features using the TF-IDF formula, whereby the TF-IDF formula is: Where t is a term, representing a specific word in a document; d is a document, representing a text unit containing multiple terms; N is the total number of documents, referring to the number of documents in the entire text set; DF(t) is the number of documents containing the term, i.e., how many documents the term t has appeared in; and TF(t,d) is the term frequency of term t in document d, representing the number of times term t appears in document d. S35. Summarize all extracted features and construct a feature matrix.

5. The optimized method for automatically identifying and organizing data according to claim 1, characterized in that, The specific steps for automatic identification in step S4 are as follows: S41. Prepare the labeled training dataset; S42. Select an appropriate machine learning algorithm; S43. Divide the dataset into a training set and a test set; S44. Train the model using the training set and evaluate it on the test set using the F1 score formula, wherein the F1 score formula is: Where Precision is the percentage of samples predicted as positive that are actually positive, and Recall is the percentage of samples that are actually positive that are correctly predicted as positive. S45. Use the trained model to identify the features extracted from the cleaned data and generate classification results.

6. The optimized method for automatically identifying and organizing data according to claim 1, characterized in that, The specific steps for data classification and organization in step S5 are as follows: S51. Associate the classification information output by the automatic identification module with the cleaned data; S52. Create a new data frame or database table and store data by category; S53. Apply sorting, grouping, and summarizing operations to build a structured dataset; S54. Confirm the integrity and consistency of the data for each category, and save it as a new dataset format.

7. The optimized method for automatically identifying and organizing data according to claim 1, characterized in that, The specific steps for optimizing the results in step S6 are as follows: S61. Define data quality standards; S62. Perform quality checks on the prepared dataset to identify potential problems; S63. Optimize the dataset based on the evaluation results; S64. Perform integrity scoring and uniqueness scoring, and generate a data quality assessment report, recording various parameters and optimization processes. The integrity scoring formula is as follows: The uniqueness scoring formula is as follows: S65. Output the final, usable optimized dataset and corresponding report documents.

8. An optimization system for automatically identifying and organizing data, characterized in that, include: The data acquisition module is used to collect data to be processed from multiple data sources in real time or periodically, and to build an initial dataset; The data processing module is used to preprocess the initial dataset and perform data cleaning operations, including removing duplicate data, filling in missing values, and formatting. The feature extraction module is used to extract key features from cleaned data using natural language processing techniques and data mining algorithms. The automatic identification module is used to analyze the extracted features using machine learning algorithms to automatically identify data types, content, and structure, and to label the classification information of the data. The data processing module is used to systematically classify and organize the data based on the results of the automatic identification module, and to build a structured dataset. The system also includes a results optimization module, which performs quality assessment and further optimization on the processed data and generates a quality assessment report. The results optimization module uses the following formula when performing the quality assessment:

9. The optimization system for automatically identifying and organizing data according to claim 8, characterized in that, The output of the data acquisition module is connected to the data processing module for transmitting the initial dataset. The output of the data processing module is connected to the feature extraction module for transmitting the cleaned data. The output of the feature extraction module is connected to the automatic recognition module for transmitting the extracted features. The output of the automatic recognition module is connected to the data processing module for transmitting the automatic recognition results. The output of the data processing module is connected to the result optimization module for transmitting the processed data.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the optimized method for automatically identifying and organizing data as described in any one of claims 1 to 7.

Images