Specific data management and analysis system based on data registration

Abstract

The invention discloses a specific data management and analysis system based on data registration, relates to the technical field of data management, and aims to solve the problems that data integration between a data source and a system may be a complex task, and data formats, structures and semantics differences of different data sources may cause difficulties and errors in the data integration process. According to the method, keyword matching is carried out on the data by effectively improving the data quality and consistency, reducing the time and complexity of data preprocessing, constructing a keyword list, carrying out word frequency and part-of-speech feature analysis, carrying out a keyword matching algorithm and the like, so that the data quality and accuracy are directly reduced. According to the method, the accuracy and efficiency of data retrieval are improved, meanwhile, the matched data can be marked and classified, a data modeling module and a data division unit can be used, model training, parameter estimation and optimization can be carried out, and the accuracy and prediction capability of data analysis are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a specific data management and analysis system based on data registration. Background Art

[0002] Specific data management and analysis systems generally refer to data management and analysis solutions designed and developed for specific fields or industries. These systems are designed to help organizations effectively collect, store, process and analyze specific types of data to support decision making, problem solving and business optimization.

[0003] Nowadays, specific data management and analysis systems rely on high-quality data, but the data may be affected by errors, omissions, duplications or inconsistencies. These problems may lead to inaccurate analysis results and affect the correctness of decision-making. In addition, data integration between multiple data sources and systems may be a complex task. The data format, structure and semantic differences of different data sources may lead to difficulties and errors in the data integration process, which will directly lead to a reduction in data quality and accuracy.

[0004] In view of the above problems, a specific data management and analysis system based on data registration is proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a specific data management and analysis system based on data registration, which solves the problem of reduced data quality and accuracy in the background technology.

[0006] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a specific data management and analysis system based on data registration, comprising:

[0007] S1, data collection and acquisition module, collects and records variables and features, and determines appropriate data collection methods based on needs and available resources, cleans the collected data, detects and handles missing values ​​in the acquired data before further analysis, deletes data rows containing missing values, detects and handles outliers in the data, converts data into appropriate formats and data types for subsequent analysis, and standardizes the data to eliminate differences in units, proportions, and ranges;

[0008] Build a keyword list module, pre-process the data to be matched, extract features for keyword matching from the data, analyze word frequency and part-of-speech features, and mark them;

[0009] Keyword matching module: For data matching keywords, you can mark them as needed, add specific tags, comments and classification tags, and further organize, filter and aggregate the marked data;

[0010] S2, data modeling, determine data entities, describe entities and attribute relationships in the data model, normalize data according to the normalization principle, eliminate redundancy, duplication and inconsistency, and create and implement data models in the selected database management system;

[0011] The data division unit trains the model on the training set and estimates the model parameters according to the selected statistical analysis method. In the regression analysis, the data of independent variables and dependent variables are collected and divided into training sets and test sets. According to the test results, the model is optimized and improved.

[0012] S3, the data result visualization module, develops data charts, supports users to filter and sort data, provides linkage between charts, and organizes the displayed charts to ensure data quality and consistency and store the data in a suitable database.

[0013] Preferably, the input unit is an original data set (including text data to be processed) and a keyword list (XYZ), and the output is a data set marked as A;

[0014] Keyword matching algorithm:

[0015] Preprocess the data set, including text cleaning and word segmentation;

[0016] Initialize an empty list A_data to store data marked as A; traverse each data in the data set: a. Perform keyword matching on the current data: match the terms in the data with the keyword list one by one;

[0017] If a matching keyword is found, the data is marked as A and added to the A_data list; b. Continue to traverse the next data; return the A_data list as the output result, which contains all the data marked as A.

[0018] Preferably, the data processing mark A cleans the data by removing or replacing special characters and punctuation marks in the text with blank characters, and converting the letters in the text to lowercase or uppercase to eliminate the differences caused by uppercase and lowercase letters. The data contains HTML tags or URL links for filtering and removal.

[0019] Preferably, the data of data processing A can be normalized for text containing dates, times or numbers, and the format can be unified to improve consistency. Existing outliers or erroneous data can be deleted, repaired or marked according to business rules, and the data file can be converted into CSV or JSON format as required.

[0020] Preferably, extract features and target variables from the data marked as A, use the feature variables as independent variables and the target variables as dependent variables, and find the minimum value min_va l and the maximum value max_va of the feature variables for each feature variable x;

[0021] Normalization formula: x_normalized = (x-min_val) / (max_val-min_val);

[0022] Calculate the mean mean_value and standard deviation stddev of the feature variable;

[0023] For each feature variable x;

[0024] Standardization processing formula: x_standardized = (x-mean_value) / stddev.

[0025] Preferably, diverse data is collected as modeling data, and the data is divided into a training set, a validation set, and a test set, wherein 70% of the data is used for training, 15% of the data is used for validation, and 15% of the data is used for testing.

[0026] Preferably, based on the data of the training set, a suitable regression model is established and model parameters are determined, and the established model is evaluated to verify its prediction performance;

[0027] Establish regression model algorithm formula:

[0028] Linear regression: Linear regression establishes the relationship between the independent variable x and the dependent variable Y through a linear relationship. Its algorithm formula is:

[0029] Y=β0+β1x1+β2x2+…+βnxn+ε

[0030] Among them, Y represents the dependent variable, x1, x2…x represent the independent variables, β0, β1, β2,…, β represent the coefficients of the model, and ε represents the error term.

[0031] Preferably, the trained model is verified using a validation set, the validation set is input into the trained model to obtain the model's prediction results, and then the model's prediction results are compared with the true labels of the validation set to evaluate the performance of the model, and the model's hyperparameters are adjusted based on the evaluation results of the validation set.

[0032] Preferably, the original data set is represented as a feature matrix X and a target vector Y, where X contains the feature information of the training samples and Y contains the corresponding target labels. The data set is divided into a training set and a validation set according to a certain ratio, and the division is performed using cross-validation.

[0033] Preferably, the parameters (weights and biases) of the model are assigned initial values, the feature data of the training set is input into the model, the prediction result is obtained by the forward propagation calculation of the model, the prediction result of the model is compared with the actual label of the training set, the value of the loss function is calculated, the prediction error of the model is measured, the gradient of the loss function with respect to the model parameters is calculated using the back propagation algorithm, and the parameters of the model are updated according to the direction of the gradient of the loss function using the optimization algorithm, and the process of forward propagation, loss calculation, back propagation and parameter update is repeated until the convergence condition is reached and stopped.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. A specific data management and analysis system based on data registration provided by the present invention effectively improves data quality and consistency by performing standardization, normalization, standardization and cleaning on data, reduces the time and complexity of data preprocessing, and realizes keyword matching of data by constructing a keyword list, performing word frequency, part-of-speech feature analysis and keyword matching algorithm, thereby improving the accuracy and efficiency of data retrieval, and can also mark and classify the matched data. The data modeling module and the data partitioning unit can perform model training, parameter estimation and optimization, further improving the accuracy and predictive ability of data analysis, supporting users to filter, sort and link data, and visualizing data results to improve data readability and operability. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the data analysis process of the present invention;

[0037] Figure 2 It is a schematic diagram of the data partitioning process of the present invention;

[0038] Figure 3 This is a schematic diagram of the data modeling operation steps of the present invention;

[0039] Figure 4 This is a mind map of the keyword matching algorithm of the present invention;

[0040] Figure 5 This is a schematic diagram of the operation flow of the visualization module of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings.

[0043] Combination Figure 1-Figure 5 The present invention provides a data registration-based specific data management and analysis system, comprising:

[0044] S1, data collection and acquisition module, collect and record variables and features, and determine the appropriate data collection method based on needs and available resources, clean the collected data, detect and process missing values ​​in the acquired data before further analysis, delete data rows containing missing values, detect and process outliers in the data, convert data into appropriate formats and data types for subsequent analysis, standardize the data to eliminate differences in units, proportions, and ranges, delete data rows containing missing values ​​during missing value processing, if the proportion of missing values ​​is small and has little impact on the overall analysis, you can directly delete the data rows containing missing values, if the proportion of missing values ​​is large or has important significance for the analysis, specific fields or samples of missing values, rescan the original data to find relevant information, focus on keywords or features that may be related to missing values ​​based on the rescanned data, make appropriate inferences or estimates based on the extracted keywords or features, combined with existing data, to fill in missing values, after filling in missing values, verify and evaluate the data to ensure that the supplemented values ​​are consistent with the overall distribution characteristics and rationality of the data. You can check the statistical indicators of the data, visualize the results, or compare them with other known information;

[0045] Build a keyword list module, pre-process the data to be matched, extract features for keyword matching from the data, analyze word frequency and part-of-speech features, and mark them. When processing specific data with a keyword list, use TF-IDF combined with word frequency and inverse document frequency for calculation:

[0046] The calculation formula of IDF is as follows:

[0047] IDF(w)=log(N / (df(w)+1));

[0048] Where N represents the total number of documents in the text set, and df(w) represents the number of documents containing word w.

[0049] The calculation formula of TF-IDF is as follows:

[0050] TF-IDF(w,d)=TF(w,d)*IDF(w);

[0051] Among them, w represents a word and d represents a document;

[0052] Keyword matching module: For data matching keywords, you can mark them as needed, add specific tags, comments and classification tags, and further organize, filter and aggregate the marked data;

[0053] Data processing: Data cleaning of the tag A, special characters and punctuation marks in the text are removed or replaced with blank characters, and letters in the text are uniformly converted to lowercase or uppercase to eliminate the differences caused by uppercase and lowercase letters. HTML tags or URL links in the data are filtered and removed;

[0054] Input unit, original data set (including text data to be processed) keyword list (XYZ), output data set marked as A;

[0055] Keyword matching algorithm:

[0056] Preprocess the data set, including text cleaning and word segmentation;

[0057] Initialize an empty list A_data to store data marked as A; traverse each data in the data set: a. Perform keyword matching on the current data: match the terms in the data with the keyword list one by one;

[0058] If a matching keyword is found, the data is marked as A and added to the A_data list; b. Continue to traverse the next data; return the A_data list as the output result, which contains all the data marked as A;

[0059] Data processing A's data can be normalized for text containing dates, times, or numbers, and the format can be unified to improve consistency. For outliers or erroneous data, these data can be deleted, repaired, or marked according to business rules. Data files can be converted to CSV or JSON formats as required.

[0060] Extract features and target variables from the data marked as A, use the feature variables as independent variables and the target variables as dependent variables, and find the minimum value min_va l and the maximum value max_va of the feature variables for each feature variable x;

[0061] Normalization formula: x_normalized = (x-min_val) / (max_val-min_val);

[0062] Calculate the mean mean_value and standard deviation stddev of the feature variable;

[0063] For each feature variable x;

[0064] Standardization processing formula: x_standardized = (x-mean_value) / stddev.

[0065] By matching with a pre-set keyword list, data containing specific keywords can be found. This method can be used in application scenarios such as classification, filtering, and searching to screen specific types of data files.

[0066] For the data marked as A, further processing can be performed to normalize information such as date, time, and numbers, and to handle outliers or erroneous data. At the same time, the characteristic variables and target variables of the data marked as A can be extracted and normalized to facilitate subsequent data analysis and modeling. The keyword matching algorithm can help us process text data quickly and efficiently, discover potential key information and features, and provide strong support for subsequent data analysis and decision-making;

[0067] S2, data modeling, determine data entities, describe entities and attribute relationships in the data model, normalize data according to the normalization principle, eliminate redundancy, duplication and inconsistency, and create and implement data models in the selected database management system;

[0068] Collect diverse data as modeling data and divide the data into training set, validation set and test set, where 70% of the data is used for training, 15% for validation and 15% for testing;

[0069] The data division unit trains the model on the training set and estimates the model parameters according to the selected statistical analysis method. In the regression analysis, the data of independent variables and dependent variables are collected and divided into training sets and test sets. According to the test results, the model is optimized and improved.

[0070] Based on the data of the training set, a suitable regression model is established and the model parameters are determined, and the established model is evaluated to verify its prediction performance;

[0071] Establish regression model algorithm formula:

[0072] Linear regression: Linear regression establishes the relationship between the independent variable x and the dependent variable Y through a linear relationship. Its algorithm formula is:

[0073] Y=β0+β1x1+β2x2+…+βnxn+ε

[0074] Among them, Y represents the dependent variable, x1, x2…x represent the independent variables, β0, β1, β2, …, β represent the coefficients of the model, and ε represents the error term;

[0075] Use the validation set to validate the trained model. Input the validation set into the trained model to obtain the model's prediction results. Then, compare the model's prediction results with the true labels of the validation set to evaluate the model's performance. Adjust the model's hyperparameters based on the validation set's evaluation results.

[0076] The original data set is represented as a feature matrix X and a target vector Y, where X contains the feature information of the training sample and Y contains the corresponding target label. The data set is divided into a training set and a validation set according to a certain ratio, and cross-validation is used for division.

[0077] Assign initial values ​​to the parameters (weights and biases) of the model, input the feature data of the training set into the model, obtain the prediction results through the forward propagation calculation of the model, compare the prediction results of the model with the actual labels of the training set, calculate the value of the loss function, measure the prediction error of the model, use the backpropagation algorithm to calculate the gradient of the loss function to the model parameters, and use the optimization algorithm to update the parameters of the model according to the direction of the gradient of the loss function. Repeat the process of forward propagation, loss calculation, backpropagation and parameter update until the convergence condition is reached;

[0078] First, the regression model is used to predict the validation set to obtain the predicted value of the model.

[0079] The predicted value of the model is represented as P, and the true label of the validation set is represented as T. For each sample, the difference between the predicted value and the true label is calculated, and the square is obtained by squaring it. It can be expressed as E = (PT)2.

[0080] Calculate the mean of the squared error, that is, add the squared errors of all samples and divide by the number of samples to get the mean square error (MSE). MSE = Σ(E) / n.

[0081] Finally, take the square root of the mean square error to get the root mean square error (RMSE), that is, RMSE = √(MSE).

[0082] By comparing the root mean square error of the predicted results with the true labels of the validation set, the prediction ability of the model can be evaluated. A smaller RMSE value indicates that the predicted results of the model are closer to the true labels and the model performance is better.

[0083] S3, data result visualization module, develops data charts, supports users to filter and sort data, provides linkage between charts, and organizes displayed charts to ensure data quality and consistency, and stores data in a suitable database.

[0084] Data preparation stage:

[0085] a. Identify the data sets that need to be displayed and analyzed and store them in a suitable MySQL database.

[0086] b. Design database table structure and fields to ensure data consistency and integrity.

[0087] Data query and filtering:

[0088] a. The user selects the data set to be queried and displayed on the interface, and provides filtering conditions such as time range and geographic location.

[0089] b. Based on the filtering conditions provided by the user, execute the corresponding query operation in the database to obtain the data that meets the conditions.

[0090] Data sorting and organization:

[0091] a. Based on user needs, allow users to select the fields and order for sorting query results.

[0092] b. Sort the query results to better display and analyze the data.

[0093] Chart generation:

[0094] a. Design the corresponding chart style and layout according to the display method selected by the user, such as bar chart, line chart or pie chart.

[0095] b. Use appropriate data visualization tools to generate charts and present the query results on the charts.

[0096] Chart linkage:

[0097] a. For charts that need to be linked, trigger data linkage events based on user interactions.

[0098] b. Update the data displayed in other related charts based on linkage events to maintain consistency and relevance between the data.

[0099] Data Storage:

[0100] a. If the user needs to save the query results or charts, provide a save function to store the data in a suitable CSV format and store it in the database.

[0101] b. During the storage process, ensure the integrity and consistency of the data and perform necessary data cleaning and processing.

[0102] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0103] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A specific data management and analysis system based on data registration, characterized in that: include S1, data collection and acquisition module, collects and records variables and features, and determines appropriate data collection methods based on needs and available resources, cleans the collected data, detects and handles missing values ​​in the acquired data before further analysis, deletes data rows containing missing values, detects and handles outliers in the data, converts data into appropriate formats and data types for subsequent analysis, and standardizes the data to eliminate differences in units, proportions, and ranges; Build a keyword list module, pre-process the data to be matched, extract features for keyword matching from the data, analyze word frequency and part-of-speech features, and mark them; Keyword matching module: For data matching keywords, you can mark them as needed, add specific tags, comments and classification tags, and further organize, filter and aggregate the marked data; S2, data modeling, determine data entities, describe entities and attribute relationships in the data model, normalize data according to the normalization principle, eliminate redundancy, duplication and inconsistency, and create and implement data models in the selected database management system; The data division unit trains the model on the training set and estimates the model parameters according to the selected statistical analysis method. In the regression analysis, the data of independent variables and dependent variables are collected and divided into training sets and test sets. According to the test results, the model is optimized and improved. S3, the data result visualization module, develops data charts, supports users to filter and sort data, provides linkage between charts, and organizes the displayed charts to ensure data quality and consistency and store the data in a suitable database.

2. A data registration-based specific data management and analysis system according to claim 1, characterized in that: Input unit, the original data set contains a list of text data keywords that need to be processed, and the output is a data set marked as A; Keyword matching algorithm: Preprocess the data set, including text cleaning and word segmentation; Initialize an empty list A_data to store data marked as A; traverse each data in the data set: a. Perform keyword matching on the current data: match the terms in the data with the keyword list one by one; If a matching keyword is found, the data is marked as A and added to the A_data list; b. Continue to traverse the next data; Returns the A_data list as output, which contains all the data marked as A.

3. A specific data management and analysis system based on data registration according to claim 2, characterized in that: Data processing marks A's data cleaning, removes special characters and punctuation marks in the text or replaces them with blank characters, and converts the letters in the text to lowercase or uppercase to eliminate the differences caused by uppercase and lowercase letters. The data contains HTML tags or URL links for filtering and cleaning.

4. A specific data management and analysis system based on data registration according to claim 2, characterized in that: Data processing A's data can be normalized for text containing dates, times, or numbers, and the format can be unified to improve consistency. Existing outliers or erroneous data can be deleted, repaired, or marked according to business rules. Data files can be converted to CSV or JSON formats as needed.

5. A specific data management and analysis system based on data registration according to claim 2, characterized in that: Extract features and target variables from the data marked as A, use the feature variables as independent variables and the target variables as dependent variables, and find the minimum value min_val and the maximum value max_va of the feature variables for each feature variable x; Normalization formula: x_normalized = (x-min_val) / (max_val-min_val); Calculate the mean value mean_value and standard deviation stddev of the feature variable; For each feature variable x; Standardization processing formula: x_standardized = (x-mean_value) / stddev.

6. A specific data management and analysis system based on data registration according to claim 1, characterized in that: Collect diverse data as modeling data and divide the data into training set, validation set and test set, where 70% of the data is used for training, 15% of the data is used for validation, and 15% of the data is used for testing.

7. A specific data management and analysis system based on data registration according to claim 6, characterized in that: Based on the data of the training set, a suitable regression model is established and the model parameters are determined, and the established model is evaluated to verify its prediction performance; Establish regression model algorithm formula: Linear regression: Linear regression establishes the relationship between the independent variable x and the dependent variable Y through a linear relationship. Its algorithm formula is: Y=β0+β1x1+β2x2+…+βnxn+ε Among them, Y represents the dependent variable, x1, x2…x represent the independent variables, β0, β1, β2,…, β represent the coefficients of the model, and ε represents the error term.

8. A specific data management and analysis system based on data registration according to claim 6, characterized in that: Use the validation set to validate the trained model. Input the validation set into the trained model to obtain the model's prediction results. Then, compare the model's prediction results with the true labels of the validation set to evaluate the model's performance. According to the evaluation results of the validation set, adjust the model's hyperparameters.

9. A specific data management and analysis system based on data registration according to claim 6, characterized in that: The original data set is represented as a feature matrix X and a target vector Y, where X contains the feature information of the training samples and Y contains the corresponding target labels. The data set is divided into a training set and a validation set according to a certain ratio, and cross-validation is used for division.

10. A specific data management and analysis system based on data registration according to claim 9, characterized in that: Assign initial values ​​to the parameter weights and biases of the model, input the feature data of the training set into the model, obtain the prediction results through the forward propagation calculation of the model, compare the prediction results of the model with the actual labels of the training set, calculate the value of the loss function, measure the prediction error of the model, use the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters, and use the optimization algorithm to update the parameters of the model according to the direction of the gradient of the loss function. Repeat the process of forward propagation, loss calculation, backpropagation and parameter update until the convergence condition is reached and stop.

Images