Algorithm auxiliary upgrading system based on big data

Abstract

The invention discloses an algorithm auxiliary upgrading system based on big data, and particularly relates to the technical field of big data algorithms, the algorithm auxiliary upgrading system comprises a data acquisition module, the output end of the data acquisition module is connected with a data processing and storage module, and the output end of the data processing and storage module is connected with a data analysis module. The output end of the data analysis module is connected with a feature engineering module, the output end of the feature engineering module is connected with an algorithm selection module, and the output end of the algorithm selection module is connected with a model engineering module. According to the method, accurate and efficient data analysis and decision support are provided for enterprises and organizations by expanding data sources, improving data quality, accurately selecting the most suitable algorithm, improving model accuracy and efficiency, analyzing data through multiple modules, optimizing feature engineering and efficiently executing model engineering, including training, evaluation, optimization and deployment, and the method is suitable for large-scale popularization and application. The data value can be better mined in the big data age, and the competitiveness is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data algorithm technology, and more specifically, to an algorithm-assisted upgrade system based on big data. Background Art

[0002] In today's big data era, the scale of data is exploding. With the popularization of the Internet and the acceleration of the digitalization process, the amount of data in various fields is accumulating at an alarming rate. Whether it is user interaction data on social media platforms, transaction records on e-commerce websites, experimental data from scientific research institutions, market data in the financial industry, etc., they are constantly enriching the ocean of big data. Enterprises and organizations face massive amounts of data every day, and the scale of these data has far exceeded the capabilities of traditional data processing methods. Taking social media as an example, the amount of text, pictures, videos and other data generated by billions of users around the world every day is so huge that it is unimaginable. The sources of data are also increasingly diversified. Enterprises and organizations are faced with massive amounts of web page data, sensor data, and data in various databases.

[0003] However, these data often have quality problems, such as missing values, duplicate values, and outliers. At the same time, the data formats are also different, which brings great challenges to the effective use of data. Traditional data analysis methods often seem to be powerless when facing big data. On the one hand, traditional data collection methods are relatively simple and it is difficult to obtain comprehensive and diverse data. For example, relying solely on manual data collection is not only inefficient but also limited in scope. On the other hand, traditional data processing and analysis methods are slow and low in accuracy when processing large-scale data. In terms of feature engineering and algorithm selection, traditional methods usually rely on manual experience and lack systematicity and scientificity, which may lead to the selection of features that are not optimized enough and the algorithm is not accurate enough, thus affecting the performance and generalization ability of the model. In terms of model engineering, the traditional model training, evaluation and optimization process is relatively cumbersome and difficult to adapt to the rapid changes and complex requirements in the big data environment. At the same time, the deployment of models also faces many difficulties, such as compatibility issues and performance bottlenecks. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides an algorithm-assisted upgrade system based on big data to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an algorithm-assisted upgrading system based on big data, comprising a data acquisition module, the output end of the data acquisition module is connected to a data processing and storage module, the output end of the data processing and storage module is connected to a data analysis module, the output end of the data analysis module is connected to a feature engineering module, the output end of the feature engineering module is connected to an algorithm selection module, and the output end of the algorithm selection module is connected to a model engineering module;

[0006] The input end of the data acquisition module is respectively connected to a web crawler module and a sensor data acquisition module, the output end of the data acquisition module is connected to a database connection module, the output end of the database connection module is connected to a data processing and storage module, the output end of the data processing and storage module is connected to a data cleaning module, the output end of the data cleaning module is connected to a data conversion module, the output end of the data conversion module is connected to a data annotation module, and the output end of the data annotation module is respectively connected to a data warehouse module and a distributed storage module;

[0007] The output end of the distributed storage module is connected to a data analysis module, and the output end of the data analysis module is respectively connected to a statistical analysis module, a correlation analysis module and a cluster analysis module, and the output ends of the statistical analysis module, the correlation analysis module and the cluster analysis module are all connected to a feature engineering module;

[0008] The output end of the feature engineering module is connected to a feature extraction module, the output end of the feature extraction module is connected to a feature optimization module, the output end of the feature optimization module is connected to a feature construction module, and the output end of the feature construction module is connected to an algorithm selection module.

[0009] Preferably, the output end of the algorithm selection module is respectively connected to a classification algorithm selection module, a regression algorithm selection module and a clustering algorithm selection module.

[0010] Preferably, the output end of the model engineering module is connected to a model training module, the output end of the model training module is connected to a model evaluation module, the output end of the model evaluation module is connected to a model optimization module, and the output end of the model optimization module is connected to a model deployment module.

[0011] Preferably, the web crawler module is used to crawl text, pictures and link information in web pages according to pre-set rules and target websites, the sensor data acquisition module is used to collect data generated by various sensors, the database connection module is used to establish connections with various databases, the data cleaning module is used to deal with noise, missing values, duplicate values ​​and erroneous data problems in the data, and the data conversion module is used to perform format conversion, normalization and standardization operations on the data.

[0012] Preferably, the data annotation module is used to add labels or annotations to the data, including classification labels, timestamps and source tags; the data warehouse module is used to store large amounts of data after integration, cleaning and preprocessing; the distributed storage module improves the capacity, reliability and scalability of data storage by dispersing and storing the data on multiple nodes or servers; the statistical analysis module performs basic statistical calculations on the data, including calculating mean, median, variance and standard deviation statistical indicators to help understand the distribution characteristics, central tendency and degree of dispersion of the data; the association analysis module is used to discover the association relationship between different variables in the data set; and the cluster analysis module is used to group data objects in the data set according to similarity.

[0013] Preferably, the feature extraction module is used to extract feature information that is valuable to the model from the original data, the feature optimization module is used to select the most representative features that are most relevant to the target variable from the numerous features that have been extracted, and the feature construction module is used to construct new features through combination and transformation based on existing features and domain knowledge.

[0014] Preferably, the classification algorithm selection module is used to select the most suitable algorithm from algorithms such as decision tree, support vector machine and neural network algorithm, the regression algorithm selection module is used to select a suitable algorithm from algorithms such as linear regression, ridge regression and random forest regression, and the clustering algorithm selection module is used to select the algorithm that best suits the data distribution and clustering target from K-means clustering, hierarchical clustering and density clustering algorithms.

[0015] Preferably, the model training module is used to train the model using the selected algorithm and prepared data, including features and labels; the model evaluation module is used to perform performance evaluation on the trained model using various evaluation indicators; the model optimization module is used to optimize the model according to the results of the model evaluation, and the optimization methods include adjusting the parameters of the model, improving the structure of the model, and adopting an integrated learning method; the model deployment module is used to deploy the trained and optimized model to an actual production environment so that it can provide services to users.

[0016] Preferably, the data cleaning module adopts at least one of the following data cleaning methods:

[0017] Missing value processing methods: For numerical data, mean filling, median filling, or filling based on machine learning model prediction can be used. For character data, mode filling or specific mark filling can be used according to data semantics and distribution to ensure that there is no analysis bias caused by missing values ​​in the data set;

[0018] The duplicate value removal method compares the key feature fields of data records and uses hash algorithms or sorting and then comparing adjacent records to identify and delete completely duplicate data records to reduce data redundancy.

[0019] Outlier detection and processing methods determine whether numerical data exceeds the normal range based on statistical analysis principles, or use clustering-based methods to identify outliers. For detected outliers, different processing methods are selected according to business scenarios, such as direct deletion, correction to reasonable values, or marking as anomalies for subsequent analysis.

[0020] Preferably, the feature construction module constructs new features based on the following method:

[0021] Feature construction based on mathematical operations: for numerical features, multiple original features are combined to construct new features through addition, subtraction, multiplication, division, and power operations;

[0022] Feature construction based on logical relationships: constructing new features based on the logical relationships between variables in the data, which helps to mine implicit logical relationships in the data and provide more discriminative features for the algorithm;

[0023] Feature construction based on domain knowledge combines professional knowledge of a specific industry or business field to transform or combine original features to construct new features, so that the constructed features are more in line with actual business needs.

[0024] Technical effects and advantages of the present invention:

[0025] 1. Through the web crawler module and sensor data collection module, web page information and sensor data are widely collected, greatly expanding the data source. The database connection module realizes the connection with various databases, which is convenient for integrating existing data. The data warehouse module and the distributed storage module are combined to ensure the storage capacity, reliability and scalability of large-scale data. At the same time, the data cleaning module uses a variety of effective methods to deal with missing values, duplicate values ​​and outliers, the data conversion module performs format conversion operations, and the data annotation module adds label annotations. These have greatly improved the data quality and laid a solid foundation for subsequent analysis and model building;

[0026] 2. In the data analysis stage, the statistical analysis module, association analysis module and cluster analysis module provide in-depth insights into data features from different perspectives. The statistical analysis module helps understand data distribution, central trend and degree of dispersion; the association analysis module mines the association between variables; the cluster analysis module groups data by similarity. In the feature engineering stage, the feature extraction, optimization and construction modules work together. The extraction module obtains valuable features from the original data, the optimization module selects the most representative and relevant features, and the construction module constructs new features based on a variety of methods, so that the model is more in line with actual business needs and improves model performance and generalization capabilities.

[0027] 3. Through the algorithm selection module, the most suitable algorithm can be accurately selected from a variety of algorithms according to different task requirements to improve the accuracy and efficiency of the model. In the model engineering stage, the training module uses the selected algorithm and data for training, the evaluation module objectively evaluates the model performance with various indicators, and the optimization module adjusts parameters, improves the structure or uses integrated learning methods to optimize the model based on the evaluation results. Finally, the deployment module puts the optimized model into the actual production environment to provide services to users, realize the actual application value of big data analysis, and provide enterprises and organizations with accurate and efficient data analysis and decision-making support. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the overall system structure of the present invention.

[0029] Figure 2 It is a schematic diagram of the data acquisition function structure of the present invention.

[0030] Figure 3 It is a schematic diagram of the data processing and storage functional structure of the present invention.

[0031] Figure 4 It is a schematic diagram of the data analysis function structure of the present invention.

[0032] Figure 5 It is a schematic diagram of the feature engineering functional structure of the present invention.

[0033] Figure 6 The schematic diagram of the algorithm selection function structure of the present invention.

[0034] Figure 7 It is a schematic diagram of the functional structure of the model engineering of the present invention.

[0035] The accompanying drawings are marked as follows: 1. Data acquisition module; 2. Data processing and storage module; 3. Data analysis module; 4. Feature engineering module; 5. Algorithm selection module; 6. Model engineering module; 7. Web crawler module; 8. Sensor data acquisition module; 9. Database connection module; 10. Data cleaning module; 11. Data conversion module; 12. Data annotation module; 13. Data warehouse module; 14. Distributed storage module; 15. Statistical analysis module; 16. Association analysis module; 17. Clustering analysis module; 18. Feature extraction module; 19. Feature optimization module; 20. Feature construction module; 21. Classification algorithm selection module; 22. Regression algorithm selection module; 23. Clustering algorithm selection module; 24. Model training module; 25. Model evaluation module; 26. Model optimization module; 27. Model deployment module. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] As attached Figure 1-7 The algorithm-assisted upgrading system based on big data shown includes a data acquisition module 1, the output end of the data acquisition module 1 is connected to a data processing and storage module 2, the output end of the data processing and storage module 2 is connected to a data analysis module 3, the output end of the data analysis module 3 is connected to a feature engineering module 4, the output end of the feature engineering module 4 is connected to an algorithm selection module 5, and the output end of the algorithm selection module 5 is connected to a model engineering module 6;

[0038] The input end of the data acquisition module 1 is respectively connected to the network crawler module 7 and the sensor data acquisition module 8, the output end of the data acquisition module 1 is connected to the database connection module 9, the output end of the database connection module 9 is connected to the data processing and storage module 2, the output end of the data processing and storage module 2 is connected to the data cleaning module 10, the output end of the data cleaning module 10 is connected to the data conversion module 11, the output end of the data conversion module 11 is connected to the data annotation module 12, and the output end of the data annotation module 12 is respectively connected to the data warehouse module 13 and the distributed storage module 14;

[0039] The output end of the distributed storage module 14 is connected to the data analysis module 3, and the output end of the data analysis module 3 is respectively connected to the statistical analysis module 15, the association analysis module 16 and the cluster analysis module 17, and the output ends of the statistical analysis module 15, the association analysis module 16 and the cluster analysis module 17 are all connected to the feature engineering module 4;

[0040] The output end of the feature engineering module 4 is connected to the feature extraction module 18 , the output end of the feature extraction module 18 is connected to the feature optimization module 19 , the output end of the feature optimization module 19 is connected to the feature construction module 20 , and the output end of the feature construction module 20 is connected to the algorithm selection module 5 .

[0041] The output end of the algorithm selection module 5 is respectively connected to a classification algorithm selection module 21 , a regression algorithm selection module 22 and a clustering algorithm selection module 23 .

[0042] The output end of the model engineering module 6 is connected to the model training module 24 , the output end of the model training module 24 is connected to the model evaluation module 25 , the output end of the model evaluation module 25 is connected to the model optimization module 26 , and the output end of the model optimization module 26 is connected to the model deployment module 27 .

[0043] The web crawler module 7 is used to crawl text, pictures and link information in web pages according to pre-set rules and target websites. The sensor data acquisition module 8 is used to collect data generated by various sensors. The database connection module 9 is used to establish connections with various databases. The data cleaning module 10 is used to deal with noise, missing values, duplicate values ​​and erroneous data problems in the data. The data conversion module 11 is used to perform format conversion, normalization and standardization operations on the data.

[0044] The data annotation module 12 is used to add labels or annotations to the data, including classification labels, timestamps and source tags. The data warehouse module 13 is used to store large amounts of data that have been integrated, cleaned and preprocessed. The distributed storage module 14 improves the capacity, reliability and scalability of data storage by distributing the data on multiple nodes or servers. The statistical analysis module 15 performs basic statistical calculations on the data, including calculating the mean, median, variance and standard deviation statistical indicators to help understand the distribution characteristics, central trend and degree of dispersion of the data. The association analysis module 16 is used to discover the association between different variables in the data set. The cluster analysis module 17 is used to group the data objects in the data set according to similarity.

[0045] The feature extraction module 18 is used to extract feature information that is valuable to the model from the original data. The feature optimization module 19 is used to select the most representative features that are most relevant to the target variable from the many features that have been extracted. The feature construction module 20 is used to construct new features through combination and transformation based on existing features and domain knowledge.

[0046] The classification algorithm selection module 21 is used to select the most suitable algorithm from algorithms such as decision tree, support vector machine and neural network algorithm. The regression algorithm selection module 22 is used to select a suitable algorithm from algorithms such as linear regression, ridge regression and random forest regression. The clustering algorithm selection module 23 is used to select the algorithm that best suits the data distribution and clustering target from K-means clustering, hierarchical clustering and density clustering algorithms.

[0047] The model training module 24 is used to train the model using the selected algorithm and prepared data, including features and labels. The model evaluation module 25 is used to evaluate the performance of the trained model using various evaluation indicators. The model optimization module 26 is used to optimize the model based on the results of the model evaluation. The optimization methods include adjusting the parameters of the model, improving the structure of the model, and adopting an integrated learning method. The model deployment module 27 is used to deploy the trained and optimized model to the actual production environment so that it can provide services to users.

[0048] The data cleaning module 10 uses at least one of the following data cleaning methods:

[0049] Missing value processing methods: For numerical data, mean filling, median filling, or filling based on machine learning model prediction can be used. For character data, mode filling or specific mark filling can be used according to data semantics and distribution to ensure that there is no analysis bias caused by missing values ​​in the data set;

[0050] The duplicate value removal method compares the key feature fields of data records and uses hash algorithms or sorting and then comparing adjacent records to identify and delete completely duplicate data records to reduce data redundancy.

[0051] Outlier detection and processing methods determine whether numerical data exceeds the normal range based on statistical analysis principles, or use clustering-based methods to identify outliers. For detected outliers, different processing methods are selected according to business scenarios, such as direct deletion, correction to reasonable values, or marking as anomalies for subsequent analysis.

[0052] The feature construction module 20 constructs new features based on the following method:

[0053] Feature construction based on mathematical operations: for numerical features, multiple original features are combined to construct new features through addition, subtraction, multiplication, division, and power operations;

[0054] Feature construction based on logical relationships: constructing new features based on the logical relationships between variables in the data, which helps to mine implicit logical relationships in the data and provide more discriminative features for the algorithm;

[0055] Feature construction based on domain knowledge combines professional knowledge of a specific industry or business field to transform or combine original features to construct new features, so that the constructed features are more in line with actual business needs.

[0056] Working principle of the present invention: the data acquisition module 1 plays a core role, and its input end is connected to the web crawler module 7 and the sensor data acquisition module 8 respectively. The web crawler module 7 crawls the text, pictures and link information in the web page according to the pre-set rules and target websites; the sensor data acquisition module 8 is used to collect data generated by various sensors;

[0057] The output end of the data acquisition module 1 is connected to the database connection module 9, which is used to establish connections with various databases. Its output end is connected to the data processing and storage module 2, and the collected data is transmitted to the data for subsequent processing. The output end of the data processing and storage module 2 is connected to the data cleaning module 10. The data cleaning module 10 uses a variety of data cleaning methods to deal with noise, missing values, duplicate values ​​and erroneous data problems in the data. For example, for missing values ​​of numerical data, mean filling, median filling or filling based on machine learning model prediction can be used. Duplicate value removal is carried out by comparing the key feature fields of the data records, using hash algorithms or comparing adjacent records after sorting. Abnormal value detection and processing is based on statistics. The data cleaning module 10 is operated by the statistical analysis principle or clustering method. The output end of the data cleaning module 10 is connected to the data conversion module 11. The data conversion module 11 is used to perform format conversion, normalization and standardization operations on the data. The output end of the data conversion module 11 is connected to the data annotation module 12. The data annotation module 12 is used to add labels or annotations to the data, including classification labels, timestamps and source tags. The output ends of the data annotation module 12 are respectively connected to the data warehouse module 13 and the distributed storage module 14. The data warehouse module 13 is used to store a large amount of data after integration, cleaning and preprocessing. The distributed storage module 14 improves the capacity, reliability and scalability of data storage by distributing and storing the data on multiple nodes or servers.

[0058] The output end of the distributed storage module 14 is connected to the data analysis module 3, and the output end of the data analysis module 3 is respectively connected to the statistical analysis module 15, the association analysis module 16 and the cluster analysis module 17. The statistical analysis module 15 performs basic statistical calculations on the data, including calculating the mean, median, variance and standard deviation statistical indicators to help understand the distribution characteristics, central trend and dispersion of the data; the association analysis module 16 is used to discover the association relationship between different variables in the data set; the cluster analysis module 17 is used to group the data objects in the data set according to similarity;

[0059] The output end of the feature engineering module 4 is connected to a feature extraction module 18, which is used to extract feature information valuable to the model from the original data. The output end of the feature extraction module 18 is connected to a feature optimization module 19, which is used to select the most representative feature that is most relevant to the target variable from the many features that have been extracted. The output end of the feature optimization module 19 is connected to a feature construction module 20, which constructs new features based on the following methods: feature construction based on mathematical operations, for numerical features, multiple original features are combined through addition, subtraction, multiplication, division, and power operations to construct new features; feature construction based on logical relationships, new features are constructed according to the logical relationships between variables in the data; feature construction based on domain knowledge, combining the professional knowledge of a specific industry or business field to transform or combine the original features to construct new features;

[0060] The output end of the algorithm selection module 5 is respectively connected to a classification algorithm selection module 21, a regression algorithm selection module 22 and a clustering algorithm selection module 23. The classification algorithm selection module 21 is used to select the most suitable algorithm from decision tree, support vector machine and neural network algorithm; the regression algorithm selection module 22 is used to select a suitable algorithm from linear regression, ridge regression and random forest regression; the clustering algorithm selection module 23 is used to select the most suitable algorithm for data distribution and clustering target from K-means clustering, hierarchical clustering and density clustering algorithms;

[0061] The output end of the model engineering module 6 is connected to a model training module 24, which is used to train the model using the selected algorithm and prepared data, including features and labels. The output end of the model training module 24 is connected to a model evaluation module 25, which is used to perform performance evaluation on the trained model using various evaluation indicators. The output end of the model evaluation module 25 is connected to a model optimization module 26, which is used to optimize the model according to the results of the model evaluation. The optimization methods include adjusting the parameters of the model, improving the structure of the model, and adopting an integrated learning method. The output end of the model optimization module 26 is connected to a model deployment module 27, which is used to deploy the trained and optimized model to an actual production environment so that it can provide services to users.

[0062] Finally, it should be noted that the drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention, and other structures can refer to common designs. In the absence of conflicts, the same embodiment and different embodiments of the present invention can be combined with each other;

[0063] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An algorithm-assisted upgrade system based on big data, comprising a data acquisition module (1), characterized in that: The output end of the data acquisition module (1) is connected to a data processing and storage module (2), the output end of the data processing and storage module (2) is connected to a data analysis module (3), the output end of the data analysis module (3) is connected to a feature engineering module (4), the output end of the feature engineering module (4) is connected to an algorithm selection module (5), and the output end of the algorithm selection module (5) is connected to a model engineering module (6); The input end of the data acquisition module (1) is respectively connected to a web crawler module (7) and a sensor data acquisition module (8); the output end of the data acquisition module (1) is connected to a database connection module (9); the output end of the database connection module (9) is connected to a data processing and storage module (2); the output end of the data processing and storage module (2) is connected to a data cleaning module (10); the output end of the data cleaning module (10) is connected to a data conversion module (11); the output end of the data conversion module (11) is connected to a data annotation module (12); the output end of the data annotation module (12) is respectively connected to a data warehouse module (13) and a distributed storage module (14); The output end of the distributed storage module (14) is connected to a data analysis module (3), the output end of the data analysis module (3) is respectively connected to a statistical analysis module (15), an association analysis module (16) and a cluster analysis module (17), and the output ends of the statistical analysis module (15), the association analysis module (16) and the cluster analysis module (17) are all connected to a feature engineering module (4); The output end of the feature engineering module (4) is connected to a feature extraction module (18), the output end of the feature extraction module (18) is connected to a feature optimization module (19), the output end of the feature optimization module (19) is connected to a feature construction module (20), and the output end of the feature construction module (20) is connected to the algorithm selection module (5).

2. The algorithm-assisted upgrade system based on big data according to claim 1, characterized in that: The output end of the algorithm selection module (5) is respectively connected to a classification algorithm selection module (21), a regression algorithm selection module (22) and a clustering algorithm selection module (23).

3. The algorithm-assisted upgrade system based on big data according to claim 1, characterized in that: The output end of the model engineering module (6) is connected to a model training module (24), the output end of the model training module (24) is connected to a model evaluation module (25), the output end of the model evaluation module (25) is connected to a model optimization module (26), and the output end of the model optimization module (26) is connected to a model deployment module (27).

4. The algorithm-assisted upgrade system based on big data according to claim 1, characterized in that: The web crawler module (7) is used to crawl text, pictures and link information in web pages according to pre-set rules and target websites. The sensor data acquisition module (8) is used to collect data generated by various sensors. The database connection module (9) is used to establish connections with various databases. The data cleaning module (10) is used to process noise, missing values, duplicate values ​​and erroneous data problems in the data. The data conversion module (11) is used to perform format conversion, normalization and standardization operations on the data.

5. The algorithm-assisted upgrade system based on big data according to claim 1, characterized in that: The data annotation module (12) is used to add labels or annotations to the data, including classification labels, timestamps and source tags. The data warehouse module (13) is used to store a large amount of data after integration, cleaning and preprocessing. The distributed storage module (14) improves the capacity, reliability and scalability of data storage by distributing and storing the data on multiple nodes or servers. The statistical analysis module (15) performs basic statistical calculations on the data, including calculating mean, median, variance and standard deviation statistical indicators to help understand the distribution characteristics, central tendency and degree of dispersion of the data. The association analysis module (16) is used to discover the association relationship between different variables in the data set. The cluster analysis module (17) is used to group the data objects in the data set according to similarity.

6. The algorithm-assisted upgrade system based on big data according to claim 1, characterized in that: The feature extraction module (18) is used to extract feature information valuable to the model from the original data, the feature optimization module (19) is used to select the most representative feature that is most relevant to the target variable from the numerous features that have been extracted, and the feature construction module (20) is used to construct new features by combining and transforming existing features and domain knowledge.

7. The algorithm-assisted upgrade system based on big data according to claim 2, characterized in that: The classification algorithm selection module (21) is used to select the most suitable algorithm from algorithms such as decision tree, support vector machine and neural network algorithm, the regression algorithm selection module (22) is used to select a suitable algorithm from algorithms such as linear regression, ridge regression and random forest regression, and the clustering algorithm selection module (23) is used to select the algorithm that best suits the data distribution and clustering target from K-means clustering, hierarchical clustering and density clustering algorithms.

8. The algorithm-assisted upgrade system based on big data according to claim 3 is characterized in that: The model training module (24) is used to train the model using the selected algorithm and the prepared data, including features and labels. The model evaluation module (25) is used to evaluate the performance of the trained model using various evaluation indicators. The model optimization module (26) is used to optimize the model according to the results of the model evaluation. The optimization methods include adjusting the parameters of the model, improving the structure of the model, and adopting an integrated learning method. The model deployment module (27) is used to deploy the trained and optimized model to an actual production environment so that it can provide services to users.

9. The algorithm-assisted upgrade system based on big data according to claim 1, characterized in that: The data cleaning module (10) adopts at least one of the following data cleaning methods: Missing value processing methods: For numerical data, mean filling, median filling, or filling based on machine learning model prediction can be used. For character data, mode filling or specific mark filling can be used according to data semantics and distribution to ensure that there is no analysis bias caused by missing values ​​in the data set; The duplicate value removal method compares the key feature fields of data records and uses hash algorithms or sorting and then comparing adjacent records to identify and delete completely duplicate data records to reduce data redundancy. Outlier detection and processing methods determine whether numerical data exceeds the normal range based on statistical analysis principles, or use clustering-based methods to identify outliers. For detected outliers, different processing methods are selected according to business scenarios, such as direct deletion, correction to reasonable values, or marking as anomalies for subsequent analysis.

10. The algorithm-assisted upgrade system based on big data according to claim 1, characterized in that: The feature construction module (20) constructs new features based on the following method: Feature construction based on mathematical operations: for numerical features, multiple original features are combined to construct new features through addition, subtraction, multiplication, division, and power operations; Feature construction based on logical relationships: constructing new features based on the logical relationships between variables in the data, which helps to mine implicit logical relationships in the data and provide more discriminative features for the algorithm; Feature construction based on domain knowledge combines professional knowledge of a specific industry or business field to transform or combine original features to construct new features, so that the constructed features are more in line with actual business needs.

Images