Aeronautical data collection and analysis system based on machine learning and web crawler

Abstract

The invention discloses an aeronautical data collection and analysis system based on machine learning and web crawlers, belongs to the technical field of data processing, and aims to solve the technical problem of how to realize automatic collection, intelligent analysis and efficient utilization of aeronautical data by using machine learning and web crawlers. Comprising a data collection module, a data preprocessing module, a machine learning module, a decision support module and a visual display module, the integrated machine learning module can process a large-scale and complex data set, multiple advanced algorithms are applied to carry out deep mining on data, and therefore valuable rules and trends hidden behind the data are found; the method not only improves the precision of data analysis, but also enables the analysis result to be more insight and foresight.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and more specifically, to an aviation data collection and analysis system based on machine learning and web crawlers. Background Art

[0002] With the rapid development of the aviation industry, the scale of aviation data has shown explosive growth. This data includes flight information, airport operation data, airline management data, etc., which is of extremely important value for the operation decisions of airlines, the traffic management of airports, and the strategic planning of the entire aviation industry. However, traditional data collection and processing methods often rely on manual labor, with low efficiency and prone to errors, and cannot meet the aviation industry's demand for efficient and accurate data processing.

[0003] In recent years, the rapid development of artificial intelligence and machine learning technologies has brought new opportunities for aviation data processing. Through machine learning algorithms, a large amount of aviation data can be deeply analyzed and mined to discover potential laws and values in the data, providing decision-making support for the aviation industry. At the same time, web crawler technology also makes it possible to automatically collect aviation data. A web crawler can simulate the behavior of a human browser and automatically scrape the required data from the Internet, greatly improving the efficiency of data collection.

[0004] How to utilize machine learning and web crawlers to achieve automated collection, intelligent analysis, and efficient utilization of aviation data is a technical problem that needs to be solved. Summary of the Invention

[0005] The technical task of the present invention is to address the above deficiencies and provide an aviation data receipt analysis method and system based on machine learning and web crawlers to solve the technical problem of how to utilize machine learning and web crawlers to achieve automated collection, intelligent analysis, and efficient utilization of aviation data.

[0006] In a first aspect, an aviation data collection and analysis system based on machine learning and web crawlers of the present invention includes a data collection module, a data preprocessing module, a machine learning module, a decision support module, and a visualization display module;

[0007] The data collection module is used to collect aviation data from various data sources based on web crawler technology. The data sources include flight information websites, airport official websites, and open data sources. The aviation data includes flight information, airport information, airline operation data, and weather information;

[0008] The data preprocessing module is used to perform data cleaning operations on the collected aviation data and store the cleaned aviation data in a database;

[0009] A variety of machine learning models are configured in the machine learning module, which is used to recommend machine learning models for users and support users in selecting a machine learning model as the target machine learning model. It is used to perform feature engineering, data transformation, and data scaling processing on the cleaned aviation data according to the requirements of the target machine learning model, so as to extract key features from the aviation data, and construct a training set, a validation set, and a test set based on the extracted key features. It is used to train the target machine learning model based on the training set, and perform model validation and parameter tuning on the trained target machine learning model based on the validation set and the test set to obtain the final target machine learning model;

[0010] The decision support module is used to call the data collection module to collect aviation data, call the data preprocessing module to perform data cleaning operations on the collected aviation data, and call the machine learning module to perform feature engineering, data transformation, and data scaling processing on the cleaned aviation data according to the requirements of the target machine learning model, so as to extract key features from the aviation data. Using the extracted key features as input, make a decision prediction through the final target machine learning model to obtain a predicted decision result, and construct the final decision support based on the predicted decision result and the expert knowledge of the domain;

[0011] The visualization display module interacts with the user through a visualization interface, which is used to display the collected aviation data and the final decision support, and provide a query service to support the user to customize query conditions and filter data operations through the query service.

[0012] Preferably, the data collection module is used to regularly collect aviation data from various data sources based on a set crawling frequency, and push and update the aviation data to the data preprocessing module through the WebSocket technology.

[0013] Preferably, the aviation data includes structured data and unstructured data, where flight information and airport information belong to structured data, and web page text, airline operation data of the comment type, and weather information belong to unstructured data;

[0014] For structured data, the data preprocessing module is used to perform the following data cleaning operations: according to the nature and distribution of the data, select to delete records containing missing values, fill them with the mean, median or mode, set a threshold based on business logic and data distribution, and identify and remove outliers based on statistical methods. Compare the unique identifiers of the aviation data to remove duplicate data. Among them, the statistical method includes the IQR rule, and the unique identifier includes ID and timestamp. Store the cleaned structured data in a relational database, organize the data using table structures and relationships, and determine the storage form of the data according to business requirements and data characteristics;

[0015] For unstructured data, the data preprocessing module is used to perform the following data cleaning operations: removing useless information, noise, and redundant data to ensure the accuracy and consistency of the data. For text-type aviation data, word segmentation, stop word removal, and stemming operations are performed. The cleaned unstructured data is stored in a non-relational database, and the storage form of the data is determined according to business requirements and data characteristics.

[0016] Preferably, the machine learning module is used to perform the following operations:

[0017] Feature engineering operations: Through exploratory data analysis, key factors that affect key business indicators are identified from aviation data as key features. The key factors that affect key business indicators include flight punctuality rate, ticket price trend, and flight popularity.

[0018] Perform feature preprocessing on the selected key features, and perform missing value filling, outlier handling, feature scaling, and feature encoding operations through feature preprocessing.

[0019] According to data characteristics, task complexity, and performance requirements, recommend or select a suitable machine learning model as the target machine learning model.

[0020] Using the training set as input, train the target machine learning model, and introduce cross-validation and grid search strategies during the model training process to automatically adjust the model parameters of the target machine learning model.

[0021] Based on the validation set and test set, evaluate the trained target machine learning model and optimize the parameters. Evaluate the generalization ability of the model through model evaluation. During parameter tuning, iteratively optimize the model with reference to the stability, interpretability, and computational efficiency of the model to obtain the final target machine learning model.

[0022] Preferably, the decision support module is used to interface with a third-party system, and is used to construct the final decision support based on the decision results predicted by the target machine learning model, the experience knowledge of domain experts, and the decision suggestions and solutions of the third party. The decision support includes route optimization, ticket pricing strategy, and airline evaluation.

[0023] In a second aspect, an aviation data receipt analysis method based on machine learning and web crawler realizes the collection and analysis of aviation data through an aviation data receipt analysis system according to any one of the first aspects. The method includes the following steps:

[0024] Data collection: Collect aviation data from various data sources based on web crawler technology. The data sources include flight information websites, airport official websites, and open data sources. The aviation data includes flight information, airport information, airline operation data, and weather information;

[0025] Data preprocessing: Perform data cleaning operations on the collected aviation data and store the cleaned aviation data in a database;

[0026] Machine learning: Recommend to the user and support the user in selecting a machine learning model as the target machine learning model. According to the requirements of the target machine learning model, perform feature engineering, data transformation, and data scaling on the cleaned aviation data to extract key features from the aviation data, and construct a training set, a validation set, and a test set based on the extracted key features. Train the target machine learning model based on the training set, and perform model validation and parameter tuning on the trained target machine learning model based on the validation set and the test set to obtain the final target machine learning model;

[0027] Decision support: Collect aviation data, perform data cleaning operations on the collected aviation data, perform feature engineering, data transformation, and data scaling on the cleaned aviation data according to the requirements of the target machine learning model to extract key features from the aviation data, and use the extracted key features as input to perform decision prediction through the final target machine learning model to obtain the predicted decision result. Construct the final decision support based on the predicted decision result and the expert knowledge in the field;

[0028] Visualization display: Display the collected aviation data and the final decision support, and provide a query service. Support the user to customize query conditions and filter data operations through the query service.

[0029] Preferably, during data collection, collect aviation data from various data sources at regular intervals based on the set crawling frequency, and push and update the aviation data to the data preprocessing module through WebSocket technology.

[0030] Preferably, the aviation data includes structured data and unstructured data. Among them, flight information and airport information belong to structured data, and airline operation data such as web page text and comments, and weather information belong to unstructured data;

[0031] For structured data, perform data cleaning operations as follows: According to the nature and distribution of the data, choose to delete records containing missing values, fill them with the mean, median, or mode, set thresholds based on business logic and data distribution, and identify and remove outliers based on statistical methods. Remove duplicate data by comparing the unique identifiers of the aviation data. Among them, the statistical method includes the IQR rule, and the unique identifier includes the ID and timestamp. Store the cleaned structured data in a relational database, organize the data using table structures and relationships, and determine the storage form of the data according to business requirements and data characteristics;

[0032] For unstructured data, perform data cleaning operations as follows: Remove useless information, noise, and redundant data to ensure the accuracy and consistency of the data. For text-type aviation data, perform word segmentation, stop word removal, and stemming operations. Store the cleaned unstructured data in a non-relational database, and determine the storage form of the data according to business requirements and data characteristics.

[0033] Preferably, machine learning includes operations:

[0034] Feature engineering operations: Through exploratory data analysis, identify key factors that affect key business metrics from aviation data as key features. The key factors that affect key business metrics include flight punctuality rate, ticket price trend, and flight popularity;

[0035] Perform feature preprocessing on the selected key features, and perform missing value filling, outlier handling, feature scaling, and feature encoding operations through feature preprocessing;

[0036] According to data characteristics, task complexity, and performance requirements, recommend or select a suitable machine learning model as the target machine learning model;

[0037] Use the training set as input to train the target machine learning model, and introduce cross-validation and grid search strategies during model training to automatically adjust the model parameters of the target machine learning model;

[0038] Based on the validation set and test set, evaluate the trained target machine learning model and optimize the parameters. Evaluate the generalization ability of the model through model evaluation. During parameter tuning, iteratively optimize the model with reference to the stability, interpretability, and computational efficiency of the model to obtain the final target machine learning model.

[0039] Preferably, when making decision support, connect to a third-party system, and build the final decision support based on the decision results predicted by the target machine learning model, the expert knowledge of domain experts, and the decision suggestions and solutions of the third party. The decision support includes route optimization, ticket pricing strategy, and airline evaluation.

[0040] The aviation data receipt analysis system and method based on machine learning and web crawlers of the present invention have the following advantages:

[0041] 1. Significantly improve data collection efficiency: Through automated web crawler technology, it can continuously and efficiently capture data from multiple websites related to the aviation field. Compared with traditional manual collection methods, it greatly saves time costs and human resources, ensuring the real-time and comprehensiveness of data;

[0042] 2. Enhance the depth and accuracy of data analysis: The integrated machine learning module can process large-scale and complex data sets, and use a variety of advanced algorithms to deeply mine the data, thereby discovering valuable laws and trends hidden behind the data. This not only improves the accuracy of data analysis, but also makes the analysis results more insightful and forward-looking;

[0043] 3. Optimize the decision-making process: The intelligent decision support module, based on the analysis results of the machine learning module, provides scientific and accurate decision-making suggestions for decision-makers in the aviation field. These suggestions are based on objective data and algorithm models, helping to reduce the bias caused by subjective judgment, improve the scientificity and effectiveness of decision-making, and promote the refined management and sustainable development of aviation enterprises;

[0044] 4. Improve user experience and information transparency: The visualization display module converts complex data analysis results into intuitive and easy-to-understand charts and images, enabling non-professionals to quickly understand the meaning behind the data. This not only enhances information transparency, but also improves the user experience, helping airlines, airports and other institutions to better communicate with the public and enhance trust;

[0045] 5. Promote the innovative development of the aviation field: Through intelligent data collection and analysis, it provides strong data support for innovation in the aviation field. Airlines can use this data to optimize flight route layouts, adjust ticket price strategies, and improve service quality. Airports can evaluate traffic flow, optimize resource allocation, and improve operational efficiency. Overall, this system promotes the digital transformation and intelligent upgrading of the aviation field;

[0046] 6. Enhance data security and compliance: During the data collection and processing process, this system pays attention to data security and compliance. It avoids putting too much pressure on the target website through reasonable crawler strategies, and at the same time complies with relevant laws and regulations to ensure the legality of data and privacy protection, providing a solid guarantee for the stable operation of aviation enterprises. Brief Description of the Drawings

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0048] The present invention will be further described below with reference to the drawings.

[0049] Figure 1 FIG. 7 is a flowchart of the working process of an aviation data collection and analysis system based on machine learning and web crawlers in Embodiment 1;

[0050] Figure 2 FIG. 11 is a working block diagram of the data preprocessing module in an aviation data collection and analysis system based on machine learning and web crawlers in Embodiment 1. Detailed implementation manners

[0051] The present invention will be further described below with reference to the drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it. However, the specific embodiments cited are not intended to limit the present invention. Without conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0052] The embodiments of the present invention provide an aviation data collection and analysis system and method based on machine learning and web crawlers, which are used to solve the technical problem of how to use machine learning and web crawlers to achieve automatic collection, intelligent analysis and efficient utilization of aviation data.

[0053] Embodiment 1:

[0054] An aviation data collection and analysis system based on machine learning and web crawlers according to the present invention includes a data collection module, a data preprocessing module, a machine learning module, a decision support module, and a visualization display module.

[0055] The data collection module is used to collect aviation data from various data sources based on web crawler technology. The data sources include flight information websites, airport official websites, and open data sources. The aviation data includes flight information, airport information, airline operation data, and weather information.

[0056] As a specific implementation of the data collection module, this module is used to regularly collect aviation data from various data sources based on the set crawling frequency, and push and update the aviation data to the data preprocessing module through WebSocket technology.

[0057] The data collection module is a crucial part of this system. It is responsible for automatically scraping aviation-related data from the Internet, providing a basis for subsequent data processing and analysis. When selecting crawling targets, this module focuses on flight information websites, airport official websites, and open data sources to ensure the comprehensiveness and real-time nature of the data.

[0058] This module will extract key information including flight information (such as flight numbers, takeoff and landing times, etc.), airport information (such as airport names, locations, etc.), airline operation data, and weather information. Through the processing and comprehensive analysis of different types of data, it is possible to understand the dynamics of the aviation market, optimize flight schedules, and enhance the passenger experience, etc.

[0059] To achieve regular or real-time crawling of data, the module sets the crawling frequency according to requirements, whether it is daily, hourly, or real-time update. For real-time data, the module will utilize WebSocket technology to achieve fast data push and update.

[0060] The data preprocessing module is used to perform data cleaning operations on the collected aviation data and store the cleaned aviation data in a database.

[0061] In this embodiment, the aviation data includes structured data and unstructured data. Among them, flight information and airport information belong to structured data, and web page text, comment-based airline operation data, and weather information belong to unstructured data.

[0062] For structured data, the data preprocessing module is used to perform the following data cleaning operations: According to the nature and distribution of the data, select to delete records containing missing values, use the mean, median, or mode for filling, set thresholds based on business logic and data distribution, and identify and remove outliers based on statistical methods. Remove duplicate data by comparing the unique identifiers of the aviation data. Among them, the statistical method includes the IQR rule, and the unique identifier includes ID and timestamp. Store the cleaned structured data in a relational database, organize the data using table structures and relationships, and determine the storage form of the data according to business requirements and data characteristics.

[0063] For unstructured data, the data preprocessing module is used to perform the following data cleaning operations: Remove useless information, noise, and redundant data to ensure the accuracy and consistency of the data. For text-type aviation data, perform word segmentation, stop word removal, and stemming operations. Store the cleaned unstructured data in a non-relational database and determine the storage form of the data according to business requirements and data characteristics.

[0064] This module is responsible for performing a series of processing and conversions on the original data obtained by the data collection module to ensure the quality and usability of the data.

[0065] First, the original data mainly includes structured data (such as flight information, airport information, etc.) and unstructured data (such as web text, comments, etc.). According to the nature and distribution of the data, choose to delete records containing missing values, fill them with statistical measures such as mean, median or mode; set thresholds and use statistical methods (such as the IQR rule) to identify and remove outliers according to business logic and data distribution; remove duplicate records by comparing the unique identifiers of the data (such as ID, timestamp, etc.).

[0066] Secondly, for unstructured data, remove useless information, noise and redundant data to ensure the accuracy and consistency of the data. For text data, perform basic preprocessing steps such as word segmentation, stop word removal, and stemming.

[0067] After data cleaning, the structured data is directly saved to a relational database (MySQL), and the table structure and relationships are used to organize the data. For unstructured data (such as text), it is saved to a non-relational database (MongoDB) for data storage and management. In addition, when saving the data, it is also necessary to determine the data storage form according to business requirements and data characteristics. Classification helps to better organize the data and facilitates subsequent analysis and processing.

[0068] Finally, the preprocessed data will be directly used for subsequent machine learning model training. According to the type and requirements of the model, perform feature engineering, data transformation, and data scaling on the data, aiming to extract the key features in the data and adjust the data distribution to improve the training effect and performance of the model. After processing, the data will be divided into a training set, a validation set, and a test set for model training, validation, and evaluation of the model.

[0069] Multiple machine learning models are configured in the machine learning module to recommend to the user and support the user in selecting a machine learning model as the target machine learning model. According to the requirements of the target machine learning model, perform feature engineering, data transformation, and data scaling on the cleaned aviation data to extract the key features in the aviation data, and construct a training set, a validation set, and a test set based on the extracted key features. Use the training set to train the target machine learning model, and use the validation set and the test set to perform model validation and parameter tuning on the trained target machine learning model to obtain the final target machine learning model.

[0070] As a specific implementation of the machine learning module, this module is used to perform the following operations:

[0071] (1) Feature engineering operations: Through exploratory data analysis, identify the key factors that affect key business metrics from aviation data as key features. The key factors that affect key business metrics include flight punctuality rate, ticket price trend, and flight popularity.

[0072] (2) Perform feature preprocessing on the selected key features, including filling missing values, handling outliers, feature scaling, and feature encoding operations through feature preprocessing.

[0073] (3) According to the data characteristics, task complexity, and performance requirements, recommend or select a suitable machine learning model as the target machine learning model.

[0074] (4) Use the training set as input to train the target machine learning model, and introduce cross-validation and grid search strategies during the model training process to automatically adjust the model parameters of the target machine learning model.

[0075] (5) Based on the validation set and test set, evaluate the trained target machine learning model and optimize the parameters. Evaluate the generalization ability of the model through model evaluation. During parameter tuning, iteratively optimize the model with reference to the stability, interpretability, and computational efficiency of the model to obtain the final target machine learning model.

[0076] In this embodiment, in the machine learning module, feature engineering is carried out first. Through exploratory data analysis (EDA), the key factors that affect key business metrics such as flight punctuality rate, ticket price trend, and route popularity can be identified. This process includes screening and evaluating existing features, and also involves constructing new features, such as seasonal adjustment factors calculated from historical data, prediction variables based on complex weather models, etc. In addition, to ensure the efficiency and accuracy of model training, necessary preprocessing of features is carried out, such as filling missing values, handling outliers, feature scaling, and encoding in the previous data preprocessing process.

[0077] According to the specific business scenario and requirements, this machine learning module can provide an intelligent model selection mechanism. This mechanism can automatically recommend or select the most suitable machine learning algorithm based on data characteristics, task complexity, and performance requirements. In the model training stage, the module supports efficient parallel computing and distributed processing technologies, which can significantly shorten the training time of large-scale data sets. At the same time, by introducing strategies such as cross-validation and grid search, the module can automatically adjust the model parameters, search for the optimal model configuration, and achieve the best prediction effect.

[0078] To ensure the accuracy and reliability of the model, the machine learning module incorporates a model validation and tuning process. By dividing the independent validation set and test set, the module can objectively evaluate the generalization ability of the model and promptly detect and address potential overfitting or underfitting issues. During the tuning process, the module not only focuses on the prediction accuracy of the model but also comprehensively considers multiple aspects such as the stability, interpretability, and computational efficiency of the model. Through iterative optimization and fine-tuning, the performance of the model is enhanced to meet the requirements of actual business.

[0079] The decision support module is used to call the data collection module to collect aviation data, call the data preprocessing module to perform data cleaning operations on the collected aviation data, and according to the requirements of the target machine learning model, call the machine learning module to perform feature engineering, data transformation, and data scaling processing on the cleaned aviation data to extract the key features in the aviation data. Using the extracted key features as input, it performs decision prediction through the final target machine learning model to obtain the predicted decision results, and constructs the final decision support based on the predicted decision results and the expert knowledge of the domain.

[0080] As a further improvement of the decision support module, the decision support module is used to interface with a third-party system, and is used to construct the final decision support based on the decision results predicted by the target machine learning model, the expert knowledge of the domain, and the decision suggestions and solutions of the third party. The decision support includes route optimization, ticket pricing strategies, and airline evaluations.

[0081] In this embodiment, the decision support module provides decision support for different scenarios based on the prediction results of the machine learning model, combined with the experience and knowledge of domain experts. The module can generate corresponding decision suggestions or solutions according to different application scenarios and requirements, such as route optimization, ticket pricing strategies, airline evaluations, etc. At the same time, the module also supports integration and docking with other systems to achieve data sharing and collaborative work.

[0082] The visualization display module interacts with users through a visualization interface, and is used to display the collected aviation data and the final decision support, and provide a query service. Through the query service, it supports users to customize query conditions and filter data operations.

[0083] In this embodiment, the visualization display module presents the analysis results of the machine learning module to users in an intuitive and easy-to-understand form. It displays the distribution, trends, etc. of the data through forms such as charts and images to help users better understand and use the analysis results. At the same time, the module also provides a user-friendly interaction interface, supporting operations such as users customizing query conditions and filtering data to meet the personalized needs of users.

[0084] Once the model has been fully verified and optimized, the machine learning module can be flexibly deployed into the production environment. By providing an API interface and visualization tools that are easy to integrate, the module can easily interface with other business systems to achieve real-time or batch data processing and predictive analysis. In practical applications, the module can provide comprehensive intelligent services for airlines, airports, tourism companies, etc., such as flight delay warnings, ticket price optimization, and route planning adjustments.

[0085] The system of this embodiment realizes the receipt and analysis of aviation data through the following steps:

[0086] (1) Determine the target websites and data sources, and write corresponding regular or real-time web crawler programs to capture multi-type data;

[0087] (2) Preprocess and clean the captured raw data to ensure the accuracy and consistency of the data; classify and store the data in the database; construct multi-source data training sets and test sets according to requirements;

[0088] (3) Use machine learning algorithms to deeply analyze and mine the preprocessed data, discover valuable information and patterns, and train the learning model;

[0089] (4) Display the analysis results in the form of charts, images, etc., for users to intuitively understand and use;

[0090] (5) Generate corresponding decision-making suggestions or solutions according to the analysis results to provide intelligent decision-making support for decision-makers in the aviation field.

[0091] Embodiment 2:

[0092] An aviation data receipt analysis method based on machine learning and web crawler of the present invention includes five steps: data collection, data preprocessing, machine learning, decision-making support, and visualization display.

[0093] Step S100 Data collection: Collect aviation data from various data sources based on web crawler technology. The data sources include flight information websites, airport official websites, and open data sources. The aviation data includes flight information, airport information, airline operation data, and weather information.

[0094] As a specific implementation of data collection, this step regularly collects aviation data from various data sources based on the set crawling frequency, and pushes and updates the aviation data to the data preprocessing module through WebSocket technology.

[0095] Data collection, as an important part of the method in this embodiment, is responsible for automatically scraping aviation-related data from the Internet, providing a basis for subsequent data processing and analysis. When selecting crawling targets, this module focuses on flight information websites, airport official websites, and open data sources to ensure the comprehensiveness and real-time nature of the data.

[0096] This step extracts key information including flight information (such as flight number, takeoff and landing times, etc.), airport information (such as airport name, location, etc.), airline operation data, and weather information. Through the processing and comprehensive analysis of different types of data, it is possible to understand the dynamics of the aviation market, optimize flight schedules, and enhance the passenger experience, etc.

[0097] To achieve regular or real-time crawling of data, this step sets the crawling frequency according to requirements, whether it is daily, hourly, or real-time update. For real-time data, the WebSocket technology will be used to achieve fast data push and update.

[0098] Step S200: Data preprocessing - Perform data cleaning operations on the collected aviation data and store the cleaned aviation data in a database.

[0099] In this embodiment, aviation data includes structured data and unstructured data. Among them, flight information and airport information belong to structured data, and web page text, comment-based airline operation data, and weather information belong to unstructured data.

[0100] For structured data, perform the following data cleaning operations: According to the nature and distribution of the data, select to delete records containing missing values, use the mean, median, or mode for filling, set thresholds based on business logic and data distribution, and identify and remove outliers based on statistical methods. Remove duplicate data by comparing the unique identifiers of aviation data. Among them, the statistical method includes the IQR rule, and the unique identifier includes ID and timestamp. Store the cleaned structured data in a relational database, organize the data using table structures and relationships, and determine the storage form of the data according to business requirements and data characteristics.

[0101] For unstructured data, perform the following data cleaning operations: Remove useless information, noise, and redundant data to ensure the accuracy and consistency of the data. For text-type aviation data, perform word segmentation, stop word removal, and stemming operations. Store the cleaned unstructured data in a non-relational database and determine the storage form of the data according to business requirements and data characteristics.

[0102] This step is responsible for performing a series of processing and conversions on the obtained original data to ensure the quality and usability of the data.

[0103] First, the original data mainly includes structured data (such as flight information, airport information, etc.) and unstructured data (such as web text, comments, etc.). According to the nature and distribution of the data, choose to delete records containing missing values, use statistics such as mean, median or mode for filling; according to business logic and data distribution, set thresholds and use statistical methods (such as the IQR rule) to identify and remove outliers; remove duplicate records by comparing the unique identifiers of the data (such as ID, timestamp, etc.);

[0104] Secondly, for unstructured data, remove useless information, noise and redundant data to ensure the accuracy and consistency of the data. For text data, perform basic preprocessing steps such as word segmentation, stop word removal, and stemming.

[0105] After data cleaning, the structured data is directly saved to a relational database (MySQL), and the table structure and relationships are used to organize the data. For unstructured data (such as text), it is saved to a non-relational database (MongoDB) for data storage and management. In addition, when saving the data, it is also necessary to determine the data storage form according to business requirements and data characteristics. Classification helps to better organize the data and facilitate subsequent analysis and processing.

[0106] Finally, the preprocessed data will be directly used for subsequent machine learning model training. According to the type and requirements of the model, perform feature engineering, data transformation and data scaling on the data, aiming to extract the key features in the data and adjust the data distribution to improve the training effect and performance of the model. After processing, the data will be divided into a training set, a validation set and a test set for model training, validation and evaluation.

[0107] Step S300 Machine Learning: Recommend to the user and support the user in selecting a machine learning model as the target machine learning model. According to the requirements of the target machine learning model, perform feature engineering, data transformation and data scaling on the cleaned aviation data to extract the key features in the aviation data, and construct a training set, a validation set and a test set based on the extracted key features. Train the target machine learning model based on the training set, and perform model validation and parameter tuning on the trained target machine learning model based on the validation set and the test set to obtain the final target machine learning model.

[0108] As a specific implementation of machine learning, this step performs the following operations:

[0109] (1) Feature engineering operation: Through exploratory data analysis, identify the key factors that affect the key business indicators from the aviation data as the key features. The key factors that affect the key business indicators include flight punctuality rate, ticket price trend, and flight popularity;

[0110] (2) Perform feature preprocessing on the selected key features, and perform missing value filling, outlier handling, feature scaling, and feature encoding operations through feature preprocessing;

[0111] (3) Recommend or select a suitable machine learning model as the target machine learning model according to the data characteristics, task complexity, and performance requirements;

[0112] (4) Use the training set as the input to train the target machine learning model, and introduce cross-validation and grid search strategies during the model training process to automatically adjust the model parameters of the target machine learning model;

[0113] (5) Based on the validation set and the test set, perform model evaluation and parameter tuning on the trained target machine learning model. Evaluate the generalization ability of the model through model evaluation, and refer to the stability, interpretability, and computational efficiency of the model for iterative optimization during parameter tuning to obtain the final target machine learning model.

[0114] In this embodiment, during the machine learning process, feature engineering needs to be carried out first. Through exploratory data analysis (EDA), the key factors affecting key business indicators such as flight punctuality rate, ticket price trend, and route popularity can be identified. This process includes the screening and evaluation of existing features, and also involves the construction of new features, such as seasonal adjustment factors calculated from historical data, prediction variables based on complex weather models, etc. In addition, to ensure the efficiency and accuracy of model training, necessary preprocessing of features will be carried out, such as missing value filling, outlier handling, feature scaling, and encoding in the previous data preprocessing process.

[0115] According to the specific business scenario and requirements, the method of this embodiment can provide an intelligent model selection mechanism. This mechanism can automatically recommend or select the most suitable machine learning algorithm based on data characteristics, task complexity, and performance requirements. In the model training stage, the module supports efficient parallel computing and distributed processing technologies, which can significantly shorten the training time of large-scale data sets. At the same time, by introducing strategies such as cross-validation and grid search, the module can automatically adjust the model parameters and find the optimal model configuration to achieve the best prediction effect.

[0116] To ensure the accuracy and reliability of the model, the method of this embodiment provides a built-in model verification and tuning process. By dividing independent validation sets and test sets, the generalization ability of the model can be objectively evaluated, and potential overfitting or underfitting problems can be discovered and solved in a timely manner. During the tuning process, while paying attention to the prediction accuracy of the model, multiple aspects such as the stability, interpretability, and computational efficiency of the model will also be considered. Through iterative optimization and fine-tuning, the performance of the model is improved to meet the needs of actual business.

[0117] Step S400 Decision Support: Collect aviation data, perform data cleaning operations on the collected aviation data, and perform feature engineering, data transformation, and data scaling on the cleaned aviation data according to the requirements of the target machine learning model to extract key features from the aviation data. Then, use the extracted key features as input and perform decision prediction through the final target machine learning model to obtain the predicted decision results. Based on the predicted decision results and the expert knowledge of the domain experts, construct the final decision support.

[0118] As a further improvement of the decision support, when providing decision support, connect to a third-party system and construct the final decision support based on the decision results predicted by the target machine learning model, the expert knowledge of the domain experts, as well as the decision suggestions and solutions of the third party. The decision support includes route optimization, ticket pricing strategies, and airline evaluations.

[0119] In the decision support process of this embodiment, based on the prediction results of the machine learning model and combined with the experience and knowledge of the domain experts, provide decision support for different scenarios. This step can generate corresponding decision suggestions or solutions according to different application scenarios and requirements, such as route optimization, ticket pricing strategies, airline evaluations, etc. At the same time, this step also supports the integration and connection with other systems to achieve data sharing and collaborative work.

[0120] Step S500 Visualization Display: Display the collected aviation data and the final decision support, and provide a query service. Through the query service, support users to customize query conditions and filter data operations.

[0121] In this embodiment, the visualization display presents the analysis results of machine learning to users in an intuitive and easy-to-understand form. Display the distribution, trends, etc. of the data through charts, images, etc. to help users better understand and use the analysis results. At the same time, this module also provides a user-friendly interaction interface, supporting users to customize query conditions, filter data, and other operations to meet the personalized needs of users.

[0122] Once the model has been fully verified and optimized, the machine learning model can be flexibly deployed into the production environment. By providing API interfaces and visualization tools that are easy to integrate, the module can easily connect to other business systems to achieve real-time or batch data processing and predictive analysis. In practical applications, the module can provide comprehensive intelligent services for airlines, airports, tourism companies, etc., such as flight delay warnings, ticket price optimization, route planning adjustments, etc.

[0123] The above has introduced in detail the aviation data collection and analysis system and method based on machine learning and web crawlers provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An aviation data collection and analysis system based on machine learning and web crawlers, characterized in that: It includes data collection module, data preprocessing module, machine learning module, decision support module and visualization display module; The data collection module is used to collect aviation data from various data sources based on web crawler technology, the data sources include flight information websites, airport official websites and open data sources, and the aviation data includes flight information, airport information, airline operation data and weather information; The data preprocessing module is used to perform data cleaning operations on the collected aviation data and store the cleaned aviation data in the database; The machine learning module is configured with a plurality of machine learning models, and is used to recommend a machine learning model to a user and support the user in selecting a machine learning model as a target machine learning model, and is used to perform feature engineering, data conversion, and data scaling processing on the cleaned aviation data according to the requirements of the target machine learning model to extract key features in the aviation data, and to construct a training set, a validation set, and a test set based on the extracted key features, and is used to perform model training on the target machine learning model based on the training set, and to perform model verification and parameter tuning on the trained target machine learning model based on the validation set and the test set, so as to obtain a final target machine learning model; The decision support module is used to call the data collection module to collect aviation data, call the data preprocessing module to perform data cleaning operations on the collected aviation data, and call the machine learning module to perform feature engineering, data conversion and data scaling on the cleaned aviation data according to the requirements of the target machine learning model to extract key features in the aviation data, and use the extracted key features as input to perform decision prediction through the final target machine learning model to obtain predicted decision results, and build final decision support based on the predicted decision results and the experience and knowledge of domain experts; The visualization display module interacts with the user through a visualization interface, is used to display the collected aviation data and the final decision support, and provides query services, and supports users to customize query conditions and filter data operations through query services.

2. The aviation data collection and analysis system based on machine learning and web crawlers according to claim 1, characterized in that: The data collection module is used to collect aviation data from various data sources based on a set crawling frequency, and push and update the aviation data to the data preprocessing module through WebSocket technology.

3. The aviation data collection and analysis system based on machine learning and web crawlers according to claim 1, characterized in that: Aviation data includes structured data and unstructured data. Flight information and airport information are structured data, while airline operation data such as web page text and comments, as well as weather information, are unstructured data. For structured data, the data preprocessing module is used to perform the following data cleaning operations: according to the nature and distribution of the data, select and delete records containing missing values, use the mean, median or mode to fill, set thresholds according to business logic and data distribution, and identify and remove outliers based on statistical methods, and remove duplicate data by comparing the unique identifiers of aviation data, wherein the statistical method includes the IQR rule, and the unique identifier includes an ID and a timestamp, and store the cleaned structured data in a relational database, organize the data using table structures and relationships, and determine the storage form of the data according to business requirements and data characteristics; For unstructured data, the data preprocessing module is used to perform the following data cleaning operations: remove useless information, noise and redundant data to ensure the accuracy and consistency of the data, perform word segmentation, remove stop words and extract stems for text-type aviation data, store the cleaned unstructured data in a non-relational database, and determine the storage form of the data based on business needs and data characteristics.

4. The aviation data collection and analysis system based on machine learning and web crawlers according to claim 1, characterized in that: The machine learning module is used to perform the following operations: Feature engineering operation: Through data exploratory analysis, key factors affecting key business indicators are identified from aviation data as key features. The key factors affecting key business indicators include flight punctuality, ticket price trends, and flight popularity. Perform feature preprocessing on the selected key features, and perform missing value filling, outlier processing, feature scaling and feature encoding operations through feature preprocessing; Recommend or select an adapted machine learning model as the target machine learning model based on data characteristics, task complexity, and performance requirements; Using the training set as input, the target machine learning model is trained, and cross-validation and grid search strategies are introduced during the model training process to automatically adjust the model parameters of the target machine learning model; Based on the validation set and test set, the trained target machine learning model is evaluated and its parameters are tuned. The generalization ability of the model is evaluated through model evaluation. When tuning the parameters, the model is iteratively optimized with reference to its stability, interpretability, and computational efficiency to obtain the final target machine learning model.

5. The aviation data collection and analysis system based on machine learning and web crawlers according to claim 1, characterized in that: The decision support module is used to connect to a third-party system to build the final decision support based on the decision results predicted by the target machine learning model, the experience and knowledge of domain experts, and the decision suggestions and plans of the third party. The decision support includes route optimization, ticket pricing strategy and airline evaluation.

6. A method for analyzing aviation data based on machine learning and web crawlers, characterized in that: Aviation data collection and analysis is achieved by an aviation data collection and analysis system based on machine learning and web crawlers as described in any one of claims 1 to 5, the method comprising the following steps: Data collection: Aviation data is collected from various data sources based on web crawler technology, including flight information websites, airport official websites and open data sources. Aviation data includes flight information, airport information, airline operation data and weather information; Data preprocessing: Perform data cleaning operations on the collected aviation data and store the cleaned aviation data in the database; Machine learning: recommends and supports users in selecting a machine learning model as the target machine learning model. According to the requirements of the target machine learning model, feature engineering, data conversion, and data scaling are performed on the cleaned aviation data to extract key features in the aviation data. Training sets, validation sets, and test sets are constructed based on the extracted key features. The target machine learning model is trained based on the training set, and the trained target machine learning model is validated and parameterized based on the validation set and test set to obtain the final target machine learning model. Decision support: Collect aviation data, clean the collected aviation data, perform feature engineering, data conversion, and data scaling on the cleaned aviation data according to the requirements of the target machine learning model to extract key features from the aviation data, and use the extracted key features as input to perform decision prediction through the final target machine learning model to obtain the predicted decision results. The final decision support is built based on the predicted decision results and the experience and knowledge of domain experts. Visual display: Display the collected aviation data and the final decision support, and provide query services. The query services support users to customize query conditions and filter data operations.

7. The aviation data collection analysis method based on machine learning and web crawlers according to claim 6 is characterized in that: During data collection, aviation data is collected from various data sources based on the set crawling frequency, and the aviation data is pushed and updated to the data preprocessing module through WebSocket technology.

8. The aviation data collection analysis method based on machine learning and web crawlers according to claim 6, characterized in that: Aviation data includes structured data and unstructured data. Flight information and airport information are structured data, while airline operation data such as web page text and comments, as well as weather information, are unstructured data. For structured data, perform the following data cleaning operations: based on the nature and distribution of the data, select and delete records containing missing values, use the mean, median or mode to fill in, set thresholds based on business logic and data distribution, and identify and remove outliers based on statistical methods, and remove duplicate data by comparing the unique identifiers of aviation data, where the statistical method includes the IQR rule, and the unique identifier includes the ID and timestamp, and store the cleaned structured data in a relational database, organize the data using table structure and relationships, and determine the storage form of the data based on business needs and data characteristics; For unstructured data, the following data cleaning operations are performed: useless information, noise and redundant data are removed to ensure data accuracy and consistency. For text-type aviation data, word segmentation, stop word removal and stem extraction operations are performed. The cleaned unstructured data is stored in a non-relational database, and the data storage form is determined based on business needs and data characteristics.

9. The aviation data collection analysis method based on machine learning and web crawlers according to claim 6, characterized in that: Machine learning involves operations such as: Feature engineering operation: Through data exploratory analysis, key factors affecting key business indicators are identified from aviation data as key features. The key factors affecting key business indicators include flight punctuality, ticket price trends, and flight popularity. Perform feature preprocessing on the selected key features, and perform missing value filling, outlier processing, feature scaling and feature encoding operations through feature preprocessing; Recommend or select an adapted machine learning model as the target machine learning model based on data characteristics, task complexity, and performance requirements; Using the training set as input, the target machine learning model is trained, and cross-validation and grid search strategies are introduced during the model training process to automatically adjust the model parameters of the target machine learning model; Based on the validation set and test set, the trained target machine learning model is evaluated and its parameters are tuned. The generalization ability of the model is evaluated through model evaluation. When tuning the parameters, the model is iteratively optimized with reference to its stability, interpretability, and computational efficiency to obtain the final target machine learning model.

10. The aviation data collection analysis method based on machine learning and web crawlers according to claim 6, characterized in that: When providing decision support, we connect to a third-party system and build the final decision support based on the decision results predicted by the target machine learning model, the experience and knowledge of domain experts, and the decision suggestions and plans of third parties. The decision support includes route optimization, ticket pricing strategy, and airline evaluation.

Images