Civil aviation data integration analysis system based on large language model

Abstract

The invention provides a civil aviation data integration analysis system based on a large language model. A construction method of the civil aviation data integration analysis system comprises the following steps that civil aviation corpora are used for training the large language model; according to a query or instruction input by a user through the natural language interaction interface, analyzing the demand of the user for the natural language instruction by using the large language model, and generating a corresponding SQL query statement; the system sends the generated SQL query statement to a database, and required data is obtained from the database; transmitting the acquired data to a BI tool for further processing and analysis, and generating corresponding reports, charts or analysis results in other forms; and returning an analysis result generated by the BI tool to the large language model, and displaying the analysis result to a user through a natural language interaction interface. Through the advanced large language model technology, various civil aviation data sources are integrated and analyzed, and the civil aviation operation efficiency and the service quality are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data analysis, and particularly relates to a civil aviation data integration analysis system based on a large language model. BACKGROUND

[0002] With the rapid development of the civil aviation industry, the number of flights and passenger flow are increasing, and the types and quantities of civil aviation data are also greatly increasing. These data include but are not limited to flight information, passenger feedback, flight data, etc. How to effectively integrate and analyze these massive data has become the key to improving the efficiency of civil aviation operation and the quality of service. Traditional data analysis methods often face the following challenges when dealing with such large and complex data: 1. Data source diversity: civil aviation data comes from a wide range of sources, with different data formats and structures, making it difficult to process uniformly; 2. Large data volume: a large amount of data generated every day requires efficient storage and processing capacity; 3. Real-time requirements: some data (such as weather data, flight dynamics) need to be processed and analyzed in real time to make timely decisions; 4. Lack of depth of analysis: traditional methods have limitations in the depth and breadth of data analysis, making it difficult to provide comprehensive insights and predictions.

[0003] Therefore, how to effectively solve the shortcomings of traditional data analysis methods and improve the efficiency and accuracy of civil aviation data analysis is a problem to be solved by those skilled in the art. SUMMARY

[0004] The present application aims to build a civil aviation data integration analysis system based on a large language model. The system is based on a large language model, which is trained using a large amount of civil aviation domain corpus to ensure that the model has a deep understanding of civil aviation professional terms. At the same time, the system can automatically expand data sources to ensure that it can adapt to various civil aviation data sources, thereby significantly improving the efficiency and accuracy of civil aviation data analysis.

[0005] In order to achieve the purpose of the application, the technical solution provided by the present application is as follows: A civil aviation data integration analysis system based on a large language model, characterized in that its construction method comprises the following steps: S1: training a large language model using civil aviation corpus; S2: according to the query or instruction input by the user through the natural language interaction interface, using the large language model to analyze the user's demand and generate the corresponding SQL query statement; S3: the system sends the generated SQL query statement to the BI tool to generate the corresponding report, chart or other form of analysis result; S4: The system returns the analysis results generated by the BI tool to the large language model, and the large language model displays the analysis results to the user through a natural language interaction interface.

[0006] Further, the step S1 specifically comprises the following steps: S1.1: Collecting data required for training the large language model; S1.2: Preprocessing the collected data, including cleaning, labeling, enhancing, and noise injection processing; S1.3: Training the large language model: first, initializing the large language model using a pre-trained model; then training the large language model, the training process includes data loading, calculating the output of the input data through the model, calculating the loss between the predicted result and the true label, updating the model parameters through the back propagation algorithm, and using but not limited to SwanLab to monitor the loss, accuracy and other indicators during the training process; S1.4: Evaluation and deployment of the large language model: use the validation set to evaluate the performance of the model to ensure that it meets the expected requirements.

[0007] Further, in the step S1.2, the operation method of cleaning, labeling, enhancing, and noise injection processing of data is as follows: (1) Data cleaning: including deduplication, denoising, and formatting, deduplication can delete duplicate records, denoising can remove irrelevant information and noise data, and formatting can unify the data format; (2) Data labeling: including named entity recognition, i.e., recognizing and labeling entities; (3) Data enhancement: including data expansion, generating more samples through translation, back-translation, etc.: (4) Noise injection: adding appropriate noise to the data to improve the robustness of the model.

[0008] Further, the step S2 comprises the following steps: S2.1: Adding new data sources to the large language model database using the BI tool; S2.2: The user inputs queries or instructions through the natural language interaction interface of the large language model; S2.3: After receiving the user's input query or instruction, the system uses the large language model to analyze the natural language instruction and understand the user's needs; S2.4: According to the analysis result, the system generates the corresponding SQL query statement using the large language model.

[0009] Further, the step S3 comprises the following steps: S3.1: Connecting the BI tool to the database; S3.2: BI tool uses the transmitted SQL query statement to query the corresponding data in the database; S3.3: Create various charts and reports according to the queried data.

[0010] The civil aviation data integration and analysis system based on the large language model has the characteristics that the large language model used by the system is trained using a large amount of civil aviation field corpus, ensuring that the model has a deep understanding of civil aviation professional terms, and the BI tool can automatically expand the data source, enabling the model to integrate and analyze multiple civil aviation data sources.

[0011] The present application has the following beneficial effects: As an advanced artificial intelligence technology, the large language model (LLM) has powerful data processing and analysis capabilities. By introducing the large language model, the effect of civil aviation data analysis can be significantly improved. The civil aviation data integration and analysis system based on the large language model has the following advantages: first, it uses civil aviation corpus to train the large model, enabling the system to efficiently analyze civil aviation data. In addition, the system has the ability to automatically expand the data source, enabling it to integrate and analyze multiple civil aviation data sources. The system can play an important role in flight scheduling optimization, passenger service improvement, safety management and other fields, improving the efficiency and accuracy of civil aviation data analysis, and providing strong technical support for the intelligent and digital transformation of the civil aviation industry. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a construction flowchart of a civil aviation data integration and analysis system based on a large language model of the present application; Figure 2 is a flowchart of generating a SQL query statement according to a user input instruction; Figure 3 is a schematic diagram of generating corresponding analysis results by BI tool analysis; Figure 4 is a flowchart of returning results. DETAILED DESCRIPTION

[0013] The present application will be described in detail below in conjunction with the drawings and examples. The examples listed in the present application are only a part of the examples of the present application. Based on the technical solutions provided by the present application, those skilled in the art can obtain other implementation manners without creative labor, which are not described in detail here.

[0014] As Figure 1 shown, a civil aviation data integration and analysis system based on a large language model, its construction method includes the following steps: S1: Train a large language model using civil aviation corpus S1.1: Collect data required for large language model training The collected data sources include public datasets such as reports published by airlines, flight logs, aviation safety reports, etc., and proprietary data such as internal documents of airlines, customer service records, etc., and ensure that data collection complies with relevant laws and regulations and ethical standards.

[0015] S1.2: Preprocess the data, clean, label and enhance the collected data Data cleaning includes deduplication: deleting duplicate records; denoising: removing irrelevant information and noise data, for example: original data: ["can slide out ALPHA turn ALPHA outside south three two four four", "say a little about which point of the sister or which song of the sister is the most favorite", "#east two five eight four down to two o'clock#"] after denoising: ["can slide out ALPHA turn ALPHA outside south three two four four", "#east two five eight four down to two o'clock#"]; formatting: uniform data format; for example: original data: ["2023-01-01", "01 / 02 / 2023", "2023-03-03"] after formatting: ["2023-01-01", "2023-01-02", "2023-03-03"].

[0016] Data labeling includes named entity recognition (NER): identifying and labeling entities (such as flight numbers, airport codes, etc.); for example: ["CSN6535, follow the lead, goodbye"] after labeling: [("CSN6535", "flight number"), ("follow the lead", "follow the lead car")]. Data augmentation includes data expansion: generating more samples through translation, back-translation, etc., for example: original sentence: "QDA9865, read back correct, ready to report (confirmed parking position 20)" after translation: "QDA9865, read back correct, ready to report (confirmed parking position 20)" after back-translation: "QDA9865, read back correct, ready to report (confirmed parking position 20)"; noise injection: adding appropriate noise to the data to improve the robustness of the model; for example: "two hundred and ten degrees @ wind thirteen knots, XXX".

[0017] Preprocess the data, convert the training data into a specific format (such as json), and the training data contains questions and outputs sql, for example: "conversations": [ { "from": "user", "value": "Query flight numbers for flights within the past week" }, { "from": "assistant", "value": "SELECT flight_id FROM flights WHEREdeparture_time>= DATE_SUB(CURDATE(), INTERVAL 7 DAY)" } ], "system": "As an assistant", }

[0018] S1.3: Training a large language model First, model initialization is performed using a pre-trained model (such as Qwen2-7B). The training process includes data loading: loading training data using a data loader; forward propagation: calculating the output from the input data using the model; loss calculation: calculating the loss between the predicted result and the true label; and backpropagation: updating the model parameters using the backpropagation algorithm. Furthermore, tools such as SwanLab are used to monitor metrics such as loss and accuracy during training.

[0019] S1.4: Evaluation and Partial Analysis of Large Language Models The model's performance is evaluated using a validation set to ensure it meets expected requirements. The fine-tuned model is then deployed to real-world applications for online inference and service.

[0020] S2: Based on the query or instruction entered by the user through the natural language interaction interface, use the large language model to parse the natural language instruction to meet the user's needs and generate the corresponding SQL query statement.

[0021] The process is as follows Figure 2 As shown, the specific method is as follows: S2.1: Use BI tools to add new data sources to the large language model database; The system adds a new data source to the database: Taking Tableau as an example, the BI tool connects Tableau to the database; a new Hyper file is created using the Hyper API, and the table structure is defined; the data source is updated, and new data is inserted into the Hyper file; the generated Hyper file is then published to Tableau Server.

[0022] S2.2: User inputs query or instruction through natural language interface of large language model: User inputs query or instruction through natural language interface, for example: "Query the punctuality rate of Boeing 777 in the past week".

[0023] S2.3: After receiving user input, the system uses large language model to analyze natural language instruction and understand user's demand: The system uses Python (Flask or Django) to receive user input, and after receiving user input, it uses large language model (such as Qwen2, etc. Open source large language model) to analyze natural language instruction and understand user's demand. For example, it is parsed that the user needs to query the punctuality rate of Boeing 777 in the past week.

[0024] S2.4: According to the analysis result, the system uses large language model to generate corresponding SQL query statement Large language model obtains the table and field in the database that stores relevant information through pre-defined database schema and field mapping; then finds the corresponding table in the database through table name, field name and synonym and semantic matching obtained by parsing user demand; then adds table structure information to the prompt so that the model can understand the schema of the database; finally, the large language model generates the corresponding SQL query. The code implementation steps are: 1. The get_table_schema() function extracts table structure information from the database and returns a dictionary containing table name and column name 2. In the query_database() function, add table structure information to the prompt so that the model can understand the schema of the database 3. Use LLMChain and PromptTemplate in LangChain library to generate SQL query. For example, the large language model generates SQL statement to query the punctuality rate of Boeing 777 in the past week. First, connect to the database and extract table structure information by executing the SQL command SHOW TABLES to get the names of all tables, then execute the DESCRIBE command for each table to get its column names, store these information in a dictionary where the key is the table name and the value is the list of column names. Then, in the query_database() function, add the extracted table structure information to the prompt. The prompt should contain the database schema (i.e. table structure information) and the user's natural language query. Then, pass this prompt to the large language model to generate the corresponding SQL query. Finally, use LLMChain and PromptTemplate in LangChain library to generate SQL query. The generated SQL statement is as follows: SELECT flight_id, departure_time, arrival_time, status FROM flights WHERE aircraft_type = 'Boeing 777' AND departure_time>= DATE_SUB(CURDATE(), INTERVAL 7 DAY) AND status = 'On Time'; S3: The system sends the generated SQL query statement to the BI tool to generate corresponding reports, charts, or other forms of analysis results.

[0025] The process is as follows Figure 3 As shown, the specific method is as follows: S3.1: Connect the BI tool to the database: Connect the BI tool to the database using the server name, database name, username, and password.

[0026] S3.2: The BI tool uses the transmitted SQL statement to query the corresponding data in the database: The BI tool uses custom SQL queries to obtain the required data.

[0027] S3.3: BI tools create various charts and reports based on the retrieved data. BI tools use previously created custom SQL queries as data sources to generate charts and reports. Taking Tableau as an example, the Tableau Hyper API is used to generate analysis results. The system connects to the database through the TSC library, executes the SQL query, and stores the results in a data frame; the generated Hyper file is published to Tableau Server; and a new workbook is created using the published data source to generate the analysis results.

[0028] S4: Return the analysis results generated by the BI tool to the large language model, which then displays the analysis results to the user through a natural language interactive interface.

[0029] The process is as follows Figure 4 As shown, the specific method is as follows: S4.1: Return the analysis results generated by the BI tool to the large language model: Return the analysis results generated by the BI tool to the large language model in the backend.

[0030] S4.2: The large language model presents the analysis results to the user through a natural language interactive interface. For example, it may display "The on-time rate of Boeing 777 flights was 85% over the past week," along with corresponding charts or reports.

[0031] This invention discloses a civil aviation data integration and analysis system based on a large language model. Based on a large language model trained using civil aviation corpus and combined with automatically expanded data sources, it can significantly improve the efficiency of civil aviation data analysis.

[0032] The above embodiments are merely illustrative of the technical solutions of the present invention. It should be noted that those skilled in the art should understand that the solution is not limited to the above embodiments. Any equivalent or approximate substitutions or changes made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solutions and inventive concepts of the present invention, should also be considered within the scope of protection of the present invention.

Claims

1. A civil aviation data integration and analysis system based on a large language model, characterized in that, The system construction method includes the following steps: S1: Use civil aviation corpus to train a large language model; S2: Based on the query or command input by the user in the natural language interaction interface, use the large language model to parse the natural language command to meet the user's needs and generate the corresponding SQL query statement. S3: The system sends the generated SQL query statement to the BI tool to generate corresponding reports, charts, or other forms of analysis results; S4: The system returns the analysis results generated by the BI tool to the large language model, which then displays the analysis results to the user through a natural language interactive interface.

2. The civil aviation data integration and analysis system based on a large language model according to claim 1, characterized in that, Step S1 includes the following steps: S1.1: Collect the data needed for training the large language model; S1.2: Preprocess the collected data: including cleaning, labeling, enhancement, and noise injection. S1.3: Training the large language model: First, the large language model is initialized using a pre-trained model; then the large language model is trained. The training process includes data loading, calculating the output of the input data through the model, calculating the loss between the predicted results and the true labels, updating the model parameters through the backpropagation algorithm, and using, but not limited to, SwanLab to monitor the loss and accuracy during the training process. S1.4: Evaluation and Deployment of Large Language Models: Use a validation set to evaluate the performance of the model and ensure that it meets the expected requirements.

3. The civil aviation data integration and analysis system based on a large language model according to claim 2, characterized in that, In step S1.2, the data cleaning, labeling, enhancement, and noise injection processes are performed as follows: (1) Data cleaning: including deduplication, noise reduction and formatting. Deduplication can delete duplicate records, noise reduction can remove irrelevant information and noisy data, and formatting can unify the data format. (2) Data annotation: including named entity recognition, i.e., identifying and annotating entities; (3) Data augmentation: This includes data augmentation, generating more samples through methods such as translation and back-translation. (4) Noise injection: Add an appropriate amount of noise to the data to improve the robustness of the model.

4. The civil aviation data integration and analysis system based on a large language model according to claim 1, characterized in that, Step S2 includes the following steps: S2.1 Use BI tools to add new data sources to the large language model database; S2.2: Users input queries or commands through the natural language interactive interface of the large language model; S2.3: After receiving a query or instruction from the user, the system uses a large language model to parse the natural language instruction and understand the user's needs; S2.4: Based on the parsing results, the system uses the large language model to generate the corresponding SQL query statement.

5. The civil aviation data integration and analysis system based on a large language model according to claim 1, characterized in that: Step S3 includes the following steps: S3.1: Connect the BI tool to the database; S3.2: The BI tool uses the transmitted SQL query statement to retrieve the corresponding data in the database; S3.3: Create various charts and reports based on the retrieved data.

6. The civil aviation data integration and analysis system based on a large language model according to claim 1, characterized in that: The large language model on which the system is based is trained using a large amount of corpus in the civil aviation field to ensure that the model has a deep understanding of civil aviation terminology. At the same time, through BI tools, the data source can be automatically expanded, enabling the model to integrate and analyze multiple data sources in the civil aviation field.

Images