Multi-modal large model flight decision-making agent construction method and system and readable storage medium

Abstract

The invention provides a multi-modal large model flight decision agent construction method and system and a readable storage medium, and the method comprises the steps: S1, receiving time sequence flight data, then carrying out the data processing, and generating a regular time sequence data set; s2, abnormal change detection is carried out, and an abnormal decision reasoning text is generated after integration; s3, judging the knowledge graph and the abnormal decision reasoning text in a preset large model, and outputting a first or second judgment result; s4, performing fault mechanism reasoning according to the first judgment result, and outputting a plurality of pieces of fault alarm information; and S5, according to the second judgment result, integrating single and clear alarm information or multiple pieces of fault alarm information with modal data to form multi-modal data, carrying out alignment and integration to form a data set in a unified form, inputting the data set into a preset large model, and carrying out judgment and control iteration. And a comprehensive fault troubleshooting scheme is output for a flight crew and maintenance maintenance to make decisions. According to the invention, the flight safety can be further improved.

Description

Technical Field

[0001] The present invention relates to the cross - technical field of computer technology and avionics system design technology, and particularly relates to a method, system and readable storage medium for constructing a multi - modal large - model flight decision - making intelligent agent.

Background Art

[0002] In the field of civil aviation, due to some incorrect decisions, safety accidents often occur. The investigation by Boeing on global flight accidents from 1996 to 2007 shows that flight accidents caused by crew errors account for 73.1%, flight accidents caused by aircraft hardware failures account for 8%, flight accidents caused by maintenance reasons account for 7.2%, flight accidents caused by weather factors account for 4.1%, and flight accidents caused by other factors such as airports account for 7.6%. It can be seen that the proportion of flight accidents caused by crew human factors is much higher than that caused by other factors. Therefore, it is particularly important for the flight crew to make correct flight decisions. Here, the decision - making not only includes the decision of whether to start flying before the flight, but also reflects various decision - making contents during the entire flight process. According to the requirements of the Advisory Circular AC60 - 22 "Aviation Decision - Making" issued by the FAA in 1991, flight decision - making is "a cognitive process method in which a pilot makes a series of operations to respond when facing a given set of scenarios". Among them, scenario perception, also known as situation awareness, refers to the prediction of the nearest future state based on the perception units in a certain spatio - temporal environment and the understanding of the meaning of these perception units. Scenario perception is divided into three levels. The first level is the perception of hidden information in the critical environment, the second level is the understanding of the impact and importance of these environmental factors on personal goals, and the third level is the prediction of potential future events of the system.

[0003] After rational thinking + scenario perception + empirical knowledge, the flight crew will comprehensively consider all possible options, give risk assessment, judgment, selection and execution, and finally judge and re - evaluate the scenario. This entire set of iterative update cognitive closed - loop of perception, thinking, execution, reflection, and re - execution... constitutes all the content of flight decision - making.

[0004] The research on flight decision-making models in the field of civil aviation has a long history. Since 1975, when Benner first proposed the DECIDE decision framework model, which is representative of the normative model decision-making (DECIDE / FOR-DEC / SOAR / DESIDE), attempts have been made to structure the decision-making process. The descriptive decision-making models represented by the SHOR / ARTFUL / RPD / PCM decision-making models pay more attention to how decision-makers use perception, knowledge, and experience to make decisions. The decision intelligence of this project integrates the achievements of normative models and descriptive models, mainly consisting of three parts: QAR data (descriptive decision-making model), flight manual knowledge graph (normative decision-making model), and intelligent agent (descriptive decision-making model). Among them, the flight manual knowledge graph is responsible for the long-term memory storage function of decision intelligence, the flight data provided by the aircraft system provides real-time dynamic state information of the aircraft and is responsible for the short-term memory storage function of decision intelligence. The decision intelligent agent is composed of planning, memory, tool use, and action, and is responsible for the information processing and decision implementation functions of decision intelligence. It is decomposed into instructions, logical bases, and examples to be responsible for the trustworthy explanation function of decision intelligence.

[0005] Therefore, it is necessary to study a method, system, and readable storage medium for constructing a multi-modal large model flight decision intelligent agent to address the deficiencies of the existing technology and solve or mitigate one or more of the above problems.

Summary of the Invention

[0006] In view of this, the present invention provides a method, system, and readable storage medium for constructing a multi-modal large model flight decision intelligent agent, aiming at innovation in software design systems and model algorithm methods for the intelligent auxiliary decision-making functions resident in electronic flight bag products and integrated modular avionics (IMA) airborne products. It can be used for autonomous fault warning in intelligent auxiliary flight of civil aircraft and generating fault troubleshooting solutions for autonomous flight decision-making of general aviation aircraft. It can reduce the workload of flight crews in dealing with abnormal alarms in complex flight scenarios, anticipate abnormal states in advance before fault alarms, remind maintenance personnel to eliminate abnormal states in a timely manner, and eliminate faults at the budding stage, which helps to further improve flight safety.

[0007] On the one hand, the present invention provides a method for constructing a multi-modal large model flight decision intelligent agent, and the method for constructing the flight decision intelligent agent includes the following steps:

[0008] S1: After receiving the sequential flight data, perform data processing to generate a regular sequential data set;

[0009] S2: Perform abnormal change detection on the regular sequential data set, and after integration, generate abnormal decision inference texts corresponding to each abnormal flight one by one;

[0010] S3: Structure the crew operation manual to generate a knowledge graph, and judge the knowledge graph and the abnormal decision-making inference text in a preset large model to output the first or second judgment result;

[0011] S4: According to the first judgment result, that is, there is no clear and unique warning information, conduct a fault mechanism inference, output multiple fault warning messages, and then input them into the preset large model to judge and sort these abnormal contents according to the fault mechanism to determine which fault is most likely to occur, and use this most likely fault warning message as the output;

[0012] S5: According to the second judgment result, that is, there is a single clear warning information, integrate the single clear warning information or multiple fault warning messages with the modal data to form multi-modal data, align and integrate the multi-modal data, form a unified form of data set and input it into the preset large model for judgment and control iteration, and output a comprehensive fault troubleshooting plan for the flight crew and maintenance personnel to make decisions.

[0013] In the aspects and any possible implementation manners as described above, a further implementation manner is provided. The data processing process in S1 includes but is not limited to data alignment, feature correlation analysis, data cleaning, and data visualization processing.

[0014] In the aspects and any possible implementation manners as described above, a further implementation manner is provided. Specifically, S2 includes:

[0015] S21: Abnormal detection, preliminarily detect the abnormal feature information with abnormal changes in the input regular time series data set through an abnormal detection tool;

[0016] S22: Abnormal decision-making inference, integrate the abnormal feature information and input it into the large model for preliminary abnormal decision-making inference to form an abnormal decision-making inference text corresponding to each abnormal flight.

[0017] In the aspects and any possible implementation manners as described above, a further implementation manner is provided. Specifically, S3 is as follows: Receive the abnormal decision-making inference text from the abnormal detection text data generation stage and the knowledge graph data after structuring the crew operation manual, judge in the large model, and output the first or second judgment result. When the result is the second judgment result, that is, there is a single clear warning information, directly perform S5. When the result is the first judgment result, that is, the unique warning information cannot be determined, perform S4 and then S5.

[0018] For the aspects and any possible implementation manners described above, a further implementation manner is provided. Specifically, S4 is as follows: For several pieces of abnormal description content for which the unique alarm information cannot be determined, the most likely fault solutions are retrieved respectively, and these fault solutions are input into a preset large model for comprehensive logical judgment and sorting, and finally multiple fault alarm messages are output.

[0019] For the aspects and any possible implementation manners described above, a further implementation manner is provided. In S5, the modal data includes but is not limited to time-series flight data, image data, voice data, and other text data. Among them, the other text data includes fault alarm data and fault knowledge graph data, the image data includes in-cabin alarm light images, fault operation images, and in-cabin and out-of-cabin status images, and the voice data includes flight crew - ATC air-ground communication voices and flight crew - maintenance crew in-cabin communication voices.

[0020] For the aspects and any possible implementation manners described above, a further implementation manner is provided. The alignment and integration process in S5 is specifically as follows: According to the different requirements of various modal data, the corresponding time-series flight data analysis tools, image analysis tools, voice recognition tools, and text analysis tools are called respectively for alignment and integration.

[0021] For the aspects and any possible implementation manners described above, a further implementation manner is provided. The preset large model in S3 and S5 is a knowledge-enhanced multi-modal large model.

[0022] For the aspects and any possible implementation manners described above, a further multi-modal large model flight decision intelligent agent construction system is provided, which is used to complete the flight decision intelligent agent construction method. The flight decision intelligent agent construction system includes:

[0023] A time-series data processing module, which is used to receive time-series flight data and perform data processing to generate a regular time-series data set;

[0024] An abnormal detection text data generation module, which is used to perform abnormal change detection on the regular time-series data set, and generate abnormal decision inference texts corresponding to abnormal flights one by one after integration;

[0025] A knowledge-enhanced large model determination module, which is used to structure the crew operation manual to generate a knowledge graph, and judge the knowledge graph and the abnormal decision inference text in a preset large model, and output a first or second judgment result;

[0026] A fault mechanism reasoning module based on a large model, which is used to perform fault mechanism reasoning according to the first judgment result, that is, there is no clear and unique alarm information, and output multiple fault alarm messages;

[0027] The multimodal intelligent agent analysis module based on large models is used to integrate the single clear warning information or multiple fault warning information with modal data according to the second judgment result, that is, there is a single clear warning information, to form multimodal data, align and integrate the multimodal data, form a dataset in a unified form, input it into a preset large model for judgment and iterative update, and output a comprehensive fault troubleshooting solution for flight crews and maintenance personnel to make decisions.

[0028] In the above aspects and any possible implementation manners, a readable storage medium is further provided, storing instructions, which when running on a computer, cause the computer to execute any one of the flight decision intelligent agent construction methods described above.

[0029] Compared with the prior art, the present invention can achieve the following technical effects:

[0030] 1. The present invention uses the original flight time-series data collected by aircraft on-board equipment as the source to generate a fault troubleshooting solution in an end-to-end process. It is no longer traditional static decision-making auxiliary information, but a dynamic decision-making process that is adjusted in real time according to the current flight state, and can more fully integrate flight situation awareness information;

[0031] 2. The present invention adopts a modular design, and different tools of large models are respectively used for targeted text generation in the three different functional modules of knowledge-enhanced determination, fault mechanism reasoning, and multimodal intelligent agents, which can better exert the knowledge advantages of large models, and the formed decision-making information is more accurate;

[0032] 3. The present invention adopts an end-to-end flight fault decision-making architecture driven by time-series flight data, logical reasoning of large models, and assisted enhancement of image / voice / text data, further improving flight situation awareness from three aspects of "vision" (image text), "hearing" (voice), and "touch" (time-series flight data obtained by aircraft sensors). The decision-making content generated in the form of multimodal intelligent agents is more credible and easier to be accepted by airworthiness authorities.

[0033] Of course, it is not necessary for any product implementing the present invention to achieve all the above-mentioned technical effects simultaneously.

Description of the Drawings

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0035] Figure 1It is a framework diagram for constructing a flight decision-making intelligent agent of a knowledge-enhanced multi-modal large model provided by an embodiment of the present invention;

[0036] Figure 2 It is a schematic diagram of the time-series data processing stage provided by an embodiment of the present invention;

[0037] Figure 3 It is a visualization schematic diagram for judging the threshold of the number of abnormal data provided by an embodiment of the present invention;

[0038] Figure 4 It is a schematic diagram for generating an abnormal detection text data set provided by an embodiment of the present invention;

[0039] Figure 5 It is a schematic diagram of an instance of the abnormal text data set provided by an embodiment of the present invention;

[0040] Figure 6 It is a schematic diagram of the determination stage of the knowledge-enhanced large model provided by an embodiment of the present invention;

[0041] Figure 7 It is a schematic diagram of the determination process of the knowledge-enhanced large model provided by an embodiment of the present invention;

[0042] Figure 8 It is a schematic diagram of the fault mechanism reasoning stage based on the large model provided by an embodiment of the present invention;

[0043] Figure 9 It is a schematic diagram of the multi-modal intelligent agent stage based on the large model provided by an embodiment of the present invention;

[0044] Figure 10 It is a post-flight analysis result diagram of a flight with fuel imbalance fault provided by an embodiment of the present invention;

[0045] Figure 11 It is a schematic diagram of the query result of the flight manual related to the fault (for flight crew use) provided by an embodiment of the present invention;

[0046] Figure 12 It is a schematic diagram of the fault troubleshooting solution result (for maintenance mechanics use) provided by an embodiment of the present invention.

Specific Embodiments

[0047] For a better understanding of the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] It should be clear that the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0049] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a", "the", and "said" used in the embodiments of the present invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0050] The present invention provides a method for constructing a multi-modal large model flight decision-making agent. The method for constructing the flight decision-making agent includes the following steps:

[0051] S1: After receiving the sequential flight data, perform data processing to generate a set of regular sequential data;

[0052] S2: Perform anomaly change detection on the set of regular sequential data, and after integration, generate anomaly decision inference texts corresponding one by one to the abnormal flights;

[0053] S3: Structure the crew operation manual to generate a knowledge graph, and judge the knowledge graph and the anomaly decision inference text in a preset large model to output a first or second judgment result;

[0054] S4: According to the first judgment result, that is, there is no clear and unique warning information, perform fault mechanism reasoning, output multiple fault warning information, and then input it into the preset large model to judge and sort these abnormal contents according to the fault mechanism to determine which fault is most likely to occur, and use this most likely fault warning information as the output;

[0055] S5: According to the second judgment result, that is, there is a single clear warning information, integrate the single clear warning information or multiple fault warning information with the modal data to form multi-modal data, align and integrate the multi-modal data to form a dataset in a unified form, input it into the preset large model for judgment and control iteration, and output a comprehensive fault troubleshooting plan for the flight crew and maintenance personnel to make decisions.

[0056] The data processing process in S1 includes but is not limited to data alignment, feature correlation analysis, data cleaning, and data visualization processing.

[0057] The specific steps of S2 include:

[0058] S21: Anomaly detection, preliminarily detect the abnormal feature information with abnormal changes in the input set of regular sequential data through an anomaly detection tool;

[0059] S22: Anomaly decision inference, integrate the abnormal feature information and input it into the large model for preliminary anomaly decision inference to form anomaly decision inference texts corresponding one by one to the abnormal flights.

[0060] Specifically, S3 is as follows: Receive the abnormal decision-making inference text from the abnormal detection text data generation stage and the knowledge graph data structured from the crew operation manual, make a judgment in the large model, and output the first or second judgment result. When the result is the second judgment result, that is, there is a single clear warning message, directly proceed to S5. When the result is the first judgment result, that is, the unique warning message cannot be determined, proceed to S4 and then S5.

[0061] Specifically, S4 is as follows: For several abnormal description contents where the unique warning message cannot be determined, retrieve the most likely fault solutions respectively, and input these fault solutions into the preset large model for comprehensive logical judgment and sorting, and finally output multiple fault warning messages.

[0062] The modal data in S5 includes but is not limited to time-series flight data, image data, voice data, and other text data. Among them, other text data includes fault warning data and fault knowledge graph data, image data includes in-cabin warning light images, fault operation images, and in-cabin and out-of-cabin status images, and voice data includes flight crew - ATC air-ground communication voices and flight crew - maintenance crew in-cabin communication voices.

[0063] Specifically, the alignment and integration process in S5 is as follows: Call the corresponding time-series flight data analysis tool, image analysis tool, voice recognition tool, and text analysis tool respectively according to the different requirements of various modal data for alignment and integration.

[0064] The present invention also provides an implementation manner. The preset large models in S3 and S5 are knowledge-enhanced multi-modal large models.

[0065] For the aspects and any possible implementation manners described above, a multi-modal large model flight decision intelligent agent construction system is further provided. The flight decision intelligent agent construction system includes:

[0066] A time-series data processing module, configured to receive time-series flight data and perform data processing to generate a regular time-series data set;

[0067] An abnormal detection text data generation module, configured to perform abnormal change detection on the regular time-series data set, and generate abnormal decision-making inference texts corresponding to abnormal flights one by one after integration;

[0068] A knowledge-enhanced large model determination module, configured to structure the crew operation manual to generate a knowledge graph, and make a judgment on the knowledge graph and the abnormal decision-making inference text in the preset large model, and output the first or second judgment result;

[0069] The fault mechanism reasoning module based on the large model is used to perform fault mechanism reasoning according to the first judgment result, that is, there is no clear and unique alarm information, and output multiple fault alarm messages;

[0070] The multi-modal intelligent agent analysis module based on the large model is used to integrate the single clear alarm information or multiple fault alarm messages with the modal data according to the second judgment result, that is, there is a single clear alarm information, to form multi-modal data, align and integrate the multi-modal data, and form a unified form of data set to be input into the preset large model for judgment and control iteration, and output a comprehensive fault troubleshooting plan for the flight crew and maintenance personnel to make decisions.

[0071] The present invention also provides a readable storage medium storing instructions, which when run on a computer, cause the computer to execute any one of the flight decision intelligent agent construction methods described above.

[0072] Embodiment 1:

[0073] The present invention provides a method for constructing a multi-modal large model flight decision intelligent agent. Using time-series flight data as the driving force, anomaly detection text generation and intelligent agent analysis decision generation as two generative modules, conducting multiple rounds of dialogue interactions with the pilot and optimizing and iterating the dialogue content with the help of an external knowledge base, and finally forming an operable fault solution based on the current flight scenario. The whole process constitutes a knowledge-enhanced multi-modal large model flight decision intelligent agent framework system. This system is divided into five stages: the time-series data processing stage, the anomaly detection text data set generation stage, the knowledge-enhanced large model determination stage, the large model-based fault mechanism reasoning stage, and the large model-based multi-modal intelligent agent stage, as Figure 1 shown.

[0074] The first stage is the time-series data processing stage. This stage receives time-series flight data as input, and after data alignment, feature correlation analysis, data cleaning, and data visualization, forms a regular time-series data set and transmits it to the subsequent anomaly detection text data generation stage.

[0075] The second stage is the anomaly detection text data generation stage. This stage includes two parts: an anomaly detection tool and anomaly decision reasoning. The anomaly detection tool initially detects whether there are abnormal change feature information in the input regular time-series data, and integrates these abnormal features and transmits them to the large model for preliminary anomaly decision reasoning, so as to form anomaly decision reasoning texts corresponding to each abnormal flight one by one, and transmit them to the knowledge-enhanced large model determination stage.

[0076] The third stage is the knowledge-enhanced large model judgment stage. This stage receives the abnormal decision-making inference text data from the abnormal detection text data generation stage and the structured knowledge graph data from the crew operation manual, etc., and makes a judgment in the large model to determine whether there is an alarm. If a single clear alarm message is determined to exist, it directly jumps to the final large model-based multimodal agent stage. If the unique alarm message cannot be clearly identified, it enters the next large model-based fault mechanism reasoning stage.

[0077] The fourth stage is the large model-based fault mechanism reasoning stage. In this stage, for several abnormal description contents where the unique alarm message cannot be clearly identified, the most likely fault solutions are retrieved respectively, and these fault solutions are input into the large model for comprehensive logical judgment and sorting. Finally, multiple fault alarm messages are output and enter the final large model-based multimodal agent stage.

[0078] The fifth stage is the large model-based multimodal agent stage. This stage mainly processes the text data and time-series flight data (such as the data recorded by the Quick Access Recorder (QAR), etc.), image data (cabin warning lights / fault operations / cabin and external status), voice data (flight crew - ATC air-ground communication, flight crew - maintenance mechanic cabin communication), and other text data (fault alarm data, fault knowledge graph data, etc.) after the fault mechanism reasoning to form multimodal data. The system inputs the multimodal data to the agent, and the agent calls the corresponding time-series flight data analysis tool, image analysis tool, voice recognition tool, and text analysis tool according to the different requirements of various modal data, so as to align and integrate these multimodal data, form a unified form of data set and input it into the agent large model for judgment and control iteration. Finally, the agent gives a more comprehensive fault troubleshooting plan for the flight crew and maintenance mechanics to make decisions.

[0079] The above five stages are further described and explained as follows:

[0080] 1. Time-series data processing stage.

[0081] In the time-series data processing stage, through data alignment, data cleaning, feature correlation analysis, and data visualization modules, the automated processing process of time-series data is carried out to establish a regular data format for the subsequent abnormal detection text generation module, and the effectiveness of the agent in identifying fault solutions is improved, as Figure 2 shown.

[0082] Among them, data alignment refers to aligning the flight times of multiple flights of the same aircraft. Due to the operating behavior habits and proficiency levels of different pilots, the start-up interval times of various devices are different for different flights. It is necessary to align the data of different flights and align the change rules of all data features according to the operating behavior habits and proficiency levels of the same pilot, so as to reduce the interference of the pilot's operating habits on data uncertainty and more objectively reflect the differences between the previous normal flight and the subsequent faulty flight.

[0083] Data cleaning refers to clearing the data feature of the sensor that cannot read valid data normally from the beginning, and automatically selecting the data feature column with relatively rich and statistically significant valid data.

[0084] Feature correlation analysis refers to performing correlation analysis on different data feature columns. Only one column is retained for multiple feature columns with strong correlation. In this way, it can be ensured that all data features in the processed data set are independent of each other, making it more credible to perform statistical analysis and determine anomalies for a single data feature.

[0085] Data visualization mainly counts the number of abnormal features for each frame of data points in the time-series data. Here, the abnormal feature threshold N is set as a hyperparameter. If the number of abnormal features of this frame of data points exceeds this threshold N, it is considered that a suspected anomaly has occurred at the moment corresponding to this frame of data points during the flight, and it is marked in red on the scatter plot, as Figure 3 shown.

[0086] 2. Abnormal detection text data set generation stage.

[0087] In the abnormal detection text data set generation stage, for the regular time-series data format, all data features of the first several flights (set to 5 flights in the present invention) are regarded as normal data, and the normal data range of each data feature in each flight stage is determined, and limited extrapolation is performed through the confidence interval to reduce the abnormal detection error caused by uncertainty. In subsequent flights, as long as the data of a certain feature exceeds the normal data confidence interval of this corresponding feature, it is considered that this feature has an anomaly. This stage is shown in the red box in Figure 4 as follows.

[0088] And so on, the list of abnormal data feature names of subsequent flights can be directly obtained, and the full names of the features corresponding to these name lists are integrated together to form the abnormal text data set of this flight. The text data set is in the three-part form of "abnormal description, abnormal analysis, abnormal conclusion" in chronological order as Figure 5As shown, each data in the abnormal text dataset corresponds to a flight with abnormal conditions. Although there are abnormalities, since the abnormal conditions have not accumulated to a certain extent, no fault alarm has been triggered to form a typical fault for the time being, but there are already signs of a fault.

[0089] 3. Knowledge-enhanced large model judgment stage.

[0090] The knowledge-enhanced large model judgment stage is mainly implemented using retrieval-augmented generation technology. It is the first large model processing stage for the abnormal text dataset, as Figure 6 shown.

[0091] Retrieval Augmented Generation is an artificial intelligence technology that combines information retrieval and text generation. It is usually used for natural language processing tasks such as question-answering systems, dialogue generation, or content summarization, and is mainly divided into an information retrieval part and a text generation part. Knowledge-enhanced large model judgment performs information retrieval on the abnormal decision inference information obtained from the abnormal detection text dataset generation stage and the fault knowledge graph information after structuring the flight manual. Here, the abnormal decision inference information is used as the question sentence, and the fault knowledge graph information is used as the answer sentence to retrieve and generate the most likely knowledge graph fragment information. Then, this fragment information is used as the input for the next text generation part and is input into the large model again to generate text generation content based on the experience of the flight manual. Its implementation process is as Figure 7 shown.

[0092] Among them, the abnormal decision inference information is the abnormal decision inference content output in the second stage. This inference content is relatively divergent. Abnormal signs related to faults have been observed, but it is impossible to accurately locate which fault. With the help of the fault handling knowledge in the flight manual knowledge graph, the most likely knowledge graph fragment can be retrieved through information retrieval first. Here, it is already possible to trace back to the hydraulic system problem in "02 Abnormal Procedures", and it is inferred that the scenarios of low pressure in the No. 2 and No. 3 hydraulic systems are most relevant to this abnormal decision inference. Immediately afterwards, based on this knowledge graph fragment, an answer set is further formed to continue to receive questions from the abnormal decision inference information. The large model will generate more targeted fault alarm information. This fault alarm information will skip the fourth stage and directly enter the fifth stage for agent processing.

[0093] 4. Fault mechanism reasoning stage based on large model.

[0094] If no clear fault alarm information can be directly obtained in the third stage, but only a list of multiple different abnormalities is obtained, then it is considered that the logical judgment ability of the large model in the third stage does not meet the expectation. At this time, it enters the fourth stage, "fault mechanism reasoning stage based on large model", as Figure 8 shown.

[0095] At this time, the output text content of this stage should be decomposed to find all possible fault contents that may cause anomalies and the corresponding fault handling solutions, and then input them into the large model to judge and sort these abnormal contents according to the fault mechanism, determine which fault is most likely to occur, and use the warning information of this most likely fault as the output to enter the 5th stage.

[0096] 5. Stage of multi-modal intelligent agent based on large model.

[0097] In the last stage, in order to make full use of the comprehensive capabilities of the multi-modal intelligent agent and go beyond text information, five modalities of data are integrated, namely text data (fault warning data, fault knowledge graph data), voice data (flight crew - ATC ground-air communication, flight crew - maintenance crew in-cabin communication), image data (in-cabin warning lights / fault operations / inside and outside cabin status), and time-series flight data (QAR data, etc.). Different tools inside the intelligent agent are used to process the corresponding modal data, and these data are aligned and comprehensively analyzed by the large model. The large model interacts with the intelligent agent through control and feedback mechanisms, enabling the intelligent agent to have the capabilities of "seeing, hearing, touching, and even having insights", forming a comprehensive and reliable fault troubleshooting plan and executing it, thus possessing the capabilities of embodied intelligence. The entire process is as Figure 9 shown.

[0098] Compared with the existing decision-making methods, the present invention calls different tools for data alignment and data fusion based on multi-modal data of time-series flight data, image data, voice data, and text data, and finally constructs a method for constructing an abnormal data training set in a unified three-stage format of "abnormal description, abnormal analysis, and abnormal conclusion"; in the present invention, a knowledge graph of flight manuals is enhanced to retrieve and generate graph fragments, assisting flight crews / maintenance crews to interact with the large model for question answering, optimizing and iteratively updating, and improving the effectiveness of multi-round interaction methods for question answering; the present invention is based on the combination of a large model and time-series flight data for fault warning judgment, as well as a method for detecting multiple fault mechanisms based on abnormal information; a fully autonomous integrated closed-loop feedback control framework with a large model and a human-machine interaction decision-making intelligent agent as the core, realizing flight scenario perception, abnormal event thinking, fault warning judgment execution, reflecting on the effect of fault warning troubleshooting, and then executing and updating the fault warning judgment results for multi-modal data such as time-series flight data, image data, voice data, and text data.

[0099] The key points of the present invention are:

[0100] 1. An end-to-end decision-making generation method driven by dynamic iterative update of time-series flight data, enhanced by multi-modal data of image data, voice data, and text data, and with a large model for fault mechanism inference and text generation;

[0101] 3. By integrating the cabin instrument image data collected by the camera with the image recognition tool integration component, the voice data of the flight crew / maintenance personnel and the intelligent agent system, and the voice recognition tool integration component, the flight manual knowledge text data and the retrieval enhanced generation tool integration component, and the time-series flight sensor bus data and the scalable intelligent agent component, according to the actual requirements of the flight scenario, flexibly select multiple scalable modules to form a scalable architecture of the dynamic flight decision-making intelligent agent.

[0102] The technology of the present invention has been verified in the QAR data of the actual operating flights of Chengdu Airlines. The faults involve three typical faults: engine failure (8 flight data), flap jam (7 flight data), and fuel imbalance (8 flight data).

[0103] Taking the flight with fuel imbalance fault as an example, it involves 7 normal flights (20170313_235729_3479643.csv, 20170314_040918_3482339.csv, 20170314_072923_3484126.csv, 20170314_102336_3486060.csv, 20170314_132426_3487539.csv, 20170314_225655_3490257.csv, 20170315_041036_3492539.csv), and 1 faulty flight (the QAR data is 20170315_081650_3493754.csv). The real fault information obtained from the post-flight analysis is as Figure 10 shown:

[0104] Without prior notice of the specific fault category, directly input the above normal flight data and faulty flight data as the original data, and use the software developed based on the model framework of the present invention (using algorithms including but not limited to the BM25 algorithm for extracting fault knowledge graph information, using ChatGLM3-6b as the large model for logical judgment. The aforementioned BM25 algorithm only provides a retrieval algorithm, and other algorithms can also be used. When using ChatGLM3-6b as the large model for logical judgment, it should not be restricted by the BM25 algorithm) model as the intermediate processing tool, and output the fault manual query text results as Figure 11 shown, and output the fault troubleshooting solution results as Figure 12 shown. The output results accurately found the fuel system fault and located the problem of the fuel pump hydraulic valve, and Figure 7Although the fuel imbalance problems caused by the air release valves analyzed by AVIC are not exactly the same, the aircraft systems where the faults are located are completely the same. This can provide guidance information for pilots' fault handling and maintenance personnel's fault diagnosis, facilitating the timely discovery of abnormal warning information 1 to 2 flights in advance, and then promptly eliminating the abnormal hidden dangers related to the faults, further ensuring flight safety.

[0105] For the future application of large models on aircraft, it is possible to consider using the Llama.cpp program written in C language to call the ChatGLM-6b large model and deploy the model algorithm under the VxWorks 653 framework, which is feasible for future airborne applications. The aforementioned VxWorks 653 framework is a type of real-time operating system, and other systems capable of model algorithm deployment can also be used.

[0106] In the present invention, the time-series flight data refers to the time-series data collected and recorded in real time by various sensors and systems on the aircraft during flight. Indexed by time stamps, it records the state, performance parameters, and environmental information of the aircraft at different time points. The specific types of time-series flight data include the following major categories: 1. Flight status data, including Altitude: the current flight altitude of the aircraft, usually measured in feet (ft) or meters (m). Speed: including Airspeed, Ground Speed, and Vertical Speed. Heading: the current heading angle of the aircraft, usually measured in degrees (°). Attitude: including Pitch, Roll, and Yaw. Position: including Longitude, Latitude, and geographical coordinates. 2. Engine data includes Engine RPM: the rotational speed of the engine, usually measured in revolutions per minute (RPM). Engine Temperature: including Exhaust Gas Temperature (EGT) and Turbine Inlet Temperature (TIT). FuelFlow: the fuel consumption rate of the engine, usually measured in pounds per hour (lb / h) or kilograms per hour (kg / h). Thrust: the thrust output of the engine, usually measured in pounds force (lbf) or kilonewtons (kN). VibrationData: the vibration level of the engine, used to monitor the health status of the engine. 3. Fuel system data, including Fuel Quantity: the remaining fuel quantity in each fuel tank, usually measured in pounds (lb) or kilograms (kg). FuelBalance: the fuel distribution between the left and right fuel tanks. Fuel Pressure: the pressure data of the fuel system. 4. Hydraulic system data, including Hydraulic Pressure: the pressure values of each hydraulic system. Hydraulic Fluid Temperature: the temperature data of the hydraulic fluid. Hydraulic FluidQuantity: the remaining quantity of the hydraulic fluid. 5. Electrical system data, including Voltage: the voltage values of each electrical system. Current: the current values of each electrical system. Battery Status: the power status of the main battery and backup battery. 6. Environmental data, including Outside Air Temperature (OAT): the temperature of the external environment of the aircraft.Barometric Pressure: The external barometric pressure value, used to calculate altitude and speed. Wind Speed and Direction: The wind speed and direction data around the aircraft. 7. Navigation system data, including: GPS data: including GPS positioning information, satellite signal strength, etc. Inertial Navigation Data: including inertial measurement unit (IMU) data such as acceleration and angular velocity. Waypoint Information: The position and distance of the aircraft's current waypoint. 8. Flight control system data, including: Control Surface Positions: including the positions of the aileron, elevator, and rudder. Autopilot Status: The current status and mode of the autopilot system. Flight Director Commands: The commands generated by the flight director system. 9. Warning and fault data, including: Warning Messages: The warning messages generated by the aircraft system, such as engine failure, hydraulic system failure, etc. Fault Codes: The fault codes recorded by the aircraft system, used for fault diagnosis. 10. Other system data, including: Cabin environment data: such as cabin temperature, humidity, barometric pressure, etc. Landing Gear Status: The retracted and extended status of the landing gear. Communication system data: including radio frequency, communication signal strength, etc. 11. Quick Access Recorder (QAR) data. QAR is a device on the aircraft used to record flight data, usually recording high-frequency time-series data, including detailed parameters of all the above categories. QAR data is an important basis for post-flight analysis and fault diagnosis.

[0107] The technology of the present invention is an end-to-end and scalable dynamic intelligent decision-making technology, mainly to solve the problem that flight crews cannot obtain effective information in a timely manner and make decision judgments quickly and accurately when encountering some special situations (such as failures, bad weather, etc.) that are likely to cause excessive workload during flight.

[0108] Among them, the end-to-end technical features are reflected in directly using the original time-series flight data obtained by aircraft sensors as input to drive a dynamic flight decision-making model composed of five sub-modules: time-series data processing, abnormal detection text dataset generation, knowledge-enhanced large model determination, fault mechanism reasoning based on the large model, and multi-modal intelligent agent based on the large model, and outputting a fault troubleshooting solution that is easy for pilots and maintenance staff to understand and use. The core of its technology is intelligent agent technology. An intelligent agent is a system that can perceive the environment, make decisions, and take actions. Among them, perceiving the environment means that the system can receive information from the surrounding environment. During the flight decision-making process, it mainly receives comprehensive environmental information such as the external meteorological conditions of the aircraft, airspace situation, and internal system status of the aircraft sensed by the aircraft body sensors. Making decisions means that the system can formulate the next action plan based on the perceived information. During the flight decision-making process, there are mainly two stakeholders. One stakeholder is the flight crew, who needs to find the disposal plan in the flight manual according to the perceived abnormal or fault warning information during the flight, temporarily perform corresponding operations on the aircraft to relieve the flight safety risk, and record the fault warning. The other stakeholder is the maintenance crew, who needs to comprehensively evaluate based on the fault warning records of the flight crew and the historical state data of the aircraft recorded by QAR data during the ground maintenance stage after the flight, find the specific system and component locations of the faults according to the fault mechanism, and replace or repair them in time to completely eliminate the faults. Taking actions means that the system executes corresponding actions according to the decision. During the flight decision-making process, after the system displays relevant fault troubleshooting solution information to the flight crew, it continuously monitors the flight status and further feedbacks the operation results according to the changes in flight data after the flight crew's fault handling operations.

[0109] The extensible technical features are reflected in that the accuracy and credibility of the fault troubleshooting solution can be improved by receiving the enhancement of in-cabin instrument image information captured by the camera, in-cabin voice of the flight crew, voice information of the communication between the flight crew and air traffic control, and text information of various fault operation manuals and flight crew manuals. Use speech recognition tools and image recognition tools (such as OCR technology) to convert the voice information and image information that may be obtained on the aircraft into instruction information that the intelligent agent can receive, form the interaction between the intelligent agent and the external world, and complete the tasks of continuous monitoring of the aircraft status and abnormal detection by calling the above-mentioned various tools. The intelligent agent can input information of different modalities through multiple interfaces and process this multi-modal information through appropriate extended tool calls, so as to achieve decision enhancement and improve the reliability of flight decisions.

[0110] The above has introduced in detail a method, system, and readable storage medium for constructing a multi-modal large model flight decision-making intelligent agent provided by the embodiments of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

[0111] As used in the specification and claims, certain terms are used to refer to specific components. Those skilled in the art should understand that hardware manufacturers may use different terms to refer to the same component. The specification and claims do not use the difference in names as a way to distinguish components, but use the difference in functions of components as the criterion for distinction. As mentioned throughout the specification and claims, the terms "comprising" and "including" are open-ended terms, so they should be interpreted as "comprising / including but not limited to". "Substantially" means within an acceptable error range. Those skilled in the art can solve the technical problem within a certain error range and basically achieve the technical effect. The subsequent description in the specification is a preferred implementation manner for implementing the present application, but the description is for the purpose of explaining the general principle of the present application and is not used to limit the scope of the present application. The protection scope of the present application shall be determined by what is defined in the appended claims.

[0112] It should also be noted that the term "including" or "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a commodity or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent in such a commodity or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the commodity or system including the said element.

[0113] It should be understood that the term "and / or" used herein is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after.

[0114] The foregoing description has shown and described several preferred embodiments of the present application. However, as previously mentioned, it should be understood that the present application is not limited to the forms disclosed herein, should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications, and environments, and can be changed within the scope of the application concept described herein through the above teachings or the techniques or knowledge in the relevant field. Any changes and variations made by those skilled in the art without departing from the spirit and scope of the present application shall fall within the protection scope of the appended claims of the present application.

Claims

1. A method for constructing a multi-modal large model flight decision-making intelligent agent, characterized in that The method for constructing a flight decision-making agent includes the following steps: S1: After receiving time-series flight data, perform data processing to generate a regular time-series data set; S2: Perform anomaly change detection on the regular time-series data set, and after integration, generate anomaly decision-making inference texts corresponding one by one to the abnormal flights; S3: Structure the crew operation manual to generate a knowledge graph, and judge the knowledge graph and the anomaly decision-making inference text in a preset large model, and output the first or second judgment result; S4: According to the first judgment result, that is, there is no clear and unique warning information, perform fault mechanism reasoning, output multiple fault warning information, and then input it into the preset large model to judge and sort these abnormal contents according to the fault mechanism to determine which fault is most likely to occur, and use this most likely fault warning information as the output; S5: According to the second judgment result, that is, there is a single clear warning information, integrate the single clear warning information or multiple fault warning information with the modal data to form multi-modal data, perform alignment and integration on the multi-modal data, form a data set in a unified form and input it into the preset large model for judgment and control iteration, and output a comprehensive fault troubleshooting plan for the flight crew and maintenance personnel to make decisions.

2. The method for constructing a flight decision-making intelligent agent according to claim 1, wherein The data processing process in S1 includes but is not limited to data alignment, feature correlation analysis, data cleaning, and data visualization processing.

3. The method for constructing a flight decision-making intelligent agent according to claim 1, wherein S2 specifically includes: S21: Anomaly detection, preliminarily detect abnormal feature information with abnormal changes in the input regular time-series data set through an anomaly detection tool; S22: Anomaly decision-making inference, integrate the abnormal feature information and input it into the large model for preliminary anomaly decision-making inference to form anomaly decision-making inference texts corresponding one by one to the abnormal flights.

4. The method for constructing a flight decision-making intelligent agent according to claim 3, wherein S3 is specifically: Receive the anomaly decision-making inference text from the anomaly detection text data generation stage and the knowledge graph data after structuring the crew operation manual, judge in the large model, and output the first or second judgment result. When the result is the second judgment result, that is, there is a single clear warning information, directly perform S5. When the result is the first judgment result, that is, the unique warning information cannot be determined, perform S4 and then S5.

5. The method for constructing a flight decision-making intelligent agent according to claim 1, wherein S4 is specifically: Retrieve the most likely fault solution for each of the abnormal description contents for which the unique warning information cannot be determined, and input these fault solutions into the preset large model for comprehensive logical judgment and sorting, and finally output multiple fault warning information.

6. The method for constructing a flight decision-making intelligent agent according to claim 1, characterized in that The modal data in S5 includes but is not limited to time-series flight data, image data, voice data, and other text data. Among them, other text data includes fault warning data and fault knowledge graph data, image data includes in-cabin warning light images, fault operation images, and in-cabin and out-of-cabin status images, and voice data includes flight crew-air traffic control ground-air communication voices and flight crew-maintenance personnel in-cabin communication voices.

7. The method for constructing a flight decision-making intelligent agent according to claim 1, wherein The alignment and integration process in S5 is specifically: Call the corresponding time-series flight data analysis tool, image analysis tool, voice recognition tool, and text analysis tool for alignment and integration according to the different requirements of various modal data.

8. The method for constructing a flight decision-making agent according to claim 1, wherein The preset large model in S3 and S5 is a knowledge-enhanced multi-modal large model.

9. A multi-modal large model flight decision-making intelligent agent construction system for implementing the flight decision-making intelligent agent construction method as described in any one of claims 1-8, characterized in that, The flight decision-making intelligent agent construction system includes: A time-series data processing module, which is used to receive time-series flight data and perform data processing to generate a regular time-series data set; An abnormal detection text data generation module, which is used to detect abnormal changes in the regular time-series data set, and generate abnormal decision-making inference texts corresponding to each abnormal flight after integration; A knowledge-enhanced large model determination module, which is used to structure the crew operation manual to generate a knowledge graph, and judge the knowledge graph and the abnormal decision-making inference text in the preset large model, and output the first or second judgment result; A fault mechanism reasoning module based on the large model, which is used to perform fault mechanism reasoning according to the first judgment result, that is, there is no clear and unique warning information, and output multiple fault warning information; A multi-modal intelligent agent analysis module based on the large model, which is used to integrate the single clear warning information or multiple fault warning information with the modal data according to the second judgment result, that is, there is a single clear warning information, to form multi-modal data, align and integrate the multi-modal data, and form a data set in a unified form to input into the preset large model for judgment and control iteration, and output a comprehensive fault troubleshooting plan for flight crews and maintenance mechanics to make decisions.

10. A readable storage medium storing instructions, characterized in that, When the instruction runs on a computer, it causes the computer to execute the flight decision-making intelligent agent construction method according to any one of claims 1-8.

Images