Pre-navigation preparation meeting scoring method and system based on large model

Abstract

The invention relates to a pre-navigation preparation meeting scoring method and system based on a large model. The invention discloses a pre-flight preparation meeting scoring method based on a large model. The pre-flight preparation meeting scoring method comprises the steps of obtaining meeting record information of a pre-flight preparation meeting and flight remark information associated with the pre-flight preparation meeting; the conference record information and the flight remark information are input into a scoring model which is constructed based on a pre-trained large model and carries out aviation field fine tuning through a pre-flight preparation annotation corpus, and the scoring model is provided with a flight operation necessary content item set; determining a flight remark information similarity score based on the conference record information and the flight remark information via a scoring model; determining, via a scoring model, a necessary content coverage score based on the conference record information and the set of flight run necessary content items; determining, via a scoring model, a dialog interactivity score based on the conference record information; and determining a comprehensive score based on the flight remark information similarity score, the necessary content coverage score, and the dialogue interactivity score.

Description

Technical Field

[0001] This disclosure relates to the field of aviation management technology, and more specifically, to a pre-flight preparation meeting scoring method and system based on a large model, as well as associated electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] Before each flight mission, the flight crew must hold a pre-flight briefing to communicate and exchange relevant information about the mission to ensure flight safety. The pre-flight briefing may cover topics such as the flight plan, weather information, contingency plans, and procedural compliance. Summary of the Invention

[0003] A brief overview of this disclosure is given below to provide a basic understanding of some aspects of it. However, it should be understood that this overview is not an exhaustive summary of this disclosure. It is not intended to identify key or essential parts of this disclosure, nor is it intended to limit the scope of this disclosure. Its purpose is merely to present certain concepts of this disclosure in a simplified form as a prelude to the more detailed description that follows.

[0004] According to a first aspect of this disclosure, a method for scoring pre-flight preparation meetings based on a large model is provided. The method includes: acquiring meeting minutes and associated flight notes from the pre-flight preparation meeting; inputting the meeting minutes and flight notes into a scoring model, which is constructed based on a pre-trained large model and fine-tuned in the aviation domain using a pre-flight preparation meeting annotation corpus; providing the scoring model with a set of necessary content items for flight operation; determining a similarity score for the flight notes based on the meeting minutes and flight notes using the scoring model; determining a necessary content coverage score based on the meeting minutes and the set of necessary content items based on the scoring model; determining a dialogue interactivity score based on the meeting minutes; and determining a comprehensive score for the pre-flight preparation meeting based on the similarity score, necessary content coverage score, and dialogue interactivity score determined by the scoring model.

[0005] According to a second aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer-executable instructions, which, when executed by the processor, cause the processor to perform the pre-flight preparation scoring method based on a large model according to a first aspect of this disclosure.

[0006] According to a third aspect of this disclosure, a computer-readable storage medium is provided having computer-executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the pre-flight preparation scoring method based on a large model according to a first aspect of this disclosure.

[0007] According to a fourth aspect of this disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor, implement the pre-flight preparation meeting scoring method based on a large model as described in the first aspect of this disclosure.

[0008] According to a fifth aspect of this disclosure, a pre-flight preparation meeting scoring system based on a large model is provided, comprising: an input module configured to acquire meeting minutes information of the pre-flight preparation meeting and flight remarks information associated with the pre-flight preparation meeting; and a scoring module coupled to the input module to receive the meeting minutes information and flight remarks information. The scoring module includes a scoring model, which is constructed based on a pre-trained large model and fine-tuned in the aviation domain using a pre-flight preparation meeting annotated corpus. The scoring model is provided with a set of necessary content items for flight operation. The scoring module is configured to: determine a similarity score of the flight remarks information of the pre-flight preparation meeting based on the meeting minutes information and the flight remarks information via the scoring model; determine a necessary content coverage score of the pre-flight preparation meeting based on the meeting minutes information and the set of necessary content items for flight operation via the scoring model; determine a dialogue interactivity score based on the meeting minutes information via the scoring model; and determine a comprehensive score of the pre-flight preparation meeting based on the similarity score of the flight remarks information, the necessary content coverage score, and the dialogue interactivity score determined by the scoring model. Attached Figure Description

[0009] The foregoing and other features and advantages of this disclosure will become clear from the following description of embodiments illustrated in conjunction with the accompanying drawings. The drawings, incorporated herein and forming a part of the specification, are further used to explain the principles of this disclosure and to enable those skilled in the art to make and use it. Wherein:

[0010] Figure 1 A flowchart is shown for a pre-flight preparation meeting scoring method based on a large model according to some embodiments of the present disclosure;

[0011] Figure 2 Non-limiting schematic diagrams of user interfaces according to some embodiments of the present disclosure are shown;

[0012] Figure 3 A flowchart illustrating a non-limiting example process of a large-model-based pre-flight preparation meeting scoring method according to some embodiments of the present disclosure is shown;

[0013] Figure 4A schematic block diagram of an electronic device according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic block diagram of a computer system on which embodiments of the present disclosure may be implemented is shown;

[0015] Figure 6 A schematic structural diagram of a pre-flight preparation meeting scoring system based on a large model, according to some embodiments of the present disclosure, is shown.

[0016] Note that in the embodiments described below, the same reference numerals are sometimes used across different figures to denote the same parts or parts with the same function, and repeated descriptions are omitted. In some cases, similar reference numerals and letters are used to denote similar items, so once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0017] For ease of understanding, the positions, dimensions, and extents of the structures shown in the accompanying drawings and other materials may not represent actual positions, dimensions, and extents. Therefore, this disclosure is not limited to the positions, dimensions, and extents disclosed in the accompanying drawings and other materials. Detailed Implementation

[0018] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0019] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this disclosure or its application or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in this disclosure. However, those skilled in the art will understand that they merely illustrate exemplary ways that can be used to implement this disclosure, and not exhaustive ways. Furthermore, the drawings are not necessarily drawn to scale, and some features may be enlarged to show details of specific components.

[0020] In addition, techniques, methods and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods and equipment should be considered part of the specification.

[0021] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0022] The quality of the pre-flight briefing directly impacts flight safety. Therefore, the content discussed during the pre-flight briefing is recorded, and the meeting minutes are scored after the meeting to determine its adequacy. If the pre-flight briefing fails, the flight crew must make corrective actions.

[0023] In related technologies, safety inspectors typically manually grade the pre-flight preparation meeting minutes (including text, audio, and video recordings) to determine the meeting's adequacy. Therefore, this grading method is inefficient. Furthermore, the grading relies heavily on the safety inspector's experience, and since different inspectors have different evaluation criteria, there is significant subjectivity in the grading. In addition, manual grading methods often only address limited, superficial dimensions, lacking depth and breadth.

[0024] To address this, this disclosure provides a large-scale model-based scoring method for pre-flight preparation meetings. It constructs a scoring model using a pre-trained large-scale model and fine-tunes the model using an aviation-specific corpus of pre-flight preparation meeting annotations. This imbues the scoring model with aviation expertise, enabling it to understand relevant information from the pre-flight preparation meeting. The scoring model then analyzes the meeting minutes to obtain a comprehensive score, standardizing and structuring the scoring process and improving the efficiency, reliability, and rationality of the scores.

[0025] The following will first combine Figure 1 A detailed description is provided of a pre-flight preparation meeting scoring method based on a large model according to various embodiments of the present disclosure (hereinafter referred to as "Method 100"). It will be understood that the actual Method 100 may include other steps, but in order to avoid obscuring the essential points of the present disclosure, these other steps will not be discussed herein and are not shown in the accompanying drawings.

[0026] like Figure 1 As shown, method 100 may include steps S102 to S112.

[0027] In step S102, the meeting minutes of the pre-flight preparation meeting and the flight notes associated with the pre-flight preparation meeting are obtained.

[0028] In some embodiments, meeting minutes may include text derived from audio files of pre-flight preparation meetings.

[0029] In some examples, meeting minutes may also include meeting documents about the pre-flight preparation meeting and participant interaction records, if any. Meeting documents may include PowerPoint presentations, PDFs, Word documents, etc., used in the meeting or otherwise associated with it; participant interaction records may describe who asked questions, who answered, etc.; and audio files may include audio and / or video files.

[0030] As a non-restrictive example, audio files of pre-flight preparation meetings can be obtained using integrated aviation-specific recording terminals, conference system interfaces (e.g., application programming interfaces (APIs) compliant with airline regulations), and dedicated interactive terminals configured by the airline, all conforming to the "Requirements for Airborne Electronic Devices" issued by the Civil Aviation Administration of China. Furthermore, meeting documents and audio files can be encrypted and stored on an aviation private cloud server compliant with the General Data Protection Regulation (GDPR) and the Civil Aviation Data Security Specification.

[0031] Meeting documents and audio files can provide the scoring model with comprehensive meeting transcripts of the pre-flight preparation meeting, thus facilitating the comprehensiveness of subsequent scoring.

[0032] In some embodiments, method 100 may further include: transcribing a speech file into timestamped text via a speech preprocessing model; and unifying different words in the text that refer to the same aviation term into the same aviation term via a text preprocessing model, and splitting the text according to conference nodes, wherein the speech preprocessing model and the text preprocessing model are models optimized with aviation domain corpus.

[0033] In some examples, the speech preprocessing model may include a denoising model and a transcription model. The denoising model reduces or removes background noise from the speech file, such as air conditioner noise or equipment noise, to improve the accuracy of subsequent transcription. The denoising model may employ adaptive filtering algorithms (e.g., but not limited to, Normalized Least Mean Square (NLMS)). The transcription model transcribes the denoised speech file into timestamped text. The transcription model may employ an Automatic Speech Recognition (ASR) model optimized with aviation-related corpora. In some examples, the ASR model may be built upon a pre-trained Whisper model (a multi-task speech-to-text model) and fine-tuned using a training dataset that includes aviation-related corpora. For example, the training dataset may include over 100,000 labeled aviation-related speech segments, comprising aviation-related audio and corresponding aviation-related text.

[0034] In some examples, text preprocessing models can include large language models optimized from aviation-related corpora. These models can cleanse (including removing redundant words and standardizing terminology) and structure the transcribed text. For example, they can remove redundant words like "um" or "this," thereby increasing the text's information density. Furthermore, they can unify different words related to the same aviation term into a single aviation term; for instance, they can unify the terms "Reduced Vertical Separation Minimum" and "RVSM" related to the aviation term "reduced vertical separation space" into "reduced vertical separation space." Unifying related words into the same aviation term through text preprocessing improves text consistency, which in turn improves the efficiency of subsequent scoring models and enhances the accuracy of text comprehension.

[0035] Furthermore, by using a text preprocessing model to split the text into meeting nodes (such as "flight planning session" and "weather analysis session"), the subsequent scoring model can perform segmented information extraction on the text of each node, thereby reducing the length of text that needs to be processed for each information extraction and reducing model illusion, because the longer the text, the higher the requirements for the model's contextual understanding ability.

[0036] Flight remarks can be considered a "standard template" prepared in advance for the pre-flight briefing, meaning the pre-flight briefing needs to interpret the flight remarks. It's understandable that the flight remarks may differ for different flight missions.

[0037] In step S104, the meeting minutes and flight notes are input into the scoring model.

[0038] The scoring model can be built upon a pre-trained large model and fine-tuned using an aviation-specific pre-flight briefing (NFB) annotation corpus. Any suitable large model, now known or developed in the future, can be used to build the scoring model. In some embodiments, the large model used is a large model employing a Transformer architecture. In some embodiments, the FFB annotation corpus includes meeting minutes from multiple real FFB sessions, along with annotations including one or more of the following: aircraft status, route selection, weather conditions, Notices of Significant Impacts (CPIs), performance limitations, and fuel calculations. For example, if a sentence from a real FFB session mentions "there will be brief thunderstorms at the landing airport," then the weather conditions mentioned in that sentence will be annotated.

[0039] The large-scale model was fine-tuned using an annotation corpus from pre-flight briefings, imbuing it with aviation expertise and enabling it to understand aviation terminology. Furthermore, by annotating content such as "aircraft status," "route selection," "weather conditions," "notification of significant impacts," "performance limitations," and "fuel calculations," the fine-tuned model can better identify these elements from meeting minutes. For example, when the pre-flight briefing mentions "there will be brief thunderstorms at the landing airport; the following alternate airports can be considered," the model can identify the "route selection" and "weather conditions" information included within the statement.

[0040] In step S106, the similarity score of the flight notes information of the pre-flight preparation meeting is determined by the scoring model based on the meeting minutes information and the flight notes information.

[0041] In some embodiments, determining the similarity score of flight notes information for a pre-flight preparation meeting based on meeting record information and flight notes information using a scoring model includes: extracting the meeting content theme of the pre-flight preparation meeting from the meeting record information and extracting the note content theme of the flight notes information from the flight notes information using the scoring model; determining the theme similarity between the meeting content theme and the note content theme; and determining the theme similarity score of the pre-flight preparation meeting based on the theme similarity; extracting aviation terms from the meeting record information and flight notes information respectively using the scoring model; determining the ratio of the number of aviation terms that exist simultaneously in the meeting record information and flight notes information to the total number of aviation terms in the flight notes information; and determining the terminology coverage score of the pre-flight preparation meeting based on the ratio; and determining the similarity score of flight notes information for the pre-flight preparation meeting based on the theme similarity score and the terminology coverage score.

[0042] In some examples, terminology coverage score has a higher weight than topic similarity score. For instance, in the broader category of flight notes similarity score, the maximum score for the terminology coverage sub-category can be higher than the maximum score for the topic similarity sub-category.

[0043] As a non-restrictive example, assume the maximum score for flight remarks similarity is 20 points, the maximum score for topic similarity is 5 points, and the maximum score for terminology coverage is 15 points. If the topic of the flight remarks is "dispatch management," while the topic of the pre-flight preparation meeting is "ground service dispatch," this indicates that the topic of the pre-flight preparation meeting is unrelated to the topic of the flight remarks, therefore the topic similarity score is 0 points. Furthermore, if only one-third of all aviation terms in the flight remarks are mentioned in the meeting minutes, the terminology coverage score is 5 points. Therefore, the total similarity score for the flight remarks in the pre-flight preparation meeting is the sum of the topic similarity score and the terminology coverage score, which is 5 points.

[0044] In step S108, the necessary content coverage score of the pre-flight preparation meeting is determined by the scoring model based on the meeting minutes information and the set of necessary content items for flight operation.

[0045] The scoring model may be provided with a set of essential flight operation content items. This set of essential flight operation content items may be summarized by aviation professionals based on best practices in aviation operations. In some embodiments, a corresponding score is assigned to each essential flight operation content item in the set. Determining the essential content coverage score of the pre-flight preparation meeting based on meeting minutes information and the set of essential flight operation content items via the scoring model may include: identifying essential flight operation content items mentioned in the pre-flight preparation meeting from the meeting minutes information using the scoring model; and determining the essential content coverage score of the pre-flight preparation meeting based on the scores corresponding to the essential flight operation content items mentioned in the pre-flight preparation meeting.

[0046] In some examples, the set of essential items for flight operations includes at least one of core alerts and other alerts. Core alerts include at least one of aircraft status, route selection, weather conditions, notices of significant impacts, performance limitations, and fuel calculations. Other alerts include at least one of earliest departure time, latest departure time, arrival time restrictions, specific risk warnings, and ad hoc notices. For example, core alerts may be assigned a higher score than other alerts. Furthermore, different essential items within core alerts may be assigned different scores. Specific risk warnings may be risk warning items from the Mandatory Occurrence Reporting and Collection System (MORCS).

[0047] As a non-restrictive example, assuming the maximum score for essential content coverage is 70 points, then the maximum score for core reminders is 67 points, and the maximum score for other reminders is 3 points. For core reminders, each reminder only scores points when mentioned, and a perfect score is only awarded if all core reminders are mentioned. For example, aircraft status scores 16 points, route selection scores 15 points, weather conditions scores 15 points, critical impact notices scores 6.8 points, performance limitations scores 6 points, and fuel calculation scores 8.2 points. For other reminders, a perfect score is awarded if any one or more of these reminders are mentioned. For example, mentioning at least one of the following—earliest departure time, latest departure time, landing time restrictions, specific risk warnings, and temporary notices—earnests 3 points.

[0048] For example, if the meeting minutes identify essential flight operation items mentioned in the pre-flight preparation meeting, such as aircraft status (e.g., scheduled maintenance status, load status, etc.), route selection (e.g., no-fly zones, alternative routes, etc.), performance limitations (e.g., decision speed, maximum climb gradient, etc.), earliest and latest takeoff times, but do not include weather conditions, important impact notices, and fuel calculations, then the score for essential content coverage of the pre-flight preparation meeting can be determined to be 40 points using a scoring model.

[0049] In step S110, a dialogue interactivity score is determined based on meeting record information using a scoring model.

[0050] In some embodiments, determining a dialogue interactivity score based on meeting transcript information via a scoring model includes: analyzing the meeting transcript information via a scoring model to score the dialogue interactivity on one or more dimensions, including dialogue rounds, question-and-answer situations, confirmation and feedback situations, naturalness, and clarity of information.

[0051] As a non-restrictive example, timestamped text can be used to determine which conversations belong to the same conversation round, which are question-and-answer conversations, and which are confirmation and feedback conversations. For example, conversations within one minute of each other and whose content is related belong to the same conversation round. Assuming the maximum score for conversation interactivity is 10 points, 1 point is awarded for each conversation round exceeding a certain threshold, and no points are awarded if it doesn't; 1 point is awarded for the presence of a question-and-answer conversation, and no points are awarded if it doesn't; 4 points are awarded for the presence of confirmation and feedback conversations, and no points are awarded if they don't; 2 points are awarded for naturalness exceeding a certain threshold, and no points are awarded if it doesn't; 2 points are awarded for clarity of information exceeding a certain threshold, and no points are awarded if it doesn't. In some cases, some or all of these dimensions can be combined for a comprehensive evaluation. For example, the three dimensions of conversation rounds, question-and-answer conversations, and confirmation and feedback conversations can be scored a total of 6 points.

[0052] By determining the interaction score of the dialogue, the interaction among flight crew members during the pre-flight briefing can be reflected. Low scores are given to phenomena such as "one-way instruction" (i.e. only the captain speaks without feedback from other crew members), which helps flight crew members improve the conduct of pre-flight briefings.

[0053] In step S112, a comprehensive score for the pre-flight preparation meeting is determined based on the similarity score of flight notes information, the necessary content coverage score, and the dialogue interactivity score of the pre-flight preparation meeting, which are determined by the scoring model.

[0054] In some embodiments, in the overall score, the flight remarks information similarity score has a first weight, the necessary content coverage score has a second weight, and the dialogue interactivity score has a third weight, wherein the second weight is greater than the first weight, and the first weight is greater than the third weight.

[0055] As a non-restrictive example, considering that for pre-flight briefings, the importance of essential flight operation content items is higher than the similarity of flight remarks information, and the similarity of flight remarks information is higher than the interactivity of the dialogue, the first weight of the flight remarks information similarity score can be 20%, the second weight of the essential content coverage score can be 70%, and the third weight of the dialogue interactivity score can be 10%. For example, the maximum score for the overall score can be 100 points, the maximum score for the flight remarks information similarity score can be 20 points, the maximum score for the essential content coverage score can be 70 points, and the maximum score for the dialogue interactivity score can be 10 points.

[0056] Therefore, the comprehensive score of the pre-flight preparation meeting determined by the scoring model can fully reflect the performance of the pre-flight preparation meeting in three dimensions: similarity of flight notes information, coverage of necessary content, and interactive dialogue. It can objectively, comprehensively, and accurately judge the quality of the pre-flight preparation meeting and achieve a more reliable pre-flight preparation meeting score.

[0057] Furthermore, compared to manual scoring methods in related technologies, the method 100 provided in this disclosure can automatically determine the overall score of the pre-flight preparation meeting through a scoring model, significantly improving scoring efficiency. Taking a single 5-minute pre-flight preparation meeting as an example, the scoring time is shortened from more than 15 minutes to less than 2 minutes, saving a significant amount of time and significantly improving scoring efficiency, thereby meeting the airline's business needs for "real-time assessment and immediate rectification".

[0058] To train the scoring ability of a large-scale model, sentences that can be scored and / or cannot be scored can be extracted from real pre-flight preparation meeting transcripts as input. The large model can then learn from expert scoring for supervised fine-tuning, and / or generate multiple scores, which can then be ranked to train a reward model for optimization through reinforcement learning. Alternatively, similar inputs can be provided to the large model, requiring it to give similar scores, and / or significantly different inputs can be provided, requiring it to give differentiated scores, thereby improving the consistency of the large-scale model's scoring. Furthermore, training through thought chains can be used, allowing the large model to explain the reasoning behind its scores before assigning them, thus improving the interpretability of the large-scale model's scoring.

[0059] After obtaining the overall score of the pre-flight briefing, the score can be fed back to the flight crew so that they can understand the quality of the pre-flight briefing and thus help them improve their pre-flight briefing preparation methods.

[0060] In some embodiments, method 100 may further include: displaying a flight remarks information similarity score, a necessary content coverage score, a dialogue interactivity score, and a comprehensive score on a user interface. In some embodiments, the scoring model may also generate corresponding scoring reasons for display on the user interface.

[0061] refer to Figure 2 This illustrates a non-limiting schematic diagram of a user interface according to some embodiments of the present disclosure. For example... Figure 2 As shown, when scoring a specific pre-flight preparation meeting, the scores obtained were as follows: similarity score of flight notes information of the pre-flight preparation meeting was 15 points, necessary content coverage score was 39.8 points, dialogue interactivity score was 10 points, and comprehensive score was 64.8 points.

[0062] In some embodiments, method 100 may further include: responding to at least one of the flight remarks information similarity score, necessary content coverage score, and dialogue interactivity score being lower than a corresponding score threshold, labeling the at least one score as a problem and generating improvement suggestions via a scoring model, and displaying the problem and improvement suggestions on the problem on a user interface. In some embodiments, in response to receiving user input associated with a problem on the user interface, meeting transcript information associated with the problem may be presented via the user interface. For example, a problem may be presented as a weakness (low score or no score), allowing administrators to click on "weakness" on the user interface to view the corresponding meeting audio clips and / or original text, facilitating verification of the scoring's reasonableness. In some cases, there may also be a need for data traceability for other score items (even high score items), so a similar configuration may be used.

[0063] Continue to refer to Figure 2 Because the flight remarks information similarity score is lower than the first score threshold (e.g., the full score for flight remarks information similarity) and the necessary content coverage score is lower than the second score threshold (e.g., the full score for necessary content coverage), the user interface displays the flight remarks information similarity score, the issue of flight remarks information similarity, and the reasoning for the score and suggestions for improvement on the issue.

[0064] Specifically, regarding the similarity of flight remarks information, the pre-flight briefing mentioned core elements such as weather analysis and alternate airport selection logic, which partially echoed the alternate route inspection requirements mentioned in the remarks. It also detailed the underlying logic of the Flight Plan Management (FPM) system and the flight plan generation mechanism, and established a complete weather monitoring plan and emergency response communication mechanism. This indicates that the aviation terminology in the meeting minutes covered some of the aviation terminology in the flight remarks. Furthermore, the topic of the presentation was consistent with the topic of the remarks, meaning the pre-flight briefing's content shared the same theme as the remarks in the flight remarks. However, the pre-flight briefing did not explicitly mention the Receiver Autonomous Integrity Monitoring (RAIM) prediction priority rules and the emergency procedures for electronic release data mentioned in the remarks; that is, some aviation terminology in the flight remarks was not covered. Therefore, the scoring model recommends supplementing the remarks with mentions of the RAIM prediction priority rules and the emergency procedures for electronic release data.

[0065] Regarding the coverage of essential content, the "weather analysis" mentioned in the pre-flight briefing covered "weather conditions," "route planning" covered "route selection," "alternate airport selection logic" covered "notification of significant impacts," and "timeline tracking" covered "other reminders." However, it failed to cover the core reminders of "aircraft status," "performance limitations," and "fuel calculation." Therefore, the scoring model recommends supplementing the coverage with information related to aircraft status, performance limitations, and fuel calculation.

[0066] In some embodiments, in response to a non-perfect score in the essential content coverage score, essential content items for flight operation that were not mentioned in the pre-flight briefing are displayed on the user interface from the set of essential content items for flight operation determined by the scoring model.

[0067] Continue to refer to Figure 2 The pre-flight briefing's essential content coverage score was 39.8 instead of the full score. Therefore, the essential flight operation items "Aircraft Status", "Performance Limitations" and "Fuel Calculation" that were not mentioned in the pre-flight briefing can be displayed on the user interface from the set of essential flight operation items determined by the scoring model.

[0068] Figure 2 The document can also show the scoring criteria for dialogue interactivity, which can help flight crews understand the scoring criteria and help them maintain good dialogue interactivity in pre-flight briefings.

[0069] Compared to manual scoring, which only outputs a final score without intermediate analysis, flight crews cannot identify shortcomings in their pre-flight briefings based on the audit results, making it difficult to help them improve their pre-flight briefings. The method provided in this disclosure supports the correlation and traceability between scoring results and meeting audio or documents. By displaying detailed scoring reasons and specific improvement suggestions for unsatisfactory scoring dimensions on the user interface, it helps flight crews understand the shortcomings of their pre-flight briefings and identify corrective measures, thereby helping them improve the quality of their pre-flight briefings and indirectly reduce flight safety risks.

[0070] In some embodiments, the scoring model can generate a scoring report that includes a comprehensive score, score curves for each dimension (flight notes similarity score, necessary content coverage score, dialogue interactivity score), a list of incomplete nodes (e.g., the meeting lacks a "weather analysis session"), and a list of omitted risk points (e.g., the meeting includes a "weather analysis session" but only analyzes the weather at the departure airport, omitting the weather at the arrival airport). This scoring report can be displayed directly or in response to user input selections on a user interface.

[0071] refer to Figure 3 This illustrates a flowchart of a non-limiting example process 200 of a pre-flight preparation meeting scoring method based on a large model according to some embodiments of the present disclosure. Figure 3 As shown, the non-limiting example over 200 includes steps S202 to S228.

[0072] Specifically, in step S204, meeting minutes documents in PPT, PDF, or Word format can be retrieved from the aviation private cloud server; in step S206, associated flight notes information can be retrieved from the aviation private cloud server; in step S208, audio files for the terminal devices used to record the pre-flight preparation meeting can be retrieved from the aviation private cloud server; then in step S210, audio preprocessing including noise reduction and transcription steps is performed on the audio files; and then in step S212, text preprocessing including cleaning steps is performed on the transcribed text.

[0073] In step S214, the topic similarity and aviation terminology coverage of the pre-flight preparation meeting are determined by the scoring model based on the cleaned text and flight notes information. Then, in step S216, the similarity score of the flight notes information is obtained by the scoring model based on the determined topic similarity and aviation terminology coverage.

[0074] In step S218, the necessary content covered and the necessary content not covered in the pre-flight preparation meeting are determined by the scoring model based on the cleaned text and the set of necessary content items for flight operation. Then, in step S220, the necessary content coverage similarity score is obtained by the scoring model based on the determined necessary content covered and the necessary content not covered.

[0075] In step S222, the scoring model determines the dialogue rounds, question and answer situation, confirmation and feedback situation, naturalness and information clarity of the pre-flight preparation meeting based on the cleaned text. Then, in step S224, the scoring model scores the dialogue interactivity based on the determined dialogue rounds, question and answer situation, confirmation and feedback situation, naturalness and information clarity to obtain a dialogue interactivity score.

[0076] In step S226, a comprehensive score is calculated by the scoring model based on the similarity score of the obtained flight remarks information, the necessary content coverage score, and the dialogue interactivity score. In step S228, the scoring result is displayed on the user interface. The scoring result may include the comprehensive score and other information that needs to be displayed (e.g., the scoring reasons in the aforementioned example).

[0077] This disclosure also provides an electronic device in another aspect. (See reference) Figure 4 This illustrates a schematic block diagram of an electronic device 300 according to some embodiments of the present disclosure. Figure 4 As shown, electronic device 300 includes processor 302 and memory 304 storing computer-executable instructions that, when executed by processor 302, cause processor 302 to perform the method 100 according to any of the foregoing embodiments. Processor 302 may be, for example, a central processing unit (CPU) of electronic device 300. Processor 302 may be any type of general-purpose processor, or it may be a processor specifically designed for pre-flight preparation assessment based on large models, such as an application-specific integrated circuit (“ASIC”). Memory 304 may be coupled to processor 302 and may include various computer-readable media accessible by processor 302. In various embodiments, memory 304 described herein may include volatile and non-volatile media, removable and non-removable media. For example, memory 304 may include any combination of: random access memory (“RAM”), dynamic RAM (“DRAM”), static RAM (“SRAM”), read-only memory (“ROM”), flash memory, cache memory, and / or any other type of non-transient computer-readable media. The memory 304 may store instructions that, when executed by the processor 302, cause the processor 302 to execute the method 100 according to any of the foregoing embodiments of the present disclosure.

[0078] The electronic device 300 is configured to perform the method 100 described in any of the foregoing embodiments, and therefore reference can be made to the description of the various embodiments of method 100 above, which will not be repeated here.

[0079] This disclosure also provides a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the pre-flight preparation scoring method based on a large model according to any of the foregoing embodiments of this disclosure.

[0080] This disclosure also provides a computer program product that may include instructions that, when executed by a processor, can implement the large-model-based pre-flight preparation meeting scoring method according to any of the foregoing embodiments of this disclosure. The instructions may be any set of instructions that will be executed directly by one or more processors, such as machine code, or any set of instructions that will be executed indirectly, such as a script. The instructions may be stored in an object code format for direct processing by one or more processors, or stored in any other computer language, including scripts or sets of independent source code modules that are interpreted on demand or compiled in advance.

[0081] Figure 5A schematic block diagram of a computer system 400 on which embodiments of the present disclosure may be implemented is shown. The computer system 400 includes a bus 402 or other communication mechanism for transmitting information, and a processing means 404 coupled to the bus 402 for processing information. The computer system 400 also includes a memory 406 coupled to the bus 402 for storing instructions to be executed by the processing means 404; the memory 406 may be random access memory (RAM) or other dynamic storage device. The memory 406 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processing means 404. The computer system 400 also includes a read-only memory (ROM) 408 or other static storage device coupled to the bus 402 for storing static information and instructions for the processing means 404. A storage device 410, such as a magnetic disk or optical disk, is provided and coupled to the bus 402 for storing information and instructions. Computer system 400 may be coupled via bus 402 to output device 412 for providing output to a user, such as, but not limited to, a display (such as a cathode ray tube (CRT) or liquid crystal display (LCD)), speakers, etc. Input device 414, such as a keyboard, mouse, microphone, etc., is coupled to bus 402 for transmitting information and command selections to processing device 404. Computer system 400 may perform embodiments of this disclosure. Consistent with certain implementations of this disclosure, results are provided by computer system 400 in response to processing device 404 executing one or more sequences of one or more instructions contained in memory 406. Such instructions may be read into memory 406 from another computer-readable medium, such as storage device 410. Execution of the sequence of instructions contained in memory 406 causes processing device 404 to perform the methods described herein. Alternatively, the teachings may be implemented using hardwired circuitry in place of or in combination with software instructions. Therefore, implementations of this disclosure are not limited to any particular combination of hardware circuitry and software. In various embodiments, computer system 400 may be connected across a network to one or more other computer systems, such as computer system 400, via network interface 416 to form a networked system. This network may include a private network or a public network such as the Internet. In a networked system, one or more computer systems may store data and supply data to other computer systems. As used herein, the term "computer-readable medium" refers to any medium that participates in providing instructions to processing device 404 for execution. Such media may take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs or magnetic disks such as storage device 410. Volatile media include dynamic memory such as memory 406. Transmission media include coaxial cables, copper wires, and optical fibers, including wiring that includes bus 402.Common forms of computer-readable media or computer program products include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic media, CD-ROMs, digital video discs (DVDs), Blu-ray discs, any other optical media, thumb drives, memory cards, RAM, PROMs and EPROMs, fast EPROMs, any other memory chips or cartridges, or any other tangible media from which a computer can read. Various forms of computer-readable media may be involved when carrying one or more sequences of one or more instructions to processing device 404 for execution. For example, instructions may initially be carried on a disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions over a telephone line using a modem. A modem local to computer system 400 may receive data over a telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector coupled to bus 402 may receive the data carried in the infrared signal and place the data on bus 402. Bus 402 carries the data to memory 406, from which processing device 404 retrieves and executes the instructions. Optionally, the instructions received by the memory 406 may be stored on the storage device 410 before or after execution by the processing device 404.

[0082] According to various embodiments, instructions configured to be executed by a processing device to perform a method are stored on a computer-readable medium. The computer-readable medium may be a device for storing digital information. For example, a computer-readable medium includes a compact disc read-only memory (CD-ROM) as known in the art for storing software. The computer-readable medium is accessed by a processor adapted to execute the instructions configured to be executed.

[0083] This disclosure also provides a pre-flight preparation meeting scoring system based on a large model. (Reference) Figure 6 This illustrates a schematic structural diagram of a large-model-based pre-flight preparation meeting scoring system 500 (which may be simply referred to as "System 500") according to some embodiments of the present disclosure. Figure 6 As shown, the system 500 may include an input module 502 and a scoring module 504. The scoring module 504 includes a scoring model 5042, which is built based on a pre-trained large model and fine-tuned in the aviation domain through a pre-flight preparation meeting labeled corpus. The scoring model is provided with a set of necessary content items for flight operation.

[0084] Input module 502 is configured to retrieve meeting minutes of the pre-flight preparation meeting and flight notes associated with the pre-flight preparation meeting.

[0085] Scoring module 504 is coupled to input module 502 to receive meeting minutes and flight notes. Scoring module 504 is configured to: determine a similarity score for flight notes information in the pre-flight preparation meeting based on the meeting minutes and flight notes information via scoring model 5042; determine a necessary content coverage score for the pre-flight preparation meeting based on the meeting minutes and the set of necessary content items for flight operation via scoring model 5042; determine a dialogue interactivity score based on the meeting minutes via scoring model 5042; and determine a comprehensive score for the pre-flight preparation meeting based on the similarity score, necessary content coverage score, and dialogue interactivity score determined by scoring model 5042.

[0086] In some embodiments, the meeting record information includes an audio file from the pre-flight preparation meeting. The input module 502 includes a data acquisition module and a data preprocessing module. The data acquisition module is configured to acquire the audio file, and the data preprocessing module is configured to: transcribe the audio file into timestamped text via a speech preprocessing model; and unify different words in the text that refer to the same aviation term into the same aviation term via a text preprocessing model, and split the text according to meeting nodes. The speech preprocessing model and the text preprocessing model are models optimized with aviation domain corpus.

[0087] In some embodiments, system 500 may further include a user interaction module, which includes a user interface and is configured to display a flight remarks information similarity score, a necessary content coverage score, a dialogue interactivity score, and a comprehensive score on the user interface. In some embodiments, the user interaction module is configured to display, on the user interface, a question labeled by a scoring model for at least one of the flight remarks information similarity score, necessary content coverage score, and dialogue interactivity score, as lower than a corresponding score threshold, along with suggestions for improvement regarding the question.

[0088] In some embodiments, the user interaction module is configured to: in response to receiving user input related to a question on the user interaction interface, present meeting record information related to the question via the user interaction interface.

[0089] In some embodiments, the user interaction module is configured to display, in response to a non-perfect score in the essential content coverage score, essential flight operation items from the set of essential flight operation items determined by the scoring model that were not mentioned in the pre-flight briefing on the user interface.

[0090] In some embodiments, the scoring module 504 is configured to: extract the meeting content theme of the pre-flight preparation meeting from the meeting record information and extract the note content theme of the flight note information from the flight note information via the scoring model 5042; determine the theme similarity between the meeting content theme and the note content theme; and determine the theme similarity score of the pre-flight preparation meeting based on the theme similarity; extract aviation terms from the meeting record information and the flight note information respectively via the scoring model; determine the ratio of the number of aviation terms that exist simultaneously in the meeting record information and the flight note information to the total number of aviation terms in the flight note information; and determine the terminology coverage score of the pre-flight preparation meeting based on the ratio; and determine the flight note information similarity score of the pre-flight preparation meeting based on the theme similarity score and the terminology coverage score.

[0091] In some embodiments, the scoring module 504 is configured to analyze meeting record information via the scoring model 5042 to score the interactivity of the conversation on one or more dimensions, including dialogue rounds, question-and-answer situations, confirmation and feedback situations, naturalness, and clarity of information.

[0092] System 500 can be used to perform various embodiments of the aforementioned method 100. Therefore, embodiments of system 500 can be referenced to various embodiments of the aforementioned method 100, and will not be repeated here.

[0093] The foregoing has described one or more exemplary embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this disclosure does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0095] While one or more embodiments of this disclosure provide the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or terminal product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment).

[0096] The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first" or "second" to denote names does not indicate any particular order.

[0097] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0098] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0101] Those skilled in the art will understand that one or more embodiments of this disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] One or more embodiments of this disclosure can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0103] The same or similar parts between the various embodiments of this disclosure can be referred to mutually, and each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to the description of the method embodiments. In the description of this disclosure, the descriptions of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., mean that the specific feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of this disclosure. In this disclosure, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this disclosure and the features of the different embodiments or examples.

[0104] Additionally, when used in this disclosure, the terms “here,” “above,” “below,” “below,” “in the following,” “overall,” and similar terms should refer to the entirety of this disclosure and not any particular part thereof. Furthermore, unless expressly stated otherwise or otherwise understood in the context in which they are used, conditional language used herein, such as “may,” “possibly,” “for example,” “like,” etc., is generally intended to express that certain embodiments include, while other embodiments do not, certain features, elements, and / or states. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or states in any way, or whether such features, elements, and / or states are included or performed in any particular embodiment.

[0105] The above description is merely an embodiment of one or more embodiments of this disclosure and is not intended to limit the scope of the one or more embodiments of this disclosure. Various modifications and variations can be made to the one or more embodiments of this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims.

Claims

1. A pre-flight preparation meeting scoring method based on a large model, comprising: Obtain the meeting minutes of the pre-flight preparation meeting and the flight notes associated with the pre-flight preparation meeting; The meeting minutes and flight notes are input into the scoring model, which is built based on a pre-trained large model and fine-tuned in the aviation field using a pre-flight preparation meeting annotated corpus. The scoring model is provided with a set of necessary content items for flight operation. The similarity score of the flight remarks information of the pre-flight preparation meeting is determined by the scoring model based on the meeting minutes information and the flight remarks information. The necessary content coverage score of the pre-flight preparation meeting is determined by the scoring model based on the meeting minutes information and the set of necessary content items for flight operation; The scoring model determines the dialogue interactivity score based on the meeting record information. as well as The overall score of the pre-flight preparation meeting is determined based on the similarity score of flight notes information, the score of necessary content coverage, and the score of dialogue interactivity determined by the scoring model.

2. The method according to claim 1, wherein, The pre-flight preparation meeting annotated corpus includes meeting minutes from multiple real pre-flight preparation meetings and annotations including one or more of the following: aircraft status, route selection, weather conditions, notices of significant impacts, performance limitations, and fuel quantity calculations.

3. The method according to claim 1, wherein, The large model is a large model that adopts the Transformer architecture.

4. The method according to claim 1, wherein, The similarity score of the flight remarks information for the pre-flight preparation meeting, determined by the scoring model based on the meeting minutes information and the flight remarks information, includes: The scoring model is used to extract the meeting content theme of the pre-flight preparation meeting from the meeting record information and the note content theme of the flight note information. The theme similarity between the meeting content theme and the note content theme is determined, and the theme similarity score of the pre-flight preparation meeting is determined based on the theme similarity. Aviation terms are extracted from the meeting minutes and flight notes information using the scoring model. The ratio of the number of aviation terms present in both the meeting minutes and flight notes to the total number of aviation terms in the flight notes information is determined. Based on this ratio, a terminology coverage score for the pre-flight preparation meeting is determined. The similarity score of the flight notes information in the pre-flight preparation meeting is determined based on the topic similarity score and the terminology coverage score.

5. The method according to claim 4, wherein, The terminology coverage score has a higher weight than the topic similarity score.

6. The method according to claim 1, wherein, Each essential content item in the set of essential content items for flight operation is assigned a corresponding score, and the score for the necessary content coverage of the pre-flight preparation meeting is determined by the scoring model based on the meeting minutes information and the set of essential content items for flight operation, including: The scoring model is used to identify essential flight operation items mentioned in the pre-flight preparation meeting from the meeting minutes information. The necessary content coverage score of the pre-flight preparation meeting is determined based on the scores corresponding to the necessary flight operation items mentioned in the pre-flight preparation meeting.

7. The method according to claim 6, wherein, The set of essential flight operation items includes at least one of core reminders and other reminders. The core reminders include at least one of aircraft status, route selection, weather conditions, notices of significant impacts, performance limitations, and fuel calculation. The other reminders include at least one of earliest departure time, latest departure time, landing time restrictions, specific risk warnings, and temporary notices.

8. The method according to claim 7, wherein, Core alerts are given a higher score than other alerts.

9. The method according to claim 1, wherein, The scoring model determines the dialogue interactivity score based on the meeting record information, including: The meeting transcript information is analyzed using the scoring model to score the interactivity of the conversation on one or more dimensions, including dialogue rounds, question and answer, confirmation and feedback, naturalness, and clarity of information.

10. The method according to claim 1, wherein, In the overall score, the flight remarks information similarity score has a first weight, the necessary content coverage score has a second weight, and the dialogue interactivity score has a third weight. Wherein, the second weight is greater than the first weight, and the first weight is greater than the third weight.

11. The method according to claim 1, comprising: The similarity score of the flight remarks information, the coverage score of the necessary content, the interactiveness score of the dialogue, and the comprehensive score are displayed on the user interface.

12. The method of claim 11, comprising: In response to at least one of the flight remarks information similarity score, the necessary content coverage score, and the dialogue interactivity score being lower than the corresponding score threshold, the scoring model identifies the problem for the at least one score and generates improvement suggestions, and displays the problem and improvement suggestions for the problem on the user interface.

13. The method of claim 12, comprising: In response to receiving user input related to the question on the user interface, meeting minutes related to the question are presented via the user interface.

14. The method according to claim 11, wherein, In response to the fact that the essential content coverage score is not full, the essential content items for flight operation that were not mentioned in the pre-flight preparation meeting are displayed on the user interface from the set of essential content items for flight operation determined by the scoring model.

15. The method according to claim 1, wherein, The meeting minutes information includes text obtained from audio files of the pre-flight preparation meeting.

16. The method of claim 15, comprising: The speech file is transcribed into timestamped text using a speech preprocessing model; as well as The text preprocessing model unifies different words in the text that refer to the same aviation term into the same aviation term, and then splits the text according to meeting nodes. The speech preprocessing model and the text preprocessing model are respectively optimized based on aviation corpora.

17. An electronic device comprising: processor; A memory storing computer-executable instructions, which, when executed by the processor, cause the processor to perform the pre-flight preparation meeting scoring method based on a large model according to any one of claims 1 to 16.

18. A computer-readable storage medium having computer-executable instructions thereon, which, when executed by a processor, cause the processor to perform the pre-flight preparation meeting scoring method based on a large model according to any one of claims 1 to 16.

19. A computer program product comprising instructions that, when executed by a processor, implement the pre-flight preparation meeting scoring method based on a large model according to any one of claims 1 to 16.

20. A pre-flight preparation meeting scoring system based on a large model, comprising: The input module is configured to obtain meeting minutes of the pre-flight preparation meeting and flight remarks associated with the pre-flight preparation meeting; as well as The scoring module, coupled to the input module, receives the meeting minutes information and the flight notes information. The scoring module includes a scoring model, which is built based on a pre-trained large model and fine-tuned using an aviation-related corpus of pre-flight preparation meeting annotations. The scoring model is provided with a set of essential flight operation content items. The scoring module is configured as follows: The scoring model determines the similarity score of the flight remarks information for the pre-flight preparation meeting based on the meeting minutes information and the flight remarks information. The necessary content coverage score of the pre-flight preparation meeting is determined by the scoring model based on the meeting minutes information and the set of necessary content items for flight operation. The scoring model determines the dialogue interactivity score based on the meeting record information, and The overall score of the pre-flight preparation meeting is determined based on the similarity score of flight notes information, the score of necessary content coverage, and the score of dialogue interactivity determined by the scoring model.

21. The system according to claim 20, wherein, The meeting minutes information includes text obtained from the audio file of the pre-flight preparation meeting. The input module includes a data acquisition module and a data preprocessing module. The data acquisition module is configured to acquire the audio file, and the data preprocessing module is configured to: The speech file is transcribed into timestamped text using a speech preprocessing model; as well as The text preprocessing model unifies different words in the text that refer to the same aviation term into the same aviation term, and then splits the text according to meeting nodes. The speech preprocessing model and the text preprocessing model are respectively optimized based on aviation corpora.

22. The system of claim 20, comprising: The user interaction module includes a user interface and is configured to display the flight remarks information similarity score, the necessary content coverage score, the dialogue interactivity score, and the comprehensive score on the user interface.

23. The system according to claim 22, wherein, The user interaction module is configured as follows: If at least one of the flight remarks information similarity score, the necessary content coverage score, and the dialogue interactivity score is lower than the corresponding score threshold, the user interface displays a question marked by the scoring model for the at least one score and improvement suggestions generated for the question.

24. The system according to claim 23, wherein, The user interaction module is configured as follows: In response to receiving user input related to the question on the user interface, meeting minutes related to the question are presented via the user interface.

25. The system according to claim 22, wherein, The user interaction module is configured as follows: In response to the fact that the essential content coverage score is not full, the essential content items for flight operation that were not mentioned in the pre-flight preparation meeting are displayed on the user interface from the set of essential content items for flight operation determined by the scoring model.

26. The system according to claim 20, wherein, The pre-flight preparation meeting annotated corpus includes meeting minutes from multiple real pre-flight preparation meetings and annotations including one or more of the following: aircraft status, route selection, weather conditions, notices of significant impacts, performance limitations, and fuel quantity calculations.

27. The system according to claim 20, wherein, The large model is a large model that adopts the Transformer architecture.

28. The system according to claim 20, wherein, The scoring module is configured as follows: The scoring model is used to extract the meeting content theme of the pre-flight preparation meeting from the meeting record information and the note content theme of the flight note information. The theme similarity between the meeting content theme and the note content theme is determined, and the theme similarity score of the pre-flight preparation meeting is determined based on the theme similarity. Aviation terms are extracted from the meeting minutes and flight notes information using the scoring model. The ratio of the number of aviation terms present in both the meeting minutes and flight notes to the total number of aviation terms in the flight notes information is determined. Based on this ratio, a terminology coverage score for the pre-flight preparation meeting is determined. The similarity score of the flight notes information in the pre-flight preparation meeting is determined based on the topic similarity score and the terminology coverage score.

29. The system according to claim 28, wherein, The terminology coverage score has a higher weight than the topic similarity score.

30. The system according to claim 20, wherein, Each essential flight operation item in the set of essential flight operation items is assigned a corresponding score, and the scoring module is configured as follows: The scoring model is used to identify essential flight operation items mentioned in the pre-flight preparation meeting from the meeting minutes information. The necessary content coverage score of the pre-flight preparation meeting is determined based on the scores corresponding to the necessary flight operation items mentioned in the pre-flight preparation meeting.

31. The system according to claim 30, wherein, The set of essential flight operation items includes at least one of core reminders and other reminders. The core reminders include at least one of aircraft status, route selection, weather conditions, notices of significant impacts, performance limitations, and fuel calculation. The other reminders include at least one of earliest departure time, latest departure time, landing time restrictions, specific risk warnings, and temporary notices.

32. The system according to claim 31, wherein, Core alerts are given a higher score than other alerts.

33. The system according to claim 20, wherein, The scoring module is configured as follows: The meeting transcript information is analyzed using the scoring model to score the interactivity of the conversation on one or more dimensions, including dialogue rounds, question and answer, confirmation and feedback, naturalness, and clarity of information.

34. The system according to claim 20, wherein, In the overall score, the flight remarks information similarity score has a first weight, the necessary content coverage score has a second weight, and the dialogue interactivity score has a third weight. Wherein, the second weight is greater than the first weight, and the first weight is greater than the third weight.

Images