Converting natural language requirements of aerospace systems into formalization using large language models

Abstract

A system for converting natural language (NL) requirements of an aerospace system to formalize to control a landing gear system of an aircraft using a large language model (LLM) includes a processor and a memory including instructions that, when executed by the processor, convert the NL requirements into formalities to control the landing gear system of the aircraft. The system is caused to use at least the LLM to revise the received NL expressions associated with the requirements of the aerospace system. The LLM is further configured to replace at least one word in the revised NL expression with a lifting atomic entity to generate an underlying sentence structure and convert the underlying sentence structure into a lifting logic. The LLM is also used for basic improvement of the logical expression and conversion of the NL expression into formalized requirements of an aerospace system. The system also controls the aerospace system based on the formal demand.

Description

[0001] Government-supported statement

[0002] The subject matter of this disclosure was carried out with the support of the United States government under contract number FA8750-20-C-0203 awarded by the Defense Advanced Research Projects Agency (DARPA). The United States government retains certain rights to the subject matter of this disclosure. Technical Field

[0003] This disclosure generally relates to the transformation of natural language (NL) requirements into formal requirements, and more specifically, to systems and methods for automatically transforming NL requirements of aerospace systems into formal requirements by using large language models (LLMs) in a controlled manner to mitigate the hallucination of LLM and ensure the accuracy of formal requirements. Background Technology

[0004] Formalization employs mathematical models, precise notation, and logical reasoning. It is useful across a wide range of disciplines and applications where precise and rigorous representations of systems and concepts are crucial. However, creating formalizations often requires a deep understanding of formal methodology, making it inaccessible to those without specialized training. The complexity of formalization also makes its development time-consuming and costly, thus hindering practitioners from applying formalization in their respective fields.

[0005] In the aerospace industry, safety and reliability are paramount. Clear and detailed requirements documentation allows for the early detection of potential risks and problems in the development process, reducing the likelihood of encountering issues in deployed systems where problem-solving can be more challenging and costly. However, engineers may prefer writing NL requirements, which often results in lengthy and vague documentation requiring careful reading to identify errors. This manual review can be both expensive and time-consuming. Formalized requirements can alleviate these problems by enabling automated requirements analysis. To make formalization more accessible to non-specificational requirements engineers, a process is needed to transform unstructured NL requirements into formalized ones. Attached Figure Description

[0006] These and other features, aspects, and advantages of this disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings, wherein similar characters throughout the drawings denote similar parts, wherein:

[0007] Figure 1 This is an example block diagram of a system according to aspects of this disclosure;

[0008] Figure 2 The natural language requirement conversion process according to aspects of this disclosure is illustrated;

[0009] Figure 3 Use based on aspects of this disclosure Figure 2An illustrative example of the natural language processing requirements conversion process;

[0010] Figure 4 Use based on aspects of this disclosure Figure 2 Another illustrative example of the natural language processing requirements; and

[0011] Figure 5 It is an illustrative depiction of a system of block diagrams constructed according to aspects of this disclosure to support the process disclosed herein. Detailed Implementation

[0012] Although this disclosure will be described in detail in particular, those skilled in the art will readily understand that various modifications, rearrangements and substitutions may be made without departing from the spirit of this disclosure.

[0013] To facilitate an understanding of the principles of this disclosure, exemplary aspects shown in the accompanying drawings will now be referenced and described using specific language. However, it should be understood that this is not intended to limit the scope of this disclosure. Any changes and further modifications to the features shown herein that would occur to those skilled in the art, as well as any additional applications of the principles of this disclosure shown herein, should be considered within the scope of this disclosure.

[0014] Formalization, as a practice emphasizing the use of rigorous mathematical models and precise notation, is useful across a wide range of disciplines and applications. In software and systems engineering, formalization and its accompanying formal analysis techniques can help detect errors early, identify defects and inconsistencies throughout the development lifecycle, and thus significantly reduce costs. Formalization enables rigorous verification and validation of systems, ensuring they meet expected requirements and function as intended. However, formal analysis requires a formal representation of the analyzed system, which can be a barrier for experts who do not employ formal methods. To make formalization more accessible, this disclosure provides a Large Language Model (LLM)-based approach for interactively transforming unstructured natural language into formalized representations. The LLM-based approach of this disclosure employs a usable, purely natural language-based interface that provides a high guarantee of transformation correctness while remaining accessible to non-experts in formal methods. More specifically, this disclosure provides a process that breaks down the transformation task into several boxes, including preprocessing for verbosity and ambiguity, entity identification, syntactic abstraction, grounding, transformation into an abstract syntax tree (AST), and reverse transformation. As described below, the effectiveness of this approach can be demonstrated through experimental evaluation of common use cases (requirements engineering) in the aerospace industry. Requirements engineering is an early stage of system development where system requirements are extracted, captured, and their deficiencies analyzed.

[0015] NL can be difficult to formalize because it is often ambiguous and there are often several ways to express the same idea. In addition, LLM presents many challenges: (i) they are prone to generating the “illusion” of incorrect information; (ii) they require a lot of computational resources; (iii) they often have licensing or data privacy issues related to corporate users (if using externally hosted LLM); and (iv) they require a lot of training data to fine-tune for specific tasks.

[0016] The process disclosed herein divides the transformation task into several boxes. The various aspects of this process include syntactic abstraction, transformation, and foundationalization, where LLM performs transformation boxes on a raised version of the sentence, preserving the underlying sentence structure while abstracting away unnecessary details. Raising is a technique used to abstract concrete entities from a sentence by replacing them with symbols. This abstraction facilitates the transformation of these sentences in subsequent processes. To complement the above aspects of the process disclosed herein, two preprocessing boxes are included: (i) identifying redundancy and ambiguity, and (ii) sentence structure identification. Box (i) helps to state sentences as clearly and concisely as possible without altering their intended semantics, while box (ii) prepares sentences for syntactic abstraction. Furthermore, two postprocessing boxes are included: (iii) transformation to an AST and a rule-based NL representation; and (iv) LLM reverse transformation. Box (iii) will detect whether the LLM output is malformed and transform the AST into a rule-based NL representation for engineers to review the correctness of its transformation, while box (iv) involves transforming the formalization back into natural language. Combining boxes (iii) and (iv) mitigates the potential illusions generated by LLM and ensures the correctness of the LLM output. All transformation boxes can be implemented using contextual learning and / or fine-tuning of the LLM, where contextual learning may not require a large training dataset as much as fine-tuning of the LLM. In general, the approach disclosed in this paper avoids excessive human effort (as in reinforcement learning with human feedback or manually generated datasets) and provides a method for cross-domain generalization.

[0017] Requirements engineering is an early stage in system development where system requirements are derived, captured, and analyzed. A common problem is that defects in requirements (such as logical inconsistencies between requirements, missing requirements, redundancy, etc.) are not discovered until later in system development, and retrospective repair of these design flaws is both expensive and time-consuming. One rigorous approach to requirements analysis is to formally state them and use formal methodological tools for requirements analysis, but as discussed earlier, this is difficult. The aerospace industry relies on capturing correct and complete requirements, which can also be formally analyzed. Precisely formalizing requirements into logical form using the requirements intent expressed in NL is a non-trivial and time-consuming process, relying not only on expertise in formal methodologies but also on requirements engineers specializing in aerospace applications.

[0018] The method disclosed herein can be used to interactively formalize a set of requirements as a prerequisite for comprehensive formal analysis. Next, some relevant concepts needed to understand the transformation process of this disclosure are introduced: "Contextual learning" involves providing task descriptions, instructions, and examples as part of the LLM input to improve the LLM's ability to perform a specified task. Providing examples is also known as "few-shot learning." This is a specific method for rapid engineering. "Fine-tuning" involves additional training of the model using a task-specific dataset (input and output pairs). "Primation" involves providing the LLM with an initial textual description of relevant context and knowledge before querying a specific task. "Syntactic abstraction" involves replacing the details of a sentence with atomic entities, leaving only the underlying sentence structure. The resulting abstract sentence is called a "lifting sentence." After transforming the lifting sentence into a lifting formula, the formula is "foundationalized" by reintroducing the details of the initial abstraction and relating them to a vocabulary of core concepts. In the context of LLM, foundationalization refers to the process of anchoring the LLM's response to real-world knowledge (e.g., knowledge in the aerospace field) and ensuring that the response remains relevant to the context. This is achieved by combining LLM with external data sources, continuous learning mechanisms, and domain-specific insights (such as aerospace). The fundamentalization process ensures that any two references to the same core concept map to the same atomic entity. Generally, the formalizations considered in this disclosure are extensions and fragments of "first-order logic" and "temporal logic." First-order logic (FOL) refers to the logic where the predicate of a sentence or statement can only refer to a single subject (e.g., "All birds fly."). Temporal logic (TL) is a modal logic where the modality refers to time. In TL, expressions about the temporal future of a path can be encoded, such as a condition that will eventually be true until another fact becomes true, and so on.

[0019] Previous approaches to transforming natural language into formal logic will now be discussed, and where this disclosure stands relative to these previous approaches. One previous approach illustrates the core idea of ​​using formal methods to assist LLMs in performing complex reasoning tasks, such as question-and-answer in the restaurant ordering domain. However, this approach is based on Answer Set Programming (ASP), a fundamentally different formalization that has been used to examine the output of LLMs for high-level natural language understanding tasks. ASP is a logic and rule-based formalization used to perform knowledge representation and reasoning tasks, modeling common-sense knowledge and performing reasoning. In contrast, the approach of this disclosure generates a formal specification of first-order temporal logic from LLMs, which is a superset of the logic used in ASP. Another previous approach introduces the idea of ​​generating training data algorithmically based on grammar, where the grammar is provided by the study of real-world signal temporal logic specification data. However, this approach requires data analysis against the target formalization (which, of course, also assumes that such data exists in the first place). The model may also perform poorly on test data other than the grammar. Yet another previous approach employs a combination of boosting and fine-tuning using the generated dataset. This dataset was generated by prompting LLM to obtain NL-TL (temporal logic) pairs, which were then corrected using reinforcement learning with human feedback. This disclosure avoids this manual effort. A previous method also used a process from (i) a boilerplate pattern to (ii) an LLM transformation to (iii) post-processing. Boilerplate input constraints allow simple transformations to be fairly accurate. The method of this disclosure does not impose such constraints on the input, eliminating the need for the user to manually provide such input.

[0020] Another existing approach proposes two main methods. In the first main method, a fine-tuning dataset is generated using GPT-4, where the validity of the logical expression (relative to NL text) is heuristically validated. This data is used to fine-tune the LLM for direct transformation. In the second main method, a (relatively small) LLM corrects the output of the original GPT transformation, which was trained using reinforcement learning with human feedback. This disclosure avoids this manual work. In another existing approach, interactive transformation is emphasized; however, interactivity is achieved by transforming sub-expressions that are more readable than the whole expression, which differs from the approach of this disclosure, which provides a purely natural language interface with the user.

[0021] Figure 1 This is an illustrative block diagram of system 100 according to aspects of this disclosure. However, it should be understood that systems employing the concepts of this disclosure for formalizing requirements are not limited to... Figure 1The specific system 100 shown. System 100 includes one or more client devices 105 running one or more applications 110. Applications 110 may include different software applications that support the organization's operations and processes. In some embodiments, one of the applications 110 may at least partially include a framework or process (e.g., Figure 2 The framework or process shown (200) helps users interactively transform unstructured natural language requirements into formalization. In some respects, unstructured natural language can be expressed in English. However, this disclosure is not limited to providing English-based natural language transformation. In some embodiments, application 110 may be configured to facilitate, support, and execute procedures to transform unstructured natural language requirements into formalization within a cloud computing platform 120.

[0022] The cloud computing platform 120 may include an interface gateway 135 to facilitate, enable, and support communication between the client device 105 and the conversion server 125. The client device 105 may execute a browser, which is used by a user to interact with the cloud computing platform 120 via the network 115.

[0023] The transformation server 125 may be associated with storage device 130, which may be, for example, but is not limited to, a storage area network. In some aspects of this disclosure, the cloud computing platform 120 may include more than one server instance (e.g., the transformation server 125) and more than one data storage device 130. Storage device 130 may include one or more query response data sources that are known or have become known, including but not limited to relational database management systems. Data stored in storage device 130 may be stored as part of a relational database, multidimensional database, Extensible Markup Language (XML) document, or any other data storage system that stores structured and / or unstructured data. Data may be distributed across several relational databases, multidimensional databases, and / or other data sources. In some embodiments, storage device 130 may be implemented as an "in-memory" database, wherein volatile (e.g., non-disk-based) storage devices (e.g., random access memory) are used both as cache storage and for storing the complete database during operation, and persistent storage devices (e.g., one or more fixed disks) are used for offline persistence and maintenance of database snapshots. Embodiments are not limited to any number or type of data sources. The transformation server 125 may include computing devices that use a computer processor to process requests.

[0024] System 100 also includes a backend system 140, which, in response to requests or calls from the cloud computing platform 120, can automatically generate executable code or instructions to perform the process of transforming unstructured natural language (NL) requirements and / or supporting the transformation of NL requirements. In some aspects of this document, users can provide instructions or requests for the transformation of one or more NL requirements, such as those implemented in application 110 and / or cloud computing platform 120 via network 115, which can coordinate with the processing of backend system 140 to generate responses to realize the transformation of NL requirement inputs.

[0025] In one example, client device 105 executes application 110 to generate a transformation of NL requirements input to the user via a user interface (UI) on the display of client device 105. The user can manipulate UI elements within the UI via a touchscreen, keyboard, or audio input to indicate and specify requirements to be automatically transformed into formalized forms in an unstructured NL format, where cloud computing platform 120 works with storage device 130 and possibly backend system 140, using, for example... Figure 2 The framework or process described in 200 generates transformations.

[0026] Figure 2 This is an illustrative description of a framework or process 200 for transforming unstructured NL requirements into a formalized framework using an LLM (e.g., ChatGPT, Llama, Mistral, Fine-tuned LLM, etc.). The process 200 disclosed herein comprises three general stages: preprocessing, core transformation, and post-processing. The preprocessing stage consists only of box 210. In box 210, the LLM accepts NL requirements (e.g., in the form of raw NL sentences) and interactively generates clear, concise NL requirements through dialogue with the user. This leverages the chatbot functionality of the LLM, as the user can ask questions about the LLM's output and provide user feedback. The LLM can be initiated, for example, by using the following prompts to maximize performance:

[0027] Problem: Suppose you are a formal methods expert advising an engineer on how to concisely and unambiguously state a system specification. Be sure to omit verbose and unnecessary details that are irrelevant to the system specification, and be sure to not alter the semantics of the specification. In the following sentence, remove any unnecessary verbosity and alert the engineer to any natural-language ambiguities: <sentence>.)

[0028] Answer: (Answer:)

[0029] The user then engages in a conversation with the chatbot until they are satisfied with a (potential) revised version of the NL input requirement. The interaction ensures that the semantics of the requirement remain unchanged.

[0030] The core transformations occur in boxes 220, 230, and 250. Box 220 performs lifting, which can also be considered atomic entity recognition. Here, LLM takes the processed NL sentence from box 215 as input and abstracts away the sentence details, leaving only the underlying sentence structure. Similar to box 210, the input is initiated, for example, with a precise description of the task:

[0031] Your task is to take English sentences as input and perform syntactic abstraction, producing "lifted" English sentences as output, where some phrases are abstracted away and replaced with atomic entities of the form [Pn abstracted details]. Start with "P1", then "P2", "P3", and so on. Aspects of the input sentence that should be maintained (not abstracted) include conditional statements, conjunctions, disjunctions, negations, equality, comparisons, temporal aspects, and units.

[0032] Then, one or more examples of context learning similar to the following are given:

[0033] Input: If you are born, then eventually you will die.

[0034] Output: If [P1\you are born], then eventually [P2\you will die].

[0035] In the example above, text in the form of [Pn\Extra details] means that the extra details are abstracted away and replaced with the atomic entity Pn.

[0036] Once the sentence is promoted to box 227, where unnecessary details are replaced with atomic entities, a transformation is performed in box 230 to generate the promoted logical expression in box 235. The promotion description could be, for example:

[0037] Your task is to translate lifted English sentences into linear temporal logic (LTL). The lifted English sentences have some details abstracted away in terms of atomic entities of the form [Pn], where "n" is some number. In LTL, only the operators & (and), | (or), ! (not), = (equals), > (greater than), < (less than), Globally, Eventually, Next, Yesterday, Since, and Until are used. The semantics of the < operator are described. The LTL translation should only include these operators and the abstract entities, without any additional English phrasing. Each entity in the input should appear somewhere in the output.<Description of the semantics of the operators.> The LTLtranslation should only include these operators and abstraction entities withno extra English verbiage.Each entity in the input should appear somewhere in the output.)

[0038] Continuing with the previous example, the context of LLM is as follows:

[0039] Input: If [P1], then eventually [P2].

[0040] Output: [P1] -> Eventually [P2].

[0041] For LLM, the transformation task is easier when operating on lifted sentences with atomic entities rather than on complete NL sentences with (potentially) complex vocabulary and verbose phrases.

[0042] The final aspect of the core transformation is fundamentalization, performed at box 250. In box 250, the elevated atomic entity of the form Pn is replaced with a concrete version representing a real concept, such as "born" and "die." The purpose of fundamentalization is to ensure that multiple references to the same core concept (within or across formulas) map to the same (fundamentalized) atomic entity. For example, consider a case where two NL sentences, "If you are born, then eventually you will die" and "If you died, then once you were born," are transformed into formalization. These two NL phrases, "you will die" (from the first sentence) and "you died" (from the second sentence), need to be referenced by the same entity, "die." To achieve this, phrase embeddings are retrieved from the LLM for (i) each value in the lift-to-unbasic entity map (LUEM) of the representation generated at box 225 (the lift-to-unbasic entity map associates each lifted atomic entity with the abstracted-away details), and (ii) each value input from box 240, where the list of basic entities represents the core concepts of the underlying system. Then, each value in the LUEM map generated at box 225 is associated with the most similar basic entity in box 240 by calculating the highest cosine similarity between the associated phrase embeddings. The list of basic entities in box 240 can be generated from the project glossary or provided manually by the user. Lifted atomic entities are abstract elements that have been symbolically replaced. Lifted atomic entities are considered indivisible and cannot be further decomposed into simpler components.

[0043] Post-processing occurs at boxes 260 and 270 and involves performing a correctness check on the basicized logical expression from box 250. First, at box 260, the string representation of the basicized logical expression is parsed into an AST representation. If parsing fails, the expression is syntactically invalid. Instead of immediate failure, an LLM self-correction is performed, instructing the LLM to examine its transformation and look for errors using hints such as the following:

[0044] The following displays a natural language sentence and its corresponding Linear Temporal Logic (LTL) transformation. However, the LTL transformation is syntactically incorrect. Please review the transformation and output the updated transformation to LTL.

[0045] sentence along with a corresponding linear temporal logic(LTL)

[0046] translation. However, the LTL translation is syntactically ill-formed.

[0047] Please review

[0048] the translation and output an updated translation to LTL.)

[0049] This provides an opportunity to fix failed transformations instead of starting entirely from scratch. Then, in Box 270, Procedure 200 performs a reverse transformation on the AST representation. This transforms the structured formal back into NL so that the user can determine whether the underlying formal generated by Procedure 200 matches the user's intent without the user manually handling the raw formal. Box 270 produces a reverse transformation of the logical form generated by the LLM, and Box 275 provides a natural and readable transformation. For example, the following instruction could be given:

[0050] Transform the following linear temporal logic expression, presented in structured notation, into natural language.

[0051] (Translate the following linear temporal logic formula,expressed

[0052] in a structured representation,to natural language.)

[0053] However, simply using LLM for inverse transformation cannot fully verify the underlying formalization, because there is no guarantee that the inverse transformation of LLM will not change the semantics (this is the problem for which this disclosure provides a solution). With this in mind, a rule-based transformation for pseudo-NL is also presented in box 260, which includes brackets indicating the scope in box 265, guaranteeing that the inverse transformation semantically matches the underlying form. For example, for the structured form:

[0054] ("hint", ("and", ("equals", "command line", "normal mode"))

[0055] ("equals", "command handle", "down")),("not", "indented sequence")),(('IMPLIES',('AND',('EQUALS','command line',

[0056] 'normal mode'),('EQUALS','command handle','down')),('NOT',

[0057] 'retraction sequence')),)

[0058] LLM reverse conversion gives:

[0059] If the command line is in normal mode and the command handle is in the down position, a retraction sequence should not occur.

[0060] command handle is down,then the retraction sequence should not

[0061] occur.)

[0062] The rule-based pseudo-NL transformation gives:

[0063] If the command line is in normal mode and the command handle is down, then the retraction sequence will not occur.

[0064] Now for reference Figure 3 This illustrates a concrete example of how process 200 transforms unstructured NL requirements into formal ones. For instance, in box 210, the LLM receives the following raw NL sentence from the user as an input requirement:

[0065] If a driver decides not to wear a seatbelt, an alert should eventually be issued to notify relevant parties and draw their attention to this particular behavior.

[0066] decides to leave the seatbelt unbuckled,it SHALL be the case that

[0067] an alert,designed for the purpose of informing relevant parties,is

[0068] eventually raised to draw attention to this particular action.)

[0069] The LLM processes the raw NL sentence input at box 210 to omit redundancy and ambiguity from the user-provided raw NL sentence, and outputs the following sentence at box 215:

[0070] If the driver's seatbelt is not fastened, an alarm should eventually be sounded.

[0071] seatbelt is unbuckled,then eventually an alert SHALL be raised.)

[0072] The chatbot can ask the user if the sentence output in box 215 matches the user's intent. If the output sentence does not match the user's intent, the user can return to the preprocessing stage in box 210 and input a revised NL input requirement processed by LLM to further eliminate redundancy and ambiguity. This process can be repeated until the user is satisfied with the revised version of the input NL requirement.

[0073] In box 220, promotion is performed, where the LLM takes the processed NL from box 215 as input, abstracts away the details of the sentences, leaving only the underlying sentence structure, and promotes it to the unbasicized entity map in box 225, as shown below:

[0074] A: "The driver's seatbelt is not fastened."

[0075] is unbuckled')

[0076] B: "An alert should be raised."

[0077] This, in turn, leaves the following NL-enhancing sentence at box 227:

[0078] If A, then eventually B.

[0079] Once promotion is performed, details unnecessary at the logical formalization level are replaced with atomic entities, transforming the promoted NL sentence into a promoted logical expression in box 240, such as "G(A->E(B))" in box 235, where G and E are the tense operators representing "globally" and "eventually". At box 250, priming is performed, replacing the promoted atomic entities with concrete versions representing real concepts to produce a primed logical expression, such as "G(driver_unbuckled->E(raise_alert))(G(driver_unbuckled->E(raise_alert)))".

[0080] At boxes 260 and 270, post-processing occurs, where a correctness check is performed on the basic formalized expression from box 250. First, at box 260, the string representation of the basic formalized expression is parsed into an AST representation. Next, at box 270, a reverse transformation is performed on the AST representation. This transforms the structured formalization back into NL so that the user can determine when the underlying formalization generated by process 200 matches the user's intent without the user having to manually process the raw formalization. Box 270 produces the reverse transformation of the LLM-generated expression, and box 275 gives the natural and readable transformation (e.g., "If the driver is unbuckled, then eventually an alert SHALL beraised"). However, since using LLM for reverse transformation cannot fully verify the underlying formalization, as mentioned above, the rule-based transformation for pseudo-NL at box 270 (pseudo-NL includes brackets indicating the scope at box 265 (e.g., (If driver_unbuckled, then raise_alert)))))) ensures that the reverse transformation semantically matches the underlying formalization.

[0081] Now for reference Figure 4 This illustrates another concrete example of how process 200 transforms unstructured aerospace NL requirements into formal requirements. In box 210, the LLM receives the following raw NL sentence from the user as input requirements:

[0082] The system should issue a request when the controller is shut down if all of the following conditions are true:

[0083] (The system shall initiate a request upon controller shutdown when

[0084] all of the following conditions are true:)

[0085] 1) Initialization is complete.

[0086] 2) Once the speed exceeds 500, the speed must be below 400.

[0087] The speed was above 500.

[0088] 3) No fault was detected.

[0089] The LLM processes the raw NL sentence input at box 210 to omit redundancy and ambiguity from the user-provided raw NL sentence, and outputs the following sentence at box 215:

[0090] After initialization, if the speed is greater than 500 but then falls below 400, no faults were detected.

[0091] And the system should initiate a request when the controller is shut down. (When initialization is...)

[0092] complete and speed is less than 400after speed was greater than 500

[0093] and there is no test fault and controller is shutdown,then the system

[0094] (shall initiate a request.)

[0095] The chatbot can ask the user if the sentence output in box 215 matches the user's intent. If the output sentence does not match the user's intent, the user can return to the preprocessing stage in box 210 and input a revised NL input requirement processed by LLM to further eliminate redundancy and ambiguity. This process can be repeated until the user is satisfied with the revised version of the input NL requirement.

[0096] In box 220, promotion is performed, where the LLM takes the processed NL from box 215 as input, abstracts away the details of the sentences, leaving only the underlying sentence structure, and promotes it to the unbasicized entity map in box 225, as shown below:

[0097] A: "Initialization"

[0098] B: "Complete"

[0099] C: "Speed"

[0100] D: "Test fault"

[0101] E: "controller"

[0102] F: "shutdown"

[0103] G: "request"

[0104] This, in turn, leaves the following NL-enhancing sentence at box 227:

[0105] If A is B and C is less than 400 after C is greater than 500 and not D and E is F, then G.

[0106] Once promotion is performed, details that are not needed at the logical formalization level are replaced by atomic entities, transforming the promoted NL sentence at box 240 into the promoted logical expression at box 235, where "Pre" refers to the previous state of the variable / expression:

[0107] (A = B and C < 400 and (Pre(C) > 500) and (Not D) and E = F) -> G((A = B and C < 400 and (Pre(C) > 500) and (Not D) and E = F) -> G)

[0108] At box 250, basicization is performed, where the promoted atomic entities are replaced with specific versions representing concrete concepts to produce the basicized logical expression:

[0109] ([initialization = complete]) and ([speed] < 400) and (Pre([speed]) > 500) and (NOT[test_fault]) and [controller = shutdown]) -> [request]

[0110] At boxes 260 and 270, post-processing occurs, where a correctness check is performed on the basic formalized expression from box 250. First, at box 260, the string representation of the basic formalized expression is parsed into an AST representation for syntax checking. Next, at box 270, a reverse transformation is performed on the AST representation. This transforms the structured formalization back into NL so that the user can determine when the underlying formalization generated by process 200 matches the user's intent without the user having to manually process the raw formalization. Box 270 produces the reverse transformation of the LLM-generated expression, giving a natural and readable transformation (e.g., "If initialization is complete and speed is less than 400 and once speed was greater than 500 and there is no test fault and there is a controller shutdown, then there is a request initiated by the system"). However, since using LLM for reverse transformation cannot fully verify the underlying formalization, as mentioned above, the rule-based transformation for pseudo-NL at box 270 (pseudo-NL includes brackets indicating ranges (e.g., "when ([initialization = complete] and ([speed])") is not fully validated) < 400) and (previously ([speed])>500) and (not [test fault]) and [controller = off]), then [request]""When ([initialization = complete]and ([speed]) < 400)and(Pre([speed])>500)and(NOT[test_fault])and[controller=shutdown]),then[request]”)) guarantees that the reverse transformation semantically matches the underlying expression. To demonstrate the ability of process 200 to mitigate the LLM illusion, consider the lifting logic output generated by box 240, as follows: (A=B and C<400 and Pre(C)>true and (notD) and E=F)->G((A=B and C<400 and Pre(C)>true and (notD) and E = F)->G), where "Pre" refers to the previous state of the variable / expression. This expression contains the grammatically incorrect expression "C>true", which can be revealed by constructing an AST. By definition, the greater than ">" operator requires two numeric operands, so the Boolean operand "true" is inappropriate in this context. To identify semantic problems in LLM-generated expressions, rule-based transformations can be examined. For example, if a semantically incorrect expression "C>400" is generated due to illusion instead of the correct form "C<400", this problem can be detected by manually checking rule-based transformations.

[0111] For testing Figure 2 The effectiveness of process 200 shown was demonstrated through a case study involving the formalization of open-source landing gear requirements for an aircraft. In this study, 19 requirements were formalized, involving a total of 28 basic entities. The target formalization is an LTL extended using equations and enumeration types.

[0112] An aircraft landing gear system typically consists of three parts: (1) a mechanical part, which contains all mechanical components, including the hydraulic system and three landing gear units; (2) a digital part, including control software; and (3) a pilot interface. Each action taken depends on the state and temporal behavior of all the physical components that make up the system. When considering such a system, the challenge is first to model and program the software part that controls the landing and retraction sequence, and second to consider the physical behavior of the hydraulic components to justify safety requirements. The landing system controls the operation of the landing gear and associated doors. The system consists of three landing gear units: forward, left, and right. Each landing gear unit contains a door, landing gear, and associated hydraulic cylinders. The system is digitally controlled, and the basic landing sequence generally includes: (1) opening the landing gear compartment door; (2) extending the landing gear; and (3) closing the door. Similarly, after takeoff, the corresponding basic retraction sequence includes: (1) opening the door; (2) retracting the landing gear; and (3) closing the door.

[0113] Regarding the pilot interface, the retraction and extension of the landing gear are controlled by the pilot via an up / down lever. For the mechanical and hydraulic components, hydraulic power is supplied to the hydraulic cylinders from an external hydraulic circuit via a set of electric valves, each activated by an electrical command from the digital section. A set of discrete sensors informs the digital section of the equipment's status. The digital section consists of two identical computing modules that execute the same control software in parallel. This control software controls the landing gear and doors, detects anomalies, and informs the pilot of the system's global status and any anomalies. The software generates commands for the allocation elements of the hydraulic system based on sensor values ​​and pilot commands. Both computing modules receive the same inputs (sensor values ​​and pilot commands). Based on these inputs, each module calculates five electrical commands (one for each electric valve). These corresponding electrical commands output from both modules are physically generated on the same wire. The implicit combination of the two outputs is an electrical "OR". Therefore, if two different computing modules send two different values ​​(true and false) on the same wire (e.g., in the event of a failure of one of the two computing modules), only the true value is transmitted to the corresponding electric valve.

[0114] Similarly, these two modules generate global Boolean state variables for the cockpit (one for each cockpit light). These outputs are synthesized by each module based on sensor data and situational awareness. Similar to the electrical commands provided to the electric valves, the Boolean state variables from the two modules are combined using a logical "OR" operation.

[0115] The purpose of the software portion of the landing gear system includes controlling the hydraulic system according to pilot instructions and the position of the mechanical components, and monitoring the system to notify the pilot in case of any abnormalities.

[0116] The term "normal mode" refers to any situation that does not involve a malfunction of the landing gear system. The term "failure mode" refers to any situation involving a combination of malfunctions with a probability greater than a predetermined threshold. When the command line is active (normal mode), the landing system responds to pilot commands by activating or deactivating the electric valves of the corresponding cylinders.

[0117] Regarding mechanical and hydraulic equipment, the most likely failure to consider is a permanent failure; for example, each electric valve may malfunction and eventually become blocked, whether it is in the closed or open state. Regarding computational modules, each of them may fail in two different ways: (1) a permanent failure, in which case the module fails to send commands to the electric valves or lights; and (2) the computational module malfunctions by sending random values ​​to the actuators.

[0118] The landing gear system must meet a set of safety requirements. These requirements are divided into two parts: normal mode requirements and failure mode requirements. The procedure 200 of this disclosure can be used to formalize NL expressions related to landing gear system requirements for both normal and failure modes. As an illustrative example, the LLM can receive the following raw NL sentences related to normal mode landing gear system requirements as input requirements:

[0119] When the command line is in normal mode, if the landing gear command handle is down, the landing gear will eventually lock and the hatch will close.

[0120] normal mode,if the landing gear command handle is DOWN,then

[0121] eventually the gears will be locked down and the doors will be

[0122] (closed.)

[0123] Process 200 is as described above regarding Figure 2-4 Each of the boxes shown in Figures 210-275 continues to process the original NL sentences above and outputs the following formalized landing gear system requirements as a response:

[0124] When the command line is active, if the landing gear command handle is downward, the landing gear will eventually lock and the hatch will close.

[0125] the landing gear command handle is DOWN, then eventually the

[0126] gears will be locked down and the doors will be closed.)

[0127] It has the corresponding logical expression:

[0128] G((Command line mode = Normal ∧ Command handle position = Down) F(Landing gear position = locked ∧ Door position = closed)). (G((command line mode = NORMAL ∧ command handle position = DOWN)) F(gears position=LOCKED DOWN∧door position=CLOSED)).)

[0129] As another illustrative example, LLM can accept the following raw NL sentences related to failure-mode landing gear system requirements as input requirements:

[0130] If, after the electric valve is opened, one of the three hatches remains locked in the closed position, the normal mode output is set to false.

[0131] locked in the closed position after stimulating the opening electro-valve, then the output normal mode is set to false.)

[0132] Process 200 is as described above regarding Figure 2-4 Each of the boxes shown in Figures 210-275 continues to process the original NL sentences above and outputs the following formalized landing gear system requirements as a response:

[0133] If more than 7 seconds have passed since the electric valve was opened and one of the three doors is still seen locked in the closed position, the red "Landing Gear System Malfunction" indicator light will illuminate.

[0134] of the three doors is still seen locked in the closed position more than

[0135] 7s after stimulating the opening electro-valve,then the red light

[0136] "landing gear system failure" is on.)

[0137] As another illustrative example, LLM can accept the following raw NL sentences related to the requirements of a normal mode landing gear system as input requirements:

[0138] When the command line is in normal mode, if the landing gear command handle is turned upwards, the landing gear will eventually lock and retract, and the hatch will close.

[0139] normal mode,if the landing gear command handle is UP,then

[0140] eventually the gears will be locked retracted and the doors will be

[0141] (closed.)

[0142] Process 200 is as described above regarding Figure 2-4 The process continues as described in each of the boxes 210-275 above, processing the original NL sentences and outputting the following formalized landing gear system requirements as a response:

[0143] When the command line is active, if the landing gear command handle has been pushed up and remains upward,

[0144] The landing gear will then lock and retract, and the hatch will be visible closing less than 15 seconds after the handle is pushed.

[0145] command handle has been pushed UP and stays UP,then the gears

[0146] will be locked retracted and the doors will be seen closed less than

[0147] 15s after the handle has been pushed.)

[0148] The formal landing gear system requirements generated by process 200 in the above example are implemented in the landing gear system control software, which is used to control the landing gear and doors, detect anomalies, and notify the pilot of the global status of the landing gear system and any anomalies.

[0149] Now for reference Figure 5 The computing device 300 is configured for use within the system 100 to perform any or more of the processes described herein, including but not limited to those described herein. Figure 1-4 The aspects disclosed herein. The computing device 300 includes a processor 305 operatively coupled to a communication device 315 for communication with other systems, a data storage device 330, one or more input devices 310 for receiving input from other systems and entities, one or more output devices 320, and a memory 325. The communication device 315 can facilitate communication with other systems and components, such as other devices, communication networks, and servers. Input devices 310 may include, for example, a keyboard, keypad, mouse or other pointing device, microphone, knob or switch, infrared (IR) port, docking station, and / or touchscreen. For example, input devices 310 can be used to input information into the computing device 300, such as natural language expressions. Output devices 320 may include, for example, a display (e.g., a screen), speakers, and / or a printer.

[0150] Data storage device 330 may include any suitable persistent storage device, including magnetic storage devices (such as magnetic tape, hard disk drives, and flash memory), solid-state storage devices, optical storage devices, read-only memory (ROM) devices, random access memory (RAM), storage class memory (SCM), or any other combination of fast access memory.

[0151] The conversion engine 335 and data 340 may include program instructions executed by the processor 305 to cause the computing device 300 to perform any or more of the processes described herein, including but not limited to... Figure 1-4 The aspects that are publicly disclosed in China.

[0152] Data 340 (cache or complete database) can be stored in volatile memory, such as memory 325. Data storage device 330 can also store data and other program code necessary to provide additional functions and / or the operation of computing device 300, such as device drivers, operating system files, etc.

[0153] Formalization enables rigorous verification and validation of systems, ensuring they meet expected requirements and function as intended. However, formal analysis requires a formal representation of the analyzed system, which can be a barrier for experts who do not employ formal methods. To make formalization more accessible and operable, this disclosure provides an LLM-based approach that interactively transforms unstructured natural language into formal representation. The LLM-based approach of this disclosure employs a usable, purely natural language-based interface that provides a high guarantee of the correctness of the transformation while remaining accessible to non-experts in formal methods.

[0154] The aspects disclosed herein are examples of this disclosure and may be embodied in various forms. For example, although some aspects are described herein as separate aspects, each aspect herein may be combined with one or more other aspects herein. The specific structural and functional details disclosed herein should not be construed as limiting, but rather serve as the basis for the claims and as a representative basis for teaching those skilled in the art to use this disclosure in various ways within virtually any reasonably detailed structure.

[0155] According to this disclosure, the phrases "in one aspect," "in multiple aspects," "in all aspects," "in some aspects," or "in other aspects" may each refer to one or more of the same or different aspects.

[0156] Further aspects of this disclosure are provided by the subject matter of the following clauses.

[0157] A system for formalizing the nonlinear requirements (NL) of an aerospace system using an LLM includes: a processor; and a memory including instructions that, when executed by the processor, cause the system to at least: receive, at the processor, an NL expression associated with the requirements of an aircraft's landing gear system as user input; revise the NL expression using an LLM to remove words that the LLM determines are unnecessary for the NL expression; replace at least one word in the revised NL expression with promoted atomic entities using the LLM to generate the underlying sentence structure of the revised NL expression; and use the LLM to... The underlying sentence structure is transformed into a lifting logic expression; the LLM is used to fundamentalize the lifting logic expression by replacing each lifting atomic entity with a core concept, such that multiple references to the same core concept map to the same fundamentalized atomic entity; the fundamentalized logic expression is parsed into an AST representation for syntax checking; the fundamentalized logic expression is transformed into a pseudo-NL expression to semantically match the fundamentalized logic expression; the LLM is used, at least in part, based on the pseudo-NL expression, to transform the NL expression into a formal requirement for the landing gear system of the aircraft; and the landing gear system of the aircraft is controlled based on the formal requirement.

[0158] According to the system described in the foregoing clause, the instructions, when executed by the processor, further cause the system to perform at least: generating an unbasicized entity map using values, the unbasicized entity map associating each promoted atom entity with the at least one word replaced by the promoted atom entity; and generating at least one basicized entity representing a core concept associated with the NL expression.

[0159] According to any of the foregoing clauses, the system wherein using the LLM to base the lifting logic comprises: retrieving each lifting atomic entity of the unbased entity map and the phrase embedding of each based entity; associating each lifting atomic entity of the unbased entity map with the most similar based entity by calculating the cosine similarity between the associated phrase embeddings of the lifting atomic entities and the based entities; and selecting the based entity with the highest cosine similarity. In aspects of this disclosure, the based entities can be written by aerospace requirements engineers, ensuring that requirements are closely related to concepts primarily derived from the aerospace field.

[0160] According to any of the foregoing clauses, in a system where using the LLM to replace at least one word in the revised NL expression with the boosted atomic entity includes providing context learning to the LLM.

[0161] According to any of the foregoing clauses, in a system where using the LLM to transform the underlying sentence structure into an improved logical expression includes providing context learning to the LLM.

[0162] According to any of the foregoing clauses, the instructions, when executed by the processor, further cause the system to initiate the LLM by providing a textual description of initial relevant context and knowledge, at least before using the LLM to revise the NL expression.

[0163] According to any of the foregoing clauses, the system wherein, when the instruction is executed by the processor, the system further causes the system to at least perform the function of receiving the instruction as user input at the processor, for the LLM to examine errors in the transformation of the basic logic expression in response to a failure to parse the basic logic expression into the AST representation.

[0164] According to any of the foregoing clauses, in a system where using the LLM to transform the underlying sentence structure into the boosting logic expression includes transforming the underlying sentence structure into temporal logic (e.g., linear temporal logic, signal temporal logic). In aspects of this disclosure, in addition to context learning, the boosting process can also utilize a fine-tuned LLM to train aerospace named entities discovered in aerospace system requirements.

[0165] According to any of the foregoing clauses, the NL expression is the NL requirement of the aerospace system.

[0166] A processor-implemented method for formalizing aerospace system NL requirements using an LLM (Limited Language Model), comprising: receiving an NL expression associated with requirements of an aircraft's landing gear system as user input; revising the NL expression using the LLM to remove words that the LLM determines are unnecessary for the NL expression; replacing at least one word in the revised NL expression with lifting atomic entities using the LLM to generate a lower-level sentence structure of the revised NL expression; transforming the lower-level sentence structure into a lifting logic expression using the LLM; fundamentalizing the lifting logic expression using the LLM by replacing each lifting atomic entity with a core concept, such that multiple references to the same core concept map to the same fundamentalized atomic entity; parsing the fundamentalized logic expression into an AST (Abstract Stamped Image) representation; transforming the NL expression into a formal requirement of the aircraft's landing gear system using the LLM, at least partially based on the fundamentalized logic expression; and controlling the aircraft's landing gear system based on the formalized requirement.

[0167] The processor implementation method according to any of the foregoing clauses further includes converting the basic logic expression into a pseudo-NL expression to semantically match the basic logic expression.

[0168] The processor implementation method according to any of the foregoing clauses further includes: generating an unbasicized entity map using values, the unbasicized entity map associating each promoted atom entity with the at least one word replaced by the promoted atom entity; and generating at least one basicized entity representing a core concept associated with the NL expression.

[0169] The method implemented by the processor according to any of the foregoing clauses, wherein using the LLM to base the lifting logic comprises: retrieving each lifting atomic entity of the unbased entity map and a phrase embedding of each based entity; associating each lifting atomic entity of the unbased entity map with the most similar based entity by calculating the cosine similarity between the associated phrase embeddings of the lifting atomic entities and the based entities; and selecting the based entity with the highest cosine similarity.

[0170] The method implemented by the processor according to any of the foregoing clauses, wherein using the LLM to replace at least one word in the revised NL expression with the boosted atomic entity includes providing context learning to the LLM.

[0171] The method implemented by the processor according to any of the foregoing clauses, wherein using the LLM to transform the underlying sentence structure into a boosting logic expression includes providing context learning to the LLM.

[0172] The method implemented by the processor according to any of the foregoing clauses further includes initiating the LLM by providing a textual description of the initial relevant context and knowledge before using the LLM to revise the NL expression.

[0173] A non-transient computer-readable storage medium storing instructions for causing a processor to execute a processor-implemented method for formalizing aerospace system NL requirements using an LLM, comprising: receiving an NL expression associated with the aerospace system as user input at the processor; revising the NL expression using the LLM to remove words that the LLM determines are unnecessary for the NL expression; using the LLM to promote atomic entities to replace at least one word in the revised NL expression to generate a low-level sentence structure of the revised NL expression; using the LLM to transform the low-level sentence structure into a promotion logic expression; using the LLM to fundamentalize the promotion logic expression by replacing each promotion atomic entity with a core concept, such that multiple references to the same core concept map to the same fundamentalized atomic entity; using the LLM to transform the NL expression into formalized requirements of the aerospace system, at least partially based on the fundamentalized logic expression; and controlling the aerospace system based on the formalized requirements.

[0174] The non-transient computer-readable storage medium according to any of the foregoing clauses, wherein the aerospace system is the landing gear system of an aircraft.

[0175] The non-transient computer-readable storage medium according to any of the foregoing clauses, wherein the method further comprises: resolving the basic logic expression to an AST representation; and using the LLM to reverse-convert the AST representation to an NL representation of the basic logic expression.

[0176] According to any of the foregoing clauses, the non-transient computer-readable storage medium, wherein the method further includes converting the basic logic expression into a pseudo-NL expression to semantically match the reverse conversion with the basic logic expression.< / sentence>

Claims

1. A system for converting the natural language (NL) requirements of aerospace systems into formalized expressions using a large language model (LLM), characterized in that, The system includes: processor; and The memory includes instructions that, when executed by the processor, cause the system to perform at least the following: The processor receives an NL expression associated with the landing gear system requirements of the aircraft as user input. Use LLM to: Revise the NL expression to remove words that the LLM determines are unnecessary for the NL expression; Replace at least one word in the revised NL expression with an elevated atomic entity to generate the underlying sentence structure of the revised NL expression; Transform the underlying sentence structure into a boosting logic expression; and The lifting logic is fundamentalized by replacing each lifting atomic entity with a core concept, so that multiple references to the same core concept are mapped to the same fundamentalized atomic entity. The basic logical expression is parsed into an abstract syntax tree (AST) representation for syntax checking; The basic logic expression is transformed into a pseudo-NL expression to semantically match the basic logic expression; Using the LLM, at least in part based on the pseudo-NL expression, the NL expression is transformed into formal requirements for the landing gear system of the aircraft; and The landing gear system of the aircraft is controlled based on the formal requirements stated herein.

2. The system according to claim 1, characterized in that, in, When the instruction is executed by the processor, it further causes the system to perform at least the following: The values ​​are used to generate an unbasic entity map, which associates each promoted atom with the at least one word that is replaced by the promoted atom; and Generate at least one basic entity representing the core concept associated with the NL expression.

3. The system according to claim 2, characterized in that, in, Using the LLM to base the boosting logic includes: Retrieve the phrase embeddings of each promoted atomic entity and each basic entity in the unbasicized entity map; Each boosted atomic entity mapped by the un-basicized entity is associated with the most similar basicized entity by calculating the highest cosine similarity between the associated phrase embeddings of the boosted atomic entities and the basicized entities; and Select the basic entity with the highest cosine similarity.

4. The system according to claim 1, characterized in that, in, Using the LLM to replace at least one word in the revised NL expression with the elevated atomic entity includes providing context learning to the LLM.

5. The system according to claim 1, characterized in that, in, Using the LLM to transform the underlying sentence structure into an improved logical expression includes providing the LLM with contextual learning.

6. The system according to claim 1, characterized in that, in, When executed by the processor, the instructions further cause the system to initiate the LLM by providing a textual description of the initial relevant context and knowledge, at least before using the LLM to revise the NL expression.

7. The system according to claim 1, characterized in that, in, When the instruction is executed by the processor, it further causes the system to at least receive the instruction as user input at the processor, for the LLM to examine errors in the transformation of the basic logic expression in response to a failure to parse the basic logic expression into the AST representation.

8. The system according to claim 1, characterized in that, in, Using the LLM to transform the underlying sentence structure into the boosted logic expression includes transforming the underlying sentence structure into tense logic.

9. The system according to claim 1, characterized in that, in, The NL expression refers to the NL requirements of aerospace systems.

10. A processor-implemented method for converting natural language (NL) requirements of aerospace systems into formalization using a large language model (LLM), characterized in that, The method includes: Receive the NL expression associated with the requirements of the aircraft's landing gear system as user input; Use LLM to: Revise the NL expression to remove words that the LLM determines are unnecessary for the NL expression; Replace at least one word in the revised NL expression with an elevated atomic entity to generate the underlying sentence structure of the revised NL expression; Transform the underlying sentence structure into a boosting logic expression; and The lifting logic is fundamentalized by replacing each lifting atomic entity with a core concept, so that multiple references to the same core concept are mapped to the same fundamentalized atomic entity. The basic logical expression is parsed into an abstract syntax tree (AST) representation; Using the LLM, at least in part based on the fundamental logical expression, the NL expression is transformed into the formal requirements of the landing gear system of the aircraft; and The landing gear system of the aircraft is controlled based on the formal requirements stated herein.

Images