Test equipment

Abstract

A test apparatus (100) for testing at least a portion of an aircraft system is disclosed. The test apparatus (100) is configured to: obtain first data (210), the first data (210) comprising a plurality of input values; and process the input values using a classifier (230), the classifier (230) being configured to model the operation of at least a portion of the aircraft system (220). Second data is obtained, the second data comprising a set of output values (240b). The output values (240b) from the second data are processed together with the output values (240a) generated by the classifier to identify differences between the operation of at least a portion of the aircraft system (220) and the operation modeled by the classifier (230).

Description

Technical Field

[0001] The present invention relates to testing at least a portion of an aircraft system. In particular, but not exclusively, the present invention relates to a test apparatus for testing the operation of at least a portion of an aircraft system based on the operation modeled by a classifier. Background Art

[0002] Aircraft typically include multiple systems, and the development and testing of aircraft may involve a significant amount of time and expense. Therefore, a system that can facilitate the development and testing of aircraft is needed. Summary of the Invention

[0003] A first aspect of the present invention provides a test device for testing at least a portion of an aircraft system, the device comprising at least one memory and at least one processor, the test device comprising program code stored in the at least one memory and configured to: obtain first data comprising a plurality of input values representing an operating state of an aircraft; process the plurality of input values using a trained classifier to generate a first set of output values, the trained classifier being configured to model the operation of at least a portion of the aircraft system; obtain second data comprising a second set of output values generated by operating at least a portion of the aircraft system based on the plurality of input values; process the first set of output values and the second set of output values to identify any differences between the operation of at least a portion of the aircraft system and the operation modeled by the trained classifier; and generate a signal indicative of the identified differences.

[0004] This can allow for the identification of discrepancies between the desired operation of an aircraft system and the way the aircraft system actually operates. Once identified, the aircraft system can be updated and / or modified to more closely match its desired operation. Operation can include the way the aircraft system interacts with a user, such as a flight crew, via one or more user displays, and / or the way equipment within the aircraft system operates.

[0005] Optionally, the test equipment is configured to: obtain training data representing expected operation of at least a portion of the aircraft system for at least one operating state, the training data comprising a further plurality of input values and a corresponding set of output values; and train the classifier using the training data.

[0006] Training the classifier based on training data representing desired operation of at least a portion of the aircraft system allows the classifier to continue to identify differences between the designed aircraft system and the desired operation of the aircraft system. Training can allow the classifier to be updated to account for changes in the desired operation of the aircraft system.

[0007] Optionally, the test equipment is in communication with at least one mobile computing device, and the test equipment is configured to receive data indicative of decisions made by a user of the at least one mobile computing device regarding operation of at least a portion of the aircraft system, and wherein the test equipment is configured to generate the training data based on the received data.

[0008] Mobile computing devices may provide an efficient way to collect data from users so that the training data accurately represents the operation of at least a portion of the aircraft system from the user's perspective. In this context, "users" may include at least crew members (including pilots), ground crew or maintenance personnel, and / or fleet managers who may be working on or off the aircraft.

[0009] Optionally, the testing device is configured to generate training data based on the identified differences.

[0010] This can allow the classifier to be trained based on its current performance. For example, if the classifier correctly identifies a difference, that identification can be strengthened, and if the classifier incorrectly identifies a difference, the classifier can be updated to avoid such errors in the future.

[0011] Optionally, generating training data based on the identified differences comprises comparing the identified differences with user input indicating a decision regarding the identified differences.

[0012] This may allow the feedback given to the classifier to represent the actual desired functionality of the aircraft system. The desired functionality perceived by the user may change, and therefore, continuously providing it in this way may allow the test equipment to evolve as preferences change.

[0013] Optionally, the plurality of input values represent at least one of: an operational state of at least one component associated with an aircraft system; and an output from at least one sensor configured to sense a corresponding environmental condition.

[0014] This may allow the test equipment to identify differences in the operation of the aircraft system and the classifier based on multiple input values that depend on both internal factors defined by the system components and external or environmental factors that may affect the operation of the aircraft.

[0015] Optionally, the operational state of at least one component associated with the aircraft system includes a control input, the control input including at least one of: a value representing the control input; a value representing a rate of change of the control input; and a value representing a difference between the control input and another control input.

[0016] Optionally, at least a portion of the aircraft system is a computer program for controlling at least a portion of the aircraft.

[0017] In this way, the test equipment can test the functionality of the computer program in the aircraft.

[0018] Optionally, at least a portion of the aircraft system includes a combination of aircraft equipment and a computer program for operating the aircraft.

[0019] In this manner, the test equipment can test the functionality of systems in the aircraft, including computer programs and the operation of components and equipment in the aircraft.

[0020] Optionally, at least a portion of the aircraft system includes aircraft equipment including at least one sensor for generating the second set of output values.

[0021] In this manner, the test equipment can test the functionality of components in an aircraft that operate mechanically and / or electrically but do not rely on computer programs to control their operation.

[0022] Optionally, at least a portion of the aircraft system includes at least one of: an avionics system, a flight control system, a brake control system, an instrument and recording system, a landing gear control system, and a fuel system.

[0023] A second aspect of the present invention provides a method for testing at least a portion of an aircraft system, the method comprising: obtaining first data comprising a plurality of input values representing an operating state of the aircraft; processing the plurality of input values using a trained classifier to generate a first set of output values, the trained classifier being configured to model the operation of at least a portion of the aircraft system; generating second data comprising a second set of output values by operating at least a portion of the aircraft system based on the plurality of input values; processing the first set of output values and the second set of output values to identify any differences between the operation of at least a portion of the aircraft system and the operation modeled by the trained classifier; and generating a signal indicative of the identified differences.

[0024] This can allow for the identification of discrepancies between the desired operation of an aircraft system and the way the aircraft system actually operates. Once identified, the aircraft system can be updated and / or modified to more closely match its desired operation. Operation can include the way the aircraft system interacts with a user, such as a flight crew, via one or more user displays, and / or the way equipment within the aircraft system operates.

[0025] Optionally, the method includes obtaining training data representing expected operation of at least a portion of the aircraft system for at least one operating state; and training the classifier using the training data.

[0026] Training the classifier based on training data representing desired operation of at least a portion of the aircraft system allows the classifier to continue to identify differences between the designed aircraft system and the desired operation of the aircraft system. Training can allow the classifier to be updated to account for changes in the desired operation of the aircraft system.

[0027] Optionally, the training data is generated from the further aircraft system during operation of the further aircraft system.

[0028] In this way, the test equipment can identify differences between the operation of at least a portion of the aircraft system and the operation of another aircraft under real-world conditions. This can increase the accuracy of the test equipment when used to test real-world scenarios.

[0029] Optionally, the training data is generated based on input from a user indicative of a decision regarding at least one operational state of the aircraft.

[0030] Taking user decisions into account can increase the reliability of the training data.The user can identify where the training data is incorrect or does not represent the expected operation of at least a portion of the aircraft system.

[0031] Optionally, the method comprises generating further training data based on said identified differences, and training the classifier using the further training data.

[0032] This can provide a feedback loop in which the ability of the test device to correctly identify differences is used to reinforce the training of the classifier.

[0033] Optionally, the method includes modifying at least a portion of an aircraft system based on the identified difference.

[0034] This allows at least a portion of the aircraft system to be adjusted to correct the discrepancy between the identified system operation and the desired operation. In this way, at least a portion of the aircraft system can function more closely as needed.

[0035] A third aspect of the present invention provides an aircraft comprising one or more aircraft systems, at least one of the one or more aircraft systems having been tested according to the method described above.

[0036] A fourth aspect of the present invention provides a test aircraft comprising one or more aircraft systems and a test device as described above for testing at least a portion of the one or more aircraft systems included in the test aircraft.

[0037] A fifth aspect of the present invention provides a test system for testing an aircraft control system, the system comprising a memory, a processor, a control input interface for receiving aircraft control inputs, and a control output interface for receiving aircraft control outputs, the test system being configured to: receive aircraft control inputs via the control input interface; process the aircraft control inputs using a trained model of the expected behavior of the aircraft control software under test to determine expected control outputs; receive actual control outputs from the aircraft control software via the control output interface; compare the expected control outputs with the actual control outputs to identify differences between them; and generate a signal indicating the difference between the expected control outputs and the actual control outputs.

[0038] This can allow for the identification of discrepancies between the desired operation of an aircraft system and the way the aircraft system actually operates. Once identified, the aircraft system can be updated and / or modified to more closely match its desired operation. Operation can include the way the aircraft system interacts with a user, such as a flight crew, via one or more user displays, and / or the way equipment within the aircraft system operates.

[0039] A sixth aspect of the present invention provides a test system for testing aircraft control software, the system comprising: a memory; a control input interface for receiving aircraft control input and storing the input in the memory; a model processing engine for processing the aircraft control input using a trained model of the expected behavior of the aircraft control software under test to determine an expected control output and store the output in the memory; a control output interface for receiving an actual aircraft control output and storing the output in the memory; a difference engine for comparing the expected control output with the actual control output to identify any differences between them; and an output generator for generating a signal indicative of the difference identified by the difference engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0041] Figure 1 is a block diagram of a test device according to an example;

[0042] Figure 2 is a block diagram illustrating the operation of a test device according to an example;

[0043] Figure 3 is a block diagram of the training process based on examples;

[0044] Figure 4 is a block diagram illustrating the operation of a test device according to an example;

[0045] Figure 5is a flowchart illustrating a method according to an example; and

[0046] Figure 6 is an illustrative diagram of an aircraft according to an example. DETAILED DESCRIPTION

[0047] Aircraft systems are becoming increasingly complex. With advancements in technology and increased levels of automation, aircraft capabilities are also increasing. Consequently, the design of systems used to monitor and control aircraft, as well as systems designed to interact with aircraft operators, such as the crew, is becoming increasingly complex. In particular, while aircraft systems can be designed based on the intended operation expected in most situations, actual aircraft use may highlight other situations where, for example, the crew may expect or prefer that the aircraft system operate and / or respond differently than designed.

[0048] Once an aircraft system is implemented, the practicalities of developing, testing, and rolling out a new system, particularly revised software, may necessitate a period of time during which the aircraft system cannot be modified. Furthermore, during testing of an aircraft system, if it is discovered that the system does not operate as expected, the development of the system may be regressed to the stage where the discrepancies were initially implemented, and testing may then need to be repeated.

[0049] Identifying discrepancies between aircraft systems (e.g., as determined by potential operators of the aircraft) and the expected operation of the aircraft systems early in the development and testing process may reduce the time required to develop aircraft systems and reduce the need for system revisions or modifications, thereby increasing the effectiveness of such systems.

[0050] Certain examples described herein involve comparing the operation of an aircraft system with the output of a classifier that models the operation of the aircraft system. The classifier is trained using data representing the desired operation of the aircraft system. The aircraft system can then be configured based on this comparison so that the system operates as expected under given circumstances. The classifier can be continuously trained so that it adapts to the desired operation of the aircraft system, which can change over time, for example, as practices, expectations, and procedures evolve.

[0051] Advantageously, according to an example, the classifier is trained using at least some information and data that is independent of the aircraft design process. In contrast, known methods for testing aircraft systems, particularly control software for aircraft systems, may involve code execution by the software design team and the use of test scripts and debuggers, which may also be designed by the software design team. A potential disadvantage of known methods is that out-of-specification scenarios may not be addressed by the software design team using the control software and code execution, test scripts, and debuggers. Such code execution, test scripts, and debuggers may therefore fail to achieve the desired operational differences for these specific scenarios.

[0052] The information and data used to train the classifier can come from a variety of sources, at least some of which are independent of the design process. For example, most of the systems and operations of a new aircraft design may be the same or very similar to the design and operation of an existing aircraft. Therefore, according to some examples, the classifier is trained using known-correct information and data associated with the operation of an existing type of aircraft. In most cases, the software design team and aircraft users, such as the crew, have the same views on aircraft operation and design parameters. For example, both groups agree that a reduction in braking performance should be immediately identified to the pilot. Other scenarios are not so clear-cut, and certain subjective criteria (such as those based on the crew's experience) may need to be taken into account. In such cases, the opinions of the software design team and aircraft users may differ. While measures will obviously be taken to ensure that the likelihood of such an occurrence is minimized, it is not possible to design and / or test every conceivable scenario that could occur. This often results in the need to revise and roll out aircraft control software after the new aircraft enters service, which is a time-consuming and expensive process. For example, as will be described herein with respect to examples, a software design team may legitimately perceive a particular aircraft operating scenario as warranting a warning condition, whereas an experienced flight crew may not perceive the scenario as warranting a warning condition. Thus, according to examples, the classifier may also be trained using experience and input from users, such as flight crew members.

[0053] Figure 1An example test device 100 for testing at least a portion of an aircraft system is shown according to an example. The test device 100 includes at least one memory 110 and at least one processor 120. The memory 110 may be any suitable combination of volatile and non-volatile memory including random access memory (RAM), read-only memory (ROM), synchronous dynamic random access memory (SDRAM), or any other suitable type of memory. The at least one processor 120 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), an application specific instruction set processor (ASIP), or any other suitable processing device. The test device 100 may include Figure 1 At least one interface 130 is shown in dashed lines. Such an interface 130 may be operable to communicate with one or more computing devices 160 and / or at least a portion of the aircraft system. The test apparatus 100 may also include a user interface 140 for interacting with a user. In an example, the user interface 140 includes a graphical display for displaying system information such as status information and / or at least one user input device for receiving input from a user to control the test apparatus 100. The test apparatus 100 is Figure 1 1 is shown as a single device. However, it should be understood that the test device 100 may include distributed devices, such as multiple distributed computing devices communicatively coupled via a wired network or a wireless network. In any case, the test device can operate by executing corresponding test software 150, which can be loaded into the main memory 110 and executed by the processor 120 at runtime.

[0054] Figure 2 A block diagram of elements of the test apparatus 100 according to an example is shown in greater detail. The block diagram shows that input data 210 is processed by at least a portion of an aircraft system 220 and independently by a trained classifier 230. Output from the system 220 and the classifier 230 is then processed by a processing engine at block 250 to generate a signal 260. The trained classifier 230 is implemented by the apparatus 100, wherein the memory 110 includes a set of instructions 150 that, when executed by the processor 120, implement the classifier 230 and the processing engine 250.

[0055] Test equipment 100 is configured to obtain first data 210, which includes a plurality of input values representing an operational state of an aircraft. The operational state of the aircraft may include a state of one or more components in or on the aircraft and an indication of environmental conditions in which the aircraft operates. According to this example, the plurality of input values represent at least one of an operational state of at least one component associated with an aircraft system and an output from at least one sensor configured to sense a corresponding environmental condition. First data 210 may represent a transient state of the aircraft, wherein first data 210 represents the state of one or more components or the environmental condition at a given moment or successive moments in time. Alternatively, first data 210 may represent a dynamic state of the aircraft, wherein first data 210 includes values representing a time period in aircraft operation. In this case, first data 210 includes a plurality of dynamically changing values representing each input.

[0056] The operational state of at least one component associated with an aircraft system may include a control input. Such a control input may be due to a crew member, such as a pilot, operating or actuating a device, equipment, or system. The control input may include a value of the control input, for example, a value representing the position of a throttle, brake actuator, and / or other control input. Alternatively or additionally, the control input may include a rate of change rather than a value. In some examples, the control input may be a multivariate quantity indicating an instantaneous value (e.g., position) and a rate of change and / or direction of change. The control input may be represented by multiple values or multiple multivariate quantities representing the operation of a device, equipment, or system over a defined time period. The control input may include a value representing the difference between the control input and another control input (e.g., the difference between the position of a first throttle corresponding to a first engine and the position of a second throttle corresponding to a second engine).

[0057] The first data 210 may be obtained from an external device via a network, for example, the data 210 may be collected in real time from an active aircraft or an aircraft simulator (or a mixture of the two) and sent to the apparatus 100, thereby allowing the aircraft system 220 to be tested for actual operating conditions and associated parameters. Alternatively, the first data 210 may be sent to the apparatus 100 by a tester, thereby allowing the test apparatus 100 to be operated remotely and / or "after the event."

[0058] The first data 210 may include a stored dataset comprising actual datasets collected from in-service aircraft. Alternatively, or in addition, the stored dataset may include data generated by a tester to test certain scenarios. The tester may manually generate the first data 210 by selecting or designing a plurality of input values representing each corresponding input to at least a portion of the aircraft system 210. Alternatively, the tester may generate the first data 210 using statistical methods. For example, the aircraft system 220 may be configured to receive multiple inputs, and the value for each input may be represented using a probability distribution. The probability distribution may be based on physical laws, component operating limits, and / or generated from actual data. Probabilities may be used to generate a dataset representing possible states of the aircraft, including states not represented in the actual data from in-service aircraft. In any case, these methods enable testing of a range of real-world scenarios and theoretical or artificial scenarios.

[0059] For example, when testing a brake control system, the plurality of input values may indicate the operating status of one or more hydraulic systems configured to operate the brakes, as well as the status of one or more replaceable (e.g., electric) motors and corresponding power supplies for operating the brakes in the event of a hydraulic failure. The plurality of input values may also include other relevant information, such as the operating status of the aircraft engines, an indication of the presence of wheel weights, and an indication of environmental conditions (such as wind speed and direction and external temperature and humidity) from one or more aircraft sensors. The plurality of input values may represent the status of equipment included in other aircraft systems. There may be a correlation or relationship between the operation of one or more additional aircraft systems and the operation of the system being tested.

[0060] return Figure 2 The test device 100 is configured to process the plurality of input values using the trained classifier 230 to generate a first set of output values 240 a based on the obtained first data 210 , where the trained classifier 230 is configured to model the operation of at least a portion of the aircraft system 220 .

[0061] Test apparatus 100 is also configured to obtain second data based on a plurality of input values, the second data comprising a second set of output values 240b generated by operating at least a portion of aircraft system 220. The aircraft system tested using apparatus 100 can be any of a number of aircraft systems. In some examples, at least a portion of aircraft system 220 is a computer program or "control software" for controlling at least a portion of an aircraft. For example, at least a portion of system 220 may include firmware specifically configured to monitor and / or control the operation of one or more components of an aircraft. In some examples, at least a portion of system 220 is configured to interact with an aircraft operator, such as a crew member. If at least a portion of aircraft system 220 is a computer program, operating at least a portion of aircraft system 220 based on a plurality of input values may include running the computer program and introducing the input values into it. The at least a portion of aircraft system 220 comprising the computer program may include the control software and may also simulate at least some aircraft equipment. For example, rather than having the control software run in conjunction with specific equipment, the behavior of the simulated equipment may be used instead. For example, a computer program can be configured to simulate the functionality of aircraft equipment, and in particular, the interaction between the control software under test and the aircraft equipment. This can reduce the cost of the testing phase and provide greater flexibility in the scenarios that can be tested. For example, the corresponding simulation can be used to implement the operation of and interaction with equipment that has not yet been integrated into the overall system.

[0062] In the examples described herein, the number of values in the first data and the size of the output value sets 240a, 240b can depend on the actual aircraft system being modeled. In some examples, the classifier 230 generates a different number of output values than the aircraft system being tested. For example, the classifier 230 can generate values that are related to the output values from the actual aircraft system, and processing the output value sets can include processing the output values from the actual aircraft system 220 to generate data that can be directly compared to the output values from the classifier 230.

[0063] In other examples, at least a portion of aircraft systems 220 includes a combination of aircraft equipment and a computer program for operating at least a portion of the aircraft (including the aircraft equipment or at least some of the aircraft equipment). For example, a control program included in at least a portion of aircraft systems 220 that controls the aircraft equipment receives inputs from the controlled equipment and / or other parts of the aircraft. The control program monitors these inputs to check whether the corresponding equipment is operating correctly and / or adjust the controlled objects. The control program can provide feedback to a user display. For example, if at least a portion of aircraft systems 220 is operating to adjust braking torque to slow the aircraft, the inputs may include speed measurements, the current torque level being applied, an indication of which brakes are being operated, and the heat level generated at each brake. Some or all of this information can be displayed to the user so that they can monitor the operation and ensure it is functioning properly. In some examples, at least a portion of aircraft systems 220 is part of a test bench. A test bench includes one or more aircraft systems that are tested during the development phase before being deployed on one or more aircraft. Alternatively, at least a portion of aircraft systems 220 may be tested while operating a test aircraft.

[0064] Alternatively, at least a portion of aircraft system 220 may not include a computer program, but may instead include aircraft equipment including at least one sensor for generating second set of output values 240b. In this case, operating at least a portion of aircraft system 220 based on the input values includes inducing an operational state in at least a portion of system 220 represented by the input values.

[0065] As discussed, at least a portion of aircraft systems 220 may include a portion of any suitable system in the aircraft. For example, at least a portion of aircraft systems may include one or more of the following: an avionics system, a flight control or navigation system, a brake control system (or other hydraulic control system), an instrumentation and recording system, a landing gear control system, and a fuel system.

[0066] The test device 100 then processes the first set of output values 240a and the second set of output values 240b, for example using a processing engine 250, to identify any differences between the operation of at least a portion of the aircraft system 220 and the operation modeled by the trained classifier 230. In this manner, the test device 100 is able to highlight any differences between how at least a portion of the aircraft system 220 actually responds and / or operates under certain conditions and how it is expected or desired to operate under those conditions. Inferences are generated based on the operational model implemented by the classifier. In some examples, processing the two sets of output values 240a, 240b may include performing a comparison between the corresponding output values to identify any differences. In other examples, processing the two sets of output values 240a, 240b may include performing a comparison between the corresponding values to identify differences that exceed a given tolerance and / or confidence level (e.g., within 10%). In these examples, the processing engine 250 may operate as or include a comparator or similar processing element. In further examples, processing of the two sets of output values 240 a , 240 b may involve more complex processing configured to identify patterns or trends in the output values that indicate differences between the operation of at least a portion of aircraft system 220 and the operation modeled by trained classifier 230 .

[0067] Figure 3 A more detailed example of at least a portion of a test aircraft system 300 is schematically shown. Figure 3 In the example, at least a portion of the aircraft system 300 under test is a computer program and can therefore be implemented by a test device 310. The test device 310 can use the trained classifier 320 to simultaneously process multiple input values I_1, I_2, and I_3 and can use the input values I_1, I_2, and I_3 to implement or run a test version of at least a portion of the aircraft system 300. For example, the test device 310 can implement one or more virtual environments in which the classifier 320 and the aircraft system 300 including the computer program can be executed to generate corresponding output value sets.

[0068] Classifier 320 receives a plurality of input values I_1, I_2, I_3 via control input interface 322 and processes these input values to generate a first set of expected output values CL_1, CL_2. The plurality of input values I_1, I_2, I_3 may include input values representing the operating state of a component in the aircraft and / or one or more environmental conditions that may limit the operating state of the aircraft. Although the first set of output values is shown as including two values, it should be understood that the first set of output values may include more or fewer values than the plurality of input values. In some examples, a set of output values includes only one output value.

[0069] At least a portion of aircraft system 300 operates based on a plurality of input values I_1, I_2, I_3, which in this example includes using the plurality of input values I_1, I_2, I_3 as input to a computer program included in at least a portion of aircraft system 300. At least a portion of aircraft system 300 generates a second set of output values O_1, O_2, which are received by test equipment 310 via control output interface 326. The first and second sets of output values are then processed by one or more comparison modules 330a, 330b (e.g., operating as processing engine 250). In the example shown, first comparison module 330a is configured to compare output values CL_1 and O_1, while second comparison module 330b is configured to compare output values CL_2 and O_2. The results R_1, R_2 of these comparisons are then used to identify any differences between the operation modeled by classifier 320 and the operation of at least a portion of aircraft system 300. In some examples, this may include performing some additional processing using any suitable statistical method to identify trends and / or systematic differences between sets of output values that indicate differences between modeled operation and operation of the actual aircraft system.

[0070] Figure 4 A flow chart of a method 400 for testing at least a portion of an aircraft system, according to an example, is shown. At block 410, method 400 includes obtaining first data comprising a plurality of input values representing an operational state of an aircraft. At block 420, method 400 includes processing the plurality of input values using a trained classifier configured to model the operation of at least a portion of the aircraft system to generate a first set of output values. At block 430, method 400 includes generating second data comprising a second set of output values by operating at least a portion of the aircraft system based on the plurality of input values. At block 440, method 400 includes processing the first set of output values and the second set of output values to identify any discrepancies between the operation of at least a portion of the aircraft system and the operation modeled by the trained classifier. At block 450, method 400 includes generating a signal indicating the identified discrepancies. Method 400 may also include modifying at least a portion of the aircraft system based on the identified discrepancies. In this manner, the aircraft system is modified so that it operates in a manner that more closely resembles the desired operation of the aircraft system.

[0071] Modifying at least a portion of the aircraft system based on the identified differences can be performed in a variety of ways. In one example, the results of the comparison can be provided to a developer of the aircraft system. The developer can then review the differences and modify the design of the aircraft system so that it complies with the operation determined by the classifier. Alternatively, when comparing the output from the aircraft system and the output from the test equipment, the test equipment can be configured to highlight specific parts of the aircraft system to be changed so that the operation of the aircraft system matches the operation determined by the classifier. In another example, the test equipment can be configured to generate a report based on the identified differences. The report can identify areas that need to be reviewed and provide one or more suggestions for modifying the aircraft system to comply with the operation of the classifier.

[0072] An example application of the test device 100 to an aircraft brake control system will now be described. First data 210 includes values indicating the operating status of components used to operate the brakes. In this example, when the hydraulic pump used to operate the brakes is inoperative (i.e., shut down) while the aircraft is moving on the ground, the brake control system is initially specified and designed to generate an Amber Alert on the pilot's display to indicate that the normal or primary power source is inoperative and that a backup power source for the brakes is being used. An Amber Alert may be an appropriate response for the brake control system because the brakes need to be operated, but not by their primary control unit (i.e., the hydraulic pump system). Such an Amber Alert, if displayed on the display before takeoff, may cause concern among the crew and could lead to grounding the aircraft, resulting in flight delays or even cancellations. However, by using the test device 100 described herein, it can be determined that there may be some situations in which the crew would perceive an Amber Alert as an inappropriate or undesirable response in the described scenario. For example, when an aircraft is being towed with its engines shut down and, therefore, the normal power source (e.g., hydraulic pump) is not active, using the backup power source for the brakes may be considered acceptable and indeed desirable in this situation. Consequently, the crew may consider an Amber Alert inappropriate and potentially a cause for undue concern. Conversely, a more appropriate pilot display may indicate that the brakes are being activated by their backup power source, but without an Amber Alert. In this case, trained classifier 230 is trained to recognize that if the primary or normal power source is not active with the engines shut down, the modeled brake control system should not display an Amber Alert. This training of trained classifier 230 is accomplished using information and data from existing aircraft and / or feedback from experienced crew members. Thus, during testing, when the output values from trained classifier 230 are compared with the output values from at least a portion of aircraft systems 220 (i.e., the brake control system), the discrepancy identified is an Amber Alert generated by the aircraft system, not by trained classifier 230.

[0073] During inspection, the system designer will recognize that the aircraft system 220 could benefit from considering changes in engine status (i.e., on or off) when evaluating whether it is appropriate to issue an Amber Alert when hydraulic braking is not possible. If this is recognized during the design process, while the aircraft is actually a test bed, the control software can be revised with relatively little delay. In contrast, if the problem is identified after the new aircraft is put into service, it may take months to deploy the new control software. In the meantime, a guidance note may need to be issued to the crew to remind them of the meaning of the Amber Alert in specific circumstances, otherwise the aircraft may be grounded unnecessarily.

[0074] In situations such as these, the test equipment 100 may be able to identify portions of the firmware that are interacting with the flight crew in an unexpected manner. Learning to operate an aircraft can require significant time and resources. Therefore, if it were possible to identify and correct discrepancies between flight crew expectations and the operation of aircraft systems, the time spent learning to operate the aircraft could be reduced as the systems become more intuitive.

[0075] In another specific example, at least a portion of aircraft systems 220 may be an automatic braking system within a braking control system. This automatic braking system is configured to automatically engage the brakes during a landing procedure and automatically disengage them during a takeoff procedure. In this case, first data 210 includes values indicating the operating status of the components used to operate the brakes and a value indicating the stage of the landing procedure the aircraft is engaged in. This includes information such as the weight on the wheels, the operating status of the engines, the orientation of the aircraft, lateral g-forces, and other relevant information. Once the aircraft makes contact with the ground, the brakes are automatically engaged to decelerate the aircraft, for example, based on a weight threshold and / or a sufficient time period associated with the detected weight on the wheels.

[0076] Sometimes, for various reasons, a landing attempt is aborted, where the aircraft initially makes contact with the ground, but instead of landing, takes off again and prepares for a second landing attempt. In this example, after the aborted landing attempt, a preprogrammed "aborted landing procedure" can be deployed, activating and / or deactivating various elements of the aircraft system in preparation for the next landing attempt. This procedure can reduce the number of system checks required by the pilot before attempting a second landing, compared to the number of checks that might have been required for the initial landing attempt. However, in certain circumstances, it has been found that flight crews prefer a full system reset after an aborted landing attempt, allowing them to perform a full landing protocol rather than a simplified aborted landing protocol during the second landing attempt. Using the test apparatus 100 described herein, it was determined that there are some situations in which flight crews would prefer a full reset of the autobrake system, as in the described scenario. In contrast, there may be other specific scenarios in which flight crews would prefer a partial engagement of the brake system rather than a full reset, and the examples herein will also be able to identify these scenarios.

[0077] In another specific example, where at least a portion of aircraft system 220 is an automatic braking system within a braking control system, the automatic braking system is configured to automatically apply the brakes during a landing procedure. In practice, and in the absence of an automatic braking system, varying environmental and aircraft characteristics may influence how an experienced operator decides to apply the brakes during a landing procedure. This manner of applying the brakes may result in different amounts of braking force being applied at different stages of the landing procedure. It is difficult to develop a set of rules for the automatic braking system that accurately selects the same amount of braking force as an experienced operator would in various scenarios. In some cases, pilots may apply the brakes based on experience, for example, rather than according to an operating manual, and the manner in which they apply the brakes can be quantified. In this example, if the automatic braking system does not select the same amount of braking force as an experienced pilot in a given test scenario, test equipment 100 can quantify the differences between braking procedures performed by an experienced pilot and those performed by the automatic braking system under different environmental conditions. In this way, the performance and / or behavior of the automatic braking system can be improved by programming it to more closely mimic the braking techniques used by experienced pilots in various landing conditions.

[0078] It should be understood that the preceding examples have been simplified to aid understanding. It should be understood that there may be many other factors, and therefore many other input values, present for presenting a given scenario for testing the output of an aircraft system. Therefore, using a trained classifier that has been trained based on the expected operation of the aircraft or on operational examples of other aircraft can identify exceptions to otherwise apparently reasonable operating rules. Without the use of a trained classifier, these exceptions may remain designed into the aircraft system, potentially requiring system updates later in the development phase or even after the new aircraft (or aircraft system) is introduced.

[0079] More generally, once a discrepancy is identified, the test apparatus 100 is configured to generate a signal 260 indicative of the discrepancy. This signal 260 may be used to provide information to a user of the test apparatus 100 identifying how the operation of at least a portion of the aircraft system 220 differs from the operation modeled by the trained classifier 230. This may be used to notify the user so that at least a portion of the system 220 may be investigated and, if necessary, modified to reduce or eliminate the discrepancy. The signal 260 may include data, a data set, or some other representation of data or information indicative of the identified discrepancy, which may be transmitted and / or stored in memory and displayed, as appropriate.

[0080] For the identified differences to be useful, trained classifier 230 should accurately represent the expected operation of at least a portion of aircraft system 220. To this end, test equipment can be configured to train classifier 230. Alternatively, classifier training can be performed elsewhere, and the trained classifier 230 can be loaded into test equipment 100. Classifier 230 can be trained by obtaining training data representing the expected operation of at least a portion of aircraft system 220 for at least one operational state. The training data includes at least another plurality of input values (and preferably more instances of the input values) and at least one corresponding set of output values. Each of the additional plurality of input values is associated with a corresponding set of output values. Similar to the test phase, the plurality of input values represent the operational state of the aircraft. The plurality of input values include values indicating the state of one or more equipment in aircraft system 220 and / or representing one or more environmental conditions associated with the state of the aircraft. The output values represent the output of aircraft system 220 based on the input values. For example, the output values may correspond to information displayed on a user display and / or may include control parameters corresponding to the control of the aircraft system in the operational state represented by the input values.

[0081] Figure 5Training data 500 is shown as input to a training engine 520, comprising at least another plurality of input values 510a and corresponding at least one set of output values 510b. The training engine 520 can be implemented via appropriate program code in conjunction with at least one memory 110 and at least one processor 120; alternatively, the training engine 520 can be fully deployed as part of another system. In any case, the training engine 520 can process the training data 500 in an appropriate manner to generate a trained classifier 230. In some examples, the training engine 520 can be used to further train an already trained classifier 230 based on the training data 500. For example, the training engine 520 can continue to train and / or retrain the classifier 230 as new training data becomes available and / or as aircraft system requirements change or processes evolve over time. In some examples, the classifier 230 comprises a neural network. In other examples, the classifier 230 implements at least one of a random forest algorithm, a naive Bayes classifier, a support vector machine, a linear regression machine learning algorithm, or any other suitable algorithm or classifier suitable for the functions described herein. For example, a supervised learning algorithm can be used to analyze training data (including input values and corresponding output values) to infer a reasonable learning function that maps input to output. The learning function can be represented by a neural network including an input layer, an output layer, and at least one hidden layer, wherein the nodes of at least one hidden layer region are associated with one or more weights. Training the neural network includes generating and / or updating one or more weights using the input values in the input layer and the output values in the output layer. The learning function can then be tested on a subset of training data that was not used to train the learning function. This can allow the system to be verified before being applied to test data.

[0082] In some examples, the test device 100, 310 can be configured to generate training data 300 based on the identified differences. Generating training data 300 based on the identified differences includes comparing the identified differences with user input indicating a decision regarding the identified differences. For example, after identifying the differences, the generated signal 260 can be used to display information related to the identified differences via the user interface 140. The user can then provide input indicating a decision via the user interface, such as confirming that the identified differences represent an unexpected response of the aircraft system, or alternatively indicating that the classifier 230, 320 is incorrect. In this case, if the classifier is incorrect, training data can be generated and used to train the classifier 230, 320 so that the error does not occur again. Alternatively, if the classifier 230, 320 has correctly identified an unexpected response in the aircraft system, training data can be generated to reinforce the correct identification. The availability of training data 300 impacts the ability of the classifier 230 , 320 to model the expected operation of at least a portion of the aircraft system 220 , 300 and, therefore, to identify differences in the operation of the actual system 220 , 300 and the modeled system.

[0083] In some examples, the test apparatus 100, 310 may be connected to at least one mobile computing device such as Figure 1 The mobile computing device 160 may communicate with the device 160 in the test apparatus 100 , 310 . The mobile computing device 160 may be any suitable combination of hardware and software capable of communicating with the test apparatus 100 , 310 . For example, the mobile computing device may be a smartphone, a tablet computing device, a portable personal computer, or any other suitable device. The test apparatus 100 , 310 may be configured to receive data indicating a decision made by a user of at least one mobile computing device 160 regarding the operation of at least a portion of the aircraft system 220 , 300 . The test apparatus 100 , 310 may be configured to generate training data 300 based on the received data. The mobile computing device 160 may be used by a person, such as a flight crew, for whom at least a portion of the aircraft system 220 , 300 is designed. During system and / or aircraft operation, the flight crew may be prompted, via an appropriate application executed on the mobile computing device, to review one or more operational states of the aircraft and provide feedback regarding the operation of at least a portion of the aircraft system 220 , 300 . Such prompts may be triggered by the test apparatus 100 , 310 to coincide with corresponding events occurring during testing.

[0084] In other examples, the mobile computing device 160 can be used after a test flight, where, after the flight, the flight crew can be prompted by the device to review one or more aspects of the operation of the aircraft system under test. Information generated by the flight crew's decisions can be used to generate training data for training, updating, or modifying the classifiers 230, 320. Thus, the training data can accurately represent the expected and desired operation of the aircraft system from the flight crew's perspective. This can allow the test equipment 100, 310 to identify additional operational conditions in which at least a portion of the aircraft system is not responding and / or operating as expected.

[0085] Figure 6 An example of an aircraft 600 is shown, which may alternatively be an aircraft test stand. Aircraft 600 includes at least one aircraft system, such as hydraulic brakes 610 and a brake control system 620, which are coupled to test equipment 100 and provide input 210 thereto for testing according to the examples herein.

[0086] In some examples, a test device as described herein is disclosed. The test device is used to test at least a portion of one or more aircraft systems included in a test aircraft. The test aircraft may include, for example, Figure 6 The entire aircraft 600 is shown. Alternatively, the test aircraft may comprise a portion or portions of an aircraft, for example, as part of a test bench for testing one or more aircraft systems. The test aircraft includes test equipment that can receive inputs generated in the test aircraft. The inputs generated in the test aircraft can be provided to both the actual aircraft system being tested and the test equipment testing the aircraft system. Outputs from the system and the test equipment can then be generated in the test aircraft and compared.

[0087] In some examples, a test system for testing aircraft control software includes memory, a processor, a control input interface for receiving aircraft control inputs, and a control output interface for receiving aircraft control outputs. The aircraft control software may be part of an aircraft system that includes both software for controlling aircraft operations and aircraft equipment. The test system is configured to receive aircraft control inputs via the control interface. The aircraft control inputs are digital inputs representing the operational status of the aircraft controls. In this example, the aircraft control software can be tested separately. In this case, the inputs and outputs used are digital inputs and outputs, which may represent inputs and outputs from aircraft equipment, such as aircraft controls. The control input interface includes any suitable combination of hardware and software. In some examples, the aircraft control inputs are represented by corresponding multiple input values. The test system processes the aircraft control inputs using a trained model of the expected behavior of the aircraft control software under test to determine the expected control outputs. For example, the trained model may include a classifier trained based on the expected operation of the aircraft control software. Actual control outputs from the aircraft control software are received via the control output interface. The actual control outputs are generated using the aircraft control software and the aircraft control inputs. The control outputs may be represented by corresponding sets of output values. The test system compares the expected control outputs with the actual control outputs to identify any discrepancies therebetween. This may indicate a discrepancy between the operation of the aircraft control software and / or aircraft systems including the aircraft control software and the expected operation of the aircraft control software. The test system then generates a signal indicating the discrepancy between the expected control outputs and the actual control outputs.

[0088] In another example, a test system for testing aircraft control software is provided, wherein the test system includes a memory, a control input interface, a model processing engine, a control output interface, a difference engine, and an output generator.

[0089] The control input interface may receive aircraft control inputs and store the inputs in memory. The aircraft control inputs may include one or more input values that may represent an operational state of one or more components of the aircraft. The model processing engine may be configured to process the aircraft control inputs using a trained model of expected behavior for the aircraft control software under test to determine an expected control output and store the output in memory. The expected model may be generated based on additional aircraft control software or using one or more computer models trained using data representing the expected behavior.

[0090] The control output interface may receive actual aircraft control outputs and store the outputs in memory. The actual aircraft control outputs are generated by operating the aircraft control software using the aircraft control inputs.

[0091] The difference engine is used to compare the expected control output with the actual control output to identify any differences therebetween. For example, the difference engine may read the actual control output and the expected control output from a non-volatile memory such as a ROM, may store the output in a temporary memory, and may process the output using a processor to determine any differences.

[0092] The output generator may be used to generate a signal indicative of differences identified by the difference engine.

[0093] It should be noted that the term "or" used herein is to be interpreted as meaning "and / or" unless explicitly stated otherwise.

Claims

1. A test device for testing at least a portion of an aircraft system, the device comprising at least one memory and at least one processor, the test device including program code stored in the at least one memory and configured to control the test device to: obtaining first data, the first data comprising a plurality of input values representing an operational state of the aircraft; processing the plurality of input values using a trained classifier to generate a first set of output values, the trained classifier configured to model operation of at least a portion of an aircraft system; obtaining second data comprising a second set of output values generated by operating at least a portion of the aircraft system based on the plurality of input values; processing the first set of output values and the second set of output values to identify any differences between operation of at least a portion of the aircraft system and operation modeled by the trained classifier; as well as A signal is generated indicative of the identified difference.

2. The test device according to claim 1, wherein: The test equipment is configured to: obtaining training data representing expected operation of at least a portion of the aircraft system for at least one operational state, the training data comprising at least another plurality of input values and corresponding at least one set of output values; as well as The classifier is trained using the training data.

3. The test device according to claim 2, wherein: The test apparatus is in communication with at least one mobile computing device and is configured to receive data indicative of a decision made by a user of the at least one mobile computing device regarding operation of at least a portion of the aircraft system, and wherein the test apparatus is configured to generate training data based on the received data.

4. The test device according to claim 2, wherein: The testing device is configured to generate training data based on the identified differences.

5. The test device according to claim 4, wherein: Generating training data based on the identified differences includes comparing the identified differences to user input indicating a decision regarding the identified differences.

6. The test device according to any one of claims 1 to 5, wherein: The plurality of input values represent at least one of the following: an operational status of at least one component associated with the aircraft system; and An output from at least one sensor configured to sense a corresponding environmental condition.

7. The testing device according to claim 6, wherein: The operational state of at least one component associated with the aircraft system includes a control input, the control input including at least one of: Indicates the value of the control input; a value representing the rate of change of the control input; and A value representing the difference between the control input and another control input.

8. The test device according to any one of claims 1 to 5, wherein: At least a portion of the aircraft system is a computer program for controlling at least a portion of the aircraft.

9. The test device according to any one of claims 1 to 5, wherein: At least a portion of the aircraft system includes a combination of aircraft equipment and a computer program for operating the aircraft.

10. The test device according to any one of claims 1 to 5, wherein At least a portion of the aircraft system includes aircraft equipment including at least one sensor for generating at least one value in the second set of output values.

11. The test device according to any one of claims 1 to 5, wherein: The at least a portion of the aircraft system includes at least one of: an avionics system, a flight control system, a brake control system, an instrument and recording system, a landing gear control system, and a fuel system.

12. A method of testing at least a portion of an aircraft system, the method comprising: obtaining first data, the first data comprising a plurality of input values representing an operational state of the aircraft; processing the plurality of input values using a trained classifier to generate a first set of output values, the trained classifier configured to model operation of at least a portion of an aircraft system; generating second data comprising a second set of output values by operating at least a portion of the aircraft system based on the plurality of input values; processing the first set of output values and the second set of output values to identify any differences between operation of at least a portion of the aircraft system and operation modeled by the trained classifier; as well as A signal is generated indicative of the identified difference.

13. The method according to claim 12, wherein: The method comprises: obtaining training data representing expected operation of at least a portion of the aircraft system for at least one operational state; and The classifier is trained using the training data.

14. The method according to claim 13, wherein The training data is generated from the further aircraft system during operation of the further aircraft system.

15. The method according to claim 13, wherein The training data is generated based on input from a user indicating a decision regarding the identified differences.

16. The method according to any one of claims 12 to 15, wherein The method includes generating further training data based on the identified differences, and training the classifier using the further training data.

17. The method according to any one of claims 12 to 15, wherein The method includes modifying at least a portion of the aircraft system based on the identified difference.

18. An aircraft comprising one or more aircraft systems, wherein: At least one of the one or more aircraft systems is tested by a method according to any one of claims 12 to 17 .

19. A test aircraft comprising one or more aircraft systems and a test device according to any one of claims 1 to 11 for testing at least a portion of the one or more aircraft systems included in the test aircraft.

20. A test system for testing aircraft control software, the system comprising a memory, a processor, a control input interface for receiving aircraft control input, and a control output interface for receiving aircraft control output, the test system being configured to: receiving aircraft control input via the control input interface; processing the aircraft control input using a trained model of expected behavior of the aircraft control software under test to determine an expected control output; receiving actual control output from the aircraft control software via the control output interface; comparing the expected control output to the actual control output to identify any differences therebetween; and A signal is generated indicative of a difference between the expected control output and the actual control output.

21. A test system for testing aircraft control software, the system comprising: Memory; a control input interface for receiving aircraft control input and storing the input in the memory; a model processing engine for processing the aircraft control inputs using a trained model of expected behavior of the aircraft control software under test to determine expected control outputs and storing the outputs in the memory; a control output interface for receiving actual aircraft control outputs and storing said outputs in said memory; a difference engine for comparing the expected control output with the actual control output to identify any differences therebetween; as well as An output generator is configured to generate a signal indicative of differences identified by the difference engine.

Images