Portable system and method for monitoring maintenance events for an aircraft

Abstract

To provide a portable system and method for monitoring maintenance events for an aircraft.SOLUTION: In the system 100, the portable monitoring system 102 includes a control unit 104 having one or more processors that access one or more technical specifications associated with a maintenance event of an aircraft. The maintenance event includes one or more steps to be completed by the operator to complete the maintenance event. The portable monitoring system also includes one or more sensors 108 that acquire data during a maintenance event of the aircraft. The data is associated with one or more of an activity of an operator and / or one or more characteristics of an aircraft during a maintenance event and includes two or more different types of modalities. The control unit monitors the maintenance event based in part on the data acquired by the sensor during the maintenance event and at least one of the technical specifications.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 663,446 (filed June 24, 2024), the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THE DISCLOSURE

[0002] Embodiments of the present disclosure generally relate to portable or wearable monitoring systems and methods for monitoring aircraft maintenance or inspection events. [Background technology]

[0003]

[0003] Ensuring that aircraft maintenance work is performed in accordance with specific manuals, such as Aircraft Maintenance Manuals (AMMs), is crucial to aviation efficiency, safety, and the fulfillment of regulatory obligations. Mechanics must strictly follow predefined procedures in specific manuals approved by regulatory authorities when performing aircraft maintenance and repairs. Deviations from the steps outlined in the manuals can pose safety and / or quality risks. Therefore, it is important to ensure adherence to maintenance procedures and audit maintenance actions and activities to ensure safety and avoid costly rework.

[0004]

[0004] In some cases, the quality of aircraft maintenance work depends on peer monitoring and post-maintenance functional testing. Human monitors are expensive, difficult to scale, and prone to error. Furthermore, the results of post-maintenance testing do not guarantee that the work was performed in accordance with regulatory approved procedures and steps. For example, if maintenance work is not performed correctly, a system may function normally for a short period of time and then fail. Furthermore, post-maintenance verification can take time to provide feedback. Often, by the time an issue is identified, damage has already occurred.

[0005] In other instances, various solutions for computer vision verification have been developed. These can be single-photo (e.g., static) or video (e.g., dynamic) solutions. One challenge with these verification solutions is the amount of training they require. For example, a single-mode model may have additional experience or learning as the basis for training, in the same spirit as showing a person still images or videos and labeling some of these images and / or videos as incorrect without context, explanation, or audio. A person relying on such labels would likely take a significant amount of time to discern core patterns, such as what are distinct events or processes, what are acceptable sequences, and what conditions and / or states should be distinguished.

[0006]

[0006] A single-mode model may be tolerant to a finite number of operations or a finite number of distinguishable states (e.g., there is only one correct way to stack boxes and one incorrect way to stack boxes). However, for highly complex and diverse operations, such as replacing aircraft landing gear or assembling aircraft wings, the training and testing costs become prohibitive because every task must be trained. There are thousands of procedures, such as fault isolation procedures, maintenance manual procedures, structural repair procedures, etc., and a single-mode computer vision model may require dedicated training and testing for every single one to establish coverage. Summary of the Invention

[0007]

[0007] What is needed is a portable, wearable, video-enabled multimodal artificial intelligence system that is empowered by the technical specifications and maintenance documentation used by aircraft maintenance personnel and can be used in real-time procedure compliance and / or verification monitoring or other support application cases.

[0008] With these needs in mind, certain embodiments of the present disclosure provide a portable monitoring system including a control unit having one or more processors that can access one or more technical specifications associated with an aircraft maintenance event. The maintenance event may include one or more steps to be completed by an operator to complete the maintenance event. The portable monitoring system includes one or more sensors that acquire data during the aircraft maintenance event. The data is associated with one or more activities of the operator during the maintenance event and / or one or more characteristics of the aircraft. The data includes two or more different types of modalities. The control unit monitors the maintenance event based in part on the data acquired by the sensors during the maintenance event and at least one of the technical specifications.

[0009] In at least one embodiment, the two or more different types of modalities may include video, text, audio, images, and / or touch. In another embodiment, the control unit may verify completion of one or more steps of a maintenance event by an operator based on each of the two or more different types of modalities of data.

[0010] In at least one embodiment, the portable monitoring system may include an output device operably coupled to the control unit. The control unit may communicate with an operator during a maintenance event via the output device. In at least one embodiment, the control unit may communicate one or more instructions to the operator for the operator to complete one or more of the one or more steps of the maintenance event.

[0011] In at least one embodiment, the control unit may confirm completion of one or more steps of the maintenance event. In at least one embodiment, the control unit may identify at least one of the one or more steps of the maintenance event as having been completed incorrectly by comparing against a completion threshold of at least one technical specification. In response to identifying at least one step as having been completed incorrectly, the control unit may communicate a notification to an operator via an output device. The notification may include a recommendation of corrective action for the operator to complete to correct the at least one incorrectly completed step. In at least one embodiment, the incorrectly completed step may be a first step. In response to the operator having completed the first step incorrectly, the control unit may communicate a notification to the operator before the operator begins a subsequent second step.

[0012] In at least one embodiment, one or more sensors and a control unit may be operably coupled to a body that may be worn by and / or coupled to an operator during a maintenance event.

[0013] In at least one embodiment, the control unit may be an artificial intelligence (AI) or machine learning system.

[0014] Certain embodiments of the present disclosure provide a method that includes accessing one or more technical specifications associated with one or more maintenance events of the aircraft, and acquiring data using one or more sensors of a portable monitoring system during the maintenance events of the aircraft. The data is associated with one or more activities of an operator during the maintenance events or one or more characteristics of the aircraft. The data includes two or more different types of modalities. The maintenance events are monitored based at least in part on the data acquired by the one or more sensors during the maintenance events and at least one of the one or more technical specifications.

[0015] Certain embodiments of the present disclosure provide a wearable or portable monitoring system including a body worn by and / or coupled to an operator during a maintenance or inspection event on an aircraft. A control unit including one or more processors is operably coupled to the body. The control unit wirelessly accesses one or more technical specifications associated with the aircraft and / or the aircraft maintenance or inspection event. One or more steps of the maintenance or inspection event are included in at least one of the one or more technical specifications. One or more sensors are operably coupled to the body and acquire data during the maintenance or inspection event. The data is associated with one or more of the operator's activities during the maintenance or inspection event or one or more characteristics of the aircraft. The data includes two or more different types of modalities. The control unit monitors the maintenance or inspection event based at least in part on the data acquired by the sensors during the maintenance or inspection event and at least one of the technical specifications. [Brief explanation of the drawings]

[0016] [Figure 1]

[0016] A schematic block diagram of a system according to one embodiment of the present disclosure is shown. [Figure 2]

[0017] 1 illustrates several examples of a portable monitoring system according to one embodiment of the present disclosure. [Figure 3]

[0018] 1 shows a flowchart of a method according to one embodiment of the present disclosure. [Figure 4]

[0019] 1 illustrates a perspective front view of an aircraft according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017]

[0020] The foregoing summary, as well as the following detailed description of specific embodiments, will be better understood when read in conjunction with the accompanying drawings. As used herein, the use of the singular form "a" or "an" preceding an element or step does not necessarily exclude a plurality of such elements or steps. Furthermore, references to "one embodiment" are not intended to be interpreted as excluding the existence of additional embodiments that incorporate features described herein. Furthermore, embodiments that "comprising" or "having" one or more elements having certain conditions may include additional elements that do not have those conditions (unless expressly stated otherwise).

[0018]

[0021] The systems and methods described herein provide a portable and / or wearable video-enabled monitoring system that includes a processor configured to utilize a multimodal artificial intelligence system to automatically audit a maintainer's work against step-by-step procedures and related requirements defined in technical specifications, such as aircraft maintenance manuals. The portable monitoring system may include pre-trained multimodal models (e.g., text and vision), where a neural net may associate semantic patterns in images and / or video and text. The monitoring system may also be connected to and / or have access to technical publications to establish context. For example, the portable monitoring system may replace text knowledge from technical specifications with video knowledge, thereby minimizing the amount of video training and testing required and maximizing generalizability across multiple maintenance activities and / or operations compared to single-modal models.

[0019]

[0022] The portable monitoring system receives, acquires, accesses, etc., temporal and multimodal signals from video, audio, text, etc. to determine whether an activity has been performed by an operator based on coherent evidence indicating a continuous process characterized by a starting state, transitional interventions, and a modified ending state. If a step is skipped or not performed correctly by the operator, the portable monitoring system may issue real-time notifications or alerts to assist the operator in correcting the deviation as needed.

[0020]

[0023] 1 shows a block diagram of a system 100 according to one embodiment of the present disclosure. The system 100 shown in FIG. 1 is merely exemplary and non-limiting.

[0021]

[0024] System 100 includes a portable monitoring system 102 that includes a control unit 104 having one or more processors. In at least one embodiment, control unit 104 is an artificial intelligence or machine learning system. In at least one embodiment, portable monitoring system 102 may be worn, coupled, carried, etc., by an operator, such as a maintenance operator. For example, portable monitoring system 102 may be moved from one location to another, may be a wireless device, etc. In at least one embodiment, portable monitoring system 102 may be referred to as a portable wearable system, a wearable monitoring device, a smart wearable device, etc.

[0022]

[0025] 2 illustrates several examples of portable monitoring systems according to several embodiments of the present disclosure. For example, the portable monitoring system 102 may include a body having a structure, shape, and / or style that may be similar to a headset 202A, a virtual reality headset 202B, a body camera system 202C, a drone 202D, etc. For example, the portable monitoring system 102 may have one or more features and / or elements that enable an operator to wear the portable monitoring system 102, such as during an inspection and / or maintenance event of the aircraft 101, systems of the aircraft 101 (e.g., landing gear, propulsion system, etc.), etc. The various embodiments of the portable monitoring systems 202A-D illustrated in FIG. 2 are exemplary only and, thus, do not limit the types of wearable and / or portable monitoring systems that may be used to monitor an operator and / or aircraft during an inspection or maintenance event.

[0023]

[0026] Returning to Figure 1, portable monitoring system 102 includes power source 106, which may represent a battery or alternative energy source that may provide power to portable monitoring system 102. Portable monitoring system 102 also includes one or more input / output devices 110 (denoted in Figure 1 as "I / O device(s)"). I / O device 110 may include and / or represent a display (e.g., an electronic monitor, television, touch screen, etc.), a keyboard, a headset, a microphone, an electronic mouse, a stylus, etc. I / O device 110 may be configured to display visual graphics, video, text, etc.

[0024]

[0027] The portable monitoring system 102 may communicate with the remote database 120 via a communication system 112. The communication system 112 may include and / or represent one or more antennas, transceivers, radios, etc., which enable wired and / or wireless communication between the portable monitoring systems 102, between the portable monitoring systems and the aircraft 101, between the portable monitoring systems and the remote database 120, between the portable monitoring systems and the storage facility 114 (which may store, for example, replacement materials 116, inventory 118, tools, etc.), etc.

[0025]

[0028] In at least one embodiment, remote database 120 may represent and / or include a data storage unit, such as a memory, that stores and / or accesses technical publications or specifications 122, past maintenance records 124, etc. In one embodiment, remote database 120 may be a physical location or, alternatively, a cloud-based data storage system. Technical specifications 122 may include and / or represent prescribed procedures (e.g., maintenance procedures, assembly procedures, etc.), manuals (e.g., manuals that may be approved by regulatory authorities when performing maintenance and / or repair on an aircraft), etc. For example, technical specifications 122 may include, without limitation, a particular original equipment manufacturer's (OEM) Fault Isolation Manual (FIM), Aircraft Maintenance Manual (AMM), Structural Repair Manual (SRM), Aircraft Illustrated Parts Catalog (AIPC), etc.

[0026]

[0029] The portable monitoring system 102 includes one or more sensors 108 that may sense or otherwise detect information. In one or more embodiments, the sensors may be and / or include a camera (e.g., a camera that captures still and / or video images), a microphone, a motion sensor, a heat sensor, a vibration sensor, a pressure sensor, etc. In at least one embodiment, one or more of the sensors 108 may include and / or represent a global positioning system sensor, a radar sensor, etc. The sensors 108 may be embedded within a portion of the body of the portable monitoring system 102, operably coupled to an exterior portion of the body of the portable monitoring system 102, etc.

[0027]

[0030] In at least one embodiment, sensors 108 may detect information from an operator performing maintenance on aircraft 101, information about the aircraft such as the portion of the aircraft being maintained, ambient information about the area surrounding the aircraft (e.g., ambient temperature, pressure, humidity, etc.), information about the tools or equipment the operator is using to perform the maintenance (e.g., drills, hammers, etc.), information about the maintenance activity being completed (e.g., draining fluids, depressurizing a system, etc.), etc. For example, sensors 108 may sense or detect characteristics associated with various systems / devices of aircraft 101 and characteristics about the maintenance being completed.

[0028]

[0031] 3 illustrates a flowchart 300 of a method according to one embodiment of the present disclosure. Referring to FIG. 1, an individual or operator may wear and / or use a portable monitoring system 102 for a maintenance event, inspection event, and / or repair event, etc., on an aircraft 101. At 302, one or more technical specifications 122 may be accessed and / or received by the control unit 104 from a remote database 120. The technical specifications 122 may be associated with maintenance work to be completed, systems and / or devices of the aircraft 101 being maintained, inspected, and / or repaired, past maintenance records for the aircraft 101, etc.

[0029]

[0032] In at least one embodiment, a maintenance event may include one or more procedural steps that an operator is required to complete in order to complete the maintenance event. In one embodiment, two or more of the steps may be sequential, whereby the operator cannot move to a second step until the first step has been completed.

[0030]

[0033] At 304, while the operator is performing the maintenance event, data may be acquired by one or more sensors 108. For example, sensors 108 may represent cameras that may capture images of the maintenance task being or has been completed, microphones that capture sounds associated with the task being completed (such as sounds of tools being used, voices from the operator), etc. As one example, sensors 108 may include sensing devices that may sense characteristics of the aircraft 101 (e.g., pressure measurements, volumetric information, force data, fluid levels, fluid temperatures, etc.).

[0031]

[0034] At 306, the control unit 104 monitors the maintenance event and the work completed by the operator during the maintenance event in terms of at least one of the technical specifications 122 based on the data acquired by the sensors 108. For example, the control unit 104 may monitor the work being completed by the operator to ensure that the operator is complying with one or more instructions of the procedure in the technical specifications 122.

[0032]

[0035] As described herein, the control unit 104 may monitor work being completed by the maintenance operator based on various types of modalities of data and / or information acquired by the sensors 108 in light of or in view of the technical specifications 122. Furthermore, the various types of modalities may be from one or more various sources (e.g., the operator, the tools being used, the portion of the aircraft being maintained or repaired, etc.). For example, the various types of modalities may include and / or represent video (e.g., video details of the maintenance work completed by the operator), text (e.g., predefined maintenance activities and / or steps specified in the technical specifications 122), audio (e.g., mechanic narration, audio status signals from troubleshooting and / or testing, noise from tools or noises resulting from maintenance activities, similar pressure reductions, fluid releases, etc.), images (e.g., tool images, part drawings and / or diagrams according to the technical specifications 122, past maintenance records, etc.), or data associated with one or more characteristics of the aircraft 101 (e.g., pressure measurements, volume information, force data, fluid levels, fluid temperatures, etc.). Optionally, one or more additional types of modalities may be used and / or relied upon by the control unit 104 to monitor the completion of the task being completed.

[0033]

[0036] In at least one embodiment, the control unit 104 may utilize various types of modalities in monitoring the completion of maintenance work being completed, such as through artificial intelligence. For example, the artificial intelligence control unit 104 may receive data in various forms (e.g., video, text, audio, images, tactile, etc.) and / or from various sources (e.g., the operator, the aircraft 101, the technical specifications 122, etc.). The various types of modalities may enable the control unit 104 to better monitor whether the maintenance work being completed is accurate, whether the completed work is sufficient or insufficient, whether the work meets regulatory standards and / or threshold requirements, etc., compared to a control unit 104 relying on a single type of modality.

[0034]

[0037] In at least one embodiment, the control unit 104 of the portable monitoring system 102 may communicate with an operator, such as via the I / O device(s) 110, before and / or during a maintenance event. For example, the operator may select a technical specification 122 to be followed and / or request assistance from the control unit 104 (e.g., by actuating a touchscreen or alternative button, verbally requesting a microphone, etc.). The request for assistance may be for a procedural step of the maintenance event according to the technical specification 122. As one example, the control unit 104 may display a portion of the technical specification 122 to the operator via a display device (e.g., displaying a portion of text, displaying video and / or images associated with a step, etc.). In another embodiment, the I / O device may be a speaker, and the control unit 104 may provide the procedural steps to the operator audibly.

[0035]

[0038] In at least one embodiment, a maintenance operation may require an operator to replace one or more structures and / or materials from the aircraft 101, may require the operator to use particular tools to complete the operation, etc. The control unit 104 may communicate with the storage facility 114 to understand whether the structures and / or materials needed for the maintenance or repair operation are available, whether particular tools are available, etc. In at least one embodiment, the control unit 104 may communicate with the operator to indicate whether replacement materials and / or tools are available, an estimated time when the materials will be available, and where the materials and / or tools may be located (e.g., within the storage facility 114, outside the storage facility, etc.).

[0036]

[0039] At 308, a determination is made whether the steps of the maintenance event were completed correctly or incorrectly compared to completion thresholds of the technical specifications 122, regulatory criteria, etc. For example, the control unit 104 may determine and / or score the tasks completed by the operator and determine whether the completed tasks meet and / or exceed regulatory criteria, completion thresholds, etc. Additionally, the control unit 104 may determine and / or score the tasks completed by the operator in real time.

[0037]

[0040] For example, the portable monitoring system 102 may monitor and / or audit a process such as the removal and replacement of landing gear of the aircraft 101, the assembly of aircraft wings, etc. According to the corresponding technical specification(s) 122 associated with the removal and replacement of the landing gear, the process requires multiple activities and / or steps that, when completed, result in a change of state (e.g., the replacement of the landing gear). The control unit 104 monitors a first step completed by an operator according to the technical specification and determines whether the first step was completed correctly or incorrectly compared to a completion threshold of the technical specification. As another example, the control unit 104 determines a completion score and / or a judgment of the task completed by the operator in the first step and determines whether the completion score meets the completion score threshold according to the technical specification 122.

[0038]

[0041] The various types of modalities of data sensed by sensors 108 and / or acquired by control unit 104 may be used by control unit 104 to determine whether the activity(ies) and / or procedure(s) completed by the operator meet and / or exceed completion threshold requirements for the corresponding step or activity. For example, rather than determining whether a first step was completed correctly based on a single static condition (e.g., new landing gear was replaced), control unit 104 relies on multiple modalities associated with the activities and / or processes the operator undertook in completing the first step (e.g., how the operator removed the landing gear, how the operator installed the new landing gear, etc.).

[0039]

[0042] Rather than determining whether a step or activity was completed correctly based on a single condition (e.g., new landing gear was replaced), the control unit 104 also examines specific activities the operator performed in replacing the landing gear. For example, the portable monitoring system 102 may audit a process by which the operator completes a first step. The system 102 may determine whether a portion of the process was performed satisfactorily, such as by comparing the portion of the process to a completion threshold or criterion according to the technical specification 122, by which the operator completed the first step. For example, the new landing gear may have been installed correctly, but the operator may have completed the portion of the process in a manner that did not meet the completion threshold or criterion, thereby compromising the installation status of the landing gear.

[0040]

[0043] If the control unit 104 determines that a step of the maintenance event was completed incorrectly, did not meet a completion threshold, etc., the method flow proceeds to 310. At 310, the operator may be notified that a step was completed incorrectly, that a portion of the process was performed incorrectly, or fell below a criteria threshold, etc., that a portion of the process caused the operator to complete a step. For example, the control unit 104 may communicate with the operator via the I / O device 110, e.g., sound an alarm, display a notification on a display screen, vibrate, etc.

[0041]

[0044] In at least one embodiment, the notification may include a corrective action recommendation or instruction for the operator to correct a step that was completed incorrectly, did not meet a completion criterion or threshold, etc. In at least one embodiment, the operator may be required to correct a step that was completed incorrectly or did not meet a completion threshold before the operator can proceed to the next step in the maintenance procedure. For example, the operator may be required to correct a first step before proceeding to a subsequent second step in the procedure.

[0042]

[0045] In at least one embodiment, the control unit 104 can also communicate alerts or notifications to the maintenance workstation and / or to a supervisor. For example, the notification may indicate that the operator completed a portion of a step incorrectly or below a threshold or criterion. The operator may need to indicate to the supervisor that a first step has been corrected before the supervisor will approve allowing the operator to proceed to the next subsequent step.

[0043]

[0046] In at least one embodiment, the control unit 104 may continuously receive live video associated with the task being completed by the operator and may detect in real time whether any of the maintenance steps have been missed, whether the steps have been performed correctly to meet detailed criteria (e.g., specific pressures, volumes, torques, etc.). If the control unit 104 detects any non-conformances within the task (based on multiple modalities of data, technical specifications, etc.), the portable monitoring system 102 may alert the operator to the oversight, provide assistance in correcting the situation, etc. For example, real-time alerts and feedback by the portable monitoring system 102 may eliminate mistakes made by the operator compared to alternative monitoring systems that do not provide real-time feedback, do not rely on various types of modalities of data to inspect completed tasks, etc.

[0044]

[0047] Returning to step 308, if the steps of the maintenance event are completed correctly or if a completion threshold or criterion is met, the method flow proceeds to 312. At 312, a determination is made whether the maintenance event is complete. For example, the control unit 104 may determine whether all of the steps of a particular maintenance or repair event are completed based on one or more technical specifications 122. If the maintenance event is not complete, the method flow proceeds back to 306 and the portable monitoring system 102 continues to monitor and / or audit maintenance work being completed by the operator. Alternatively, if the maintenance event is complete, the method flow proceeds to 314, where the maintenance event ends and the portable monitoring device terminates or stops monitoring the event.

[0045]

[0048] As used herein, terms such as "control unit," "central processing unit," "CPU," "computer," and the like may include any processor-based or microprocessor-based system, including systems using microcontrollers, reduced instruction set computers (RISC), application specific integrated circuits (ASIC), logic circuits, and any other circuits or processors, including hardware, software, or a combination thereof, capable of performing the functions described herein. The above examples are illustrative only and are thus not intended to limit in any way the definition and / or meaning of the above terms. For example, the control unit 104 of the portable monitoring system 102 may be or include one or more processors configured to control one or more operations as described herein.

[0046]

[0049] The control unit 104 of the portable monitoring system 102 is configured to execute sets of instructions stored in one or more data storage units or elements (such as one or more memories) to process data. For example, the control unit 104 may include or be coupled to one or more memories. The data storage units may also store data or other information as desired or needed. The data storage units may take the form of an information source or a physical memory element within a processing machine.

[0047]

[0050] The set of instructions may include various commands that instruct the control unit 104 as a processing machine to perform particular operations (e.g., methods and processes of various embodiments of the subject matter described herein). The set of instructions may take the form of a software program. The software may take various forms such as system software or application software. Furthermore, the software may take the form of a collection of separate programs, a program subset within a larger program, or a portion of a program. The software may also include modular programming in the form of object-oriented programming. The processing of input data by the processing machine may be in response to user commands, in response to results of previous processing, or in response to a request made by another processing machine.

[0048]

[0051] The diagrams of several examples herein may depict one or more control or processing units, such as the control unit 104 of the portable monitoring system 102. It should be understood that this processing or control unit may represent a circuit, circuitry, or portion thereof, that may be implemented as hardware having associated instructions (e.g., software stored on a tangible, non-transitory computer-readable storage medium, such as a computer hard drive, ROM, or RAM) that perform the operations described herein. The hardware may include state machine circuitry hardwired to perform the functions described herein. Optionally, the hardware may include electronic circuitry including and / or connected to one or more logic-based devices, such as a microprocessor, processor, controller, etc. Optionally, the control unit 104 may represent processing circuitry, such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), microprocessor(s), etc. The circuitry in various examples may be configured to execute one or more algorithms to perform the functions described herein. Such one or more algorithms, whether or not explicitly identified in a flowchart or method, may include aspects of the embodiments disclosed herein.

[0049]

[0052] As used herein, the terms "software" and "firmware" are used interchangeably and include any computer program stored in a data storage unit (e.g., one or more memories) for execution by a computer, including RAM memory, ROM memory, EPROM memory, EEPROM memory, and non-volatile RAM (NVRAM) memory. The types of data storage units listed above are merely exemplary and thus not limiting as to the types of memory that may be used for storing computer programs.

[0050]

[0053] 1-3 , embodiments of the present disclosure provide methods and systems that enable a computing device to quickly and efficiently analyze large amounts of data. For example, the control unit 104 may receive and analyze hundreds, thousands, millions, or more data and / or records, which may be in the form of various types of modalities. The control unit 104 may also access and / or receive hundreds, thousands, millions, or more technical specifications or publications and analyze data associated with real-time maintenance work being completed based on the technical specifications. Thus, large amounts of data that would be indistinguishable to a human being are tracked and analyzed. As described herein, vast amounts of data are efficiently organized and / or analyzed by the control unit 104. The control unit 104 analyzes the data in a relatively short time to quickly and efficiently code, decode, and execute data link messages. A human would not be able to efficiently analyze such vast amounts of data in such a short time. Therefore, embodiments of the present disclosure provide improved and efficient functionality and performance that vastly outperforms a human analyzing vast amounts of data.

[0051]

[0054] In at least one embodiment, components of the system 100, such as the control unit 104 of the portable monitoring system 102, provide and / or enable the computer system to operate as a dedicated computer system for monitoring and / or auditing maintenance, inspection, and / or repair work completed by work personnel. The control unit 104 of the portable monitoring system 102 improves upon standard computing devices by monitoring such work completed by operators via multiple types of modalities and automatically communicating with individuals (e.g., maintenance operators, supervisors, aircraft operators, etc.) in an efficient and effective manner.

[0052]

[0055] In at least one embodiment, the control unit 104 of the portable monitoring system 102 uses machine learning algorithms. The machine learning algorithms automatically analyze the sensed data and / or various types of modalities associated with the technical specifications 122 or maintenance records 124 to automatically audit the mechanic's work in real time against the step-by-step procedures and related requirements defined in the technical specifications. In at least one embodiment, all or a portion of the systems and methods described herein may be or otherwise include an artificial intelligence (AI) or machine learning system capable of automatically performing the operations of the methods described herein. For example, the processor of the control unit 104 may be an artificial intelligence or machine learning system. These types of systems may be trained externally from information and / or may be self-trained to iteratively improve the accuracy with which data is analyzed. Over time, these systems improve by identifying such information with increasing accuracy and speed, thereby significantly reducing the likelihood of any potential errors. The AI ​​or machine learning systems described herein may include techniques enabled by adaptive predictive capabilities. The techniques exhibit at least some degree of autonomous learning to automate and / or enhance pattern detection (e.g., recognizing irregularities or regularities in data), customization (e.g., creating or modifying rules to optimize record matching), etc. The systems may be trained and retrained using feedback from one or more prior analyses of data, ensemble data, and / or other such data.

[0053]

[0056] Based on this feedback, the system may be trained by adjusting one or more parameters, weights, rules, criteria, etc. used in the same analysis. This process may be performed using data or ensemble data instead of training data and may be repeated multiple times to iteratively improve the monitoring and auditing performed by the control unit 104. Training minimizes conflicts and interference by implementing an iterative training algorithm. In that case, the system is retrained with an updated set of data (e.g., data received during multiple different maintenance events of a general aircraft 101 in a general system of different aircraft, different maintenance events by the same operator, etc.) based on feedback examined prior to the system's most recent training. This provides a robust analytical model that can better identify situational information in a cost-effective and cost-efficient manner.

[0054]

[0057] For example, the system can be adapted to various use cases depending on the complexity of the overall task and steps of the work being completed. Detecting maintenance steps can involve sequential sampling of image frames from a video. The sampling frequency can be an adjustable parameter that can be tailored to different cases. Additionally, the number of time frames and / or the number of overlapping frames that can be passed to the AI ​​system can be optimized according to the particular use case to achieve the best model detection performance.

[0055]

[0058] FIG. 4 illustrates a perspective front view of an aircraft 101 according to one embodiment of the present disclosure. The aircraft 101 includes a propulsion system 412 including, for example, engines 414. Optionally, the propulsion system 412 may include more engines 414 than shown. The engines 414 are supported by wings 416 of the aircraft 101. In other embodiments, the engines 414 may be supported by a fuselage 418 and / or a tail section 420. The tail section 420 may also support a horizontal stabilizer 422 and a vertical stabilizer 424. The fuselage 418 of the aircraft 101 defines an interior cabin, which may include a flight deck or cockpit, one or more work areas (e.g., a galley, a crew baggage area, etc.), one or more passenger areas (e.g., first class, business class, and economy class), one or more restrooms, etc. FIG. 4 illustrates an example of an aircraft 101. It should be understood that aircraft 101 may be sized, shaped, and configured differently than that shown in FIG.

[0056]

[0059] Optionally, embodiments of the present disclosure may be used with various other types of vehicles, such as automobiles, trains, ships, spacecraft, etc. Also, optionally, embodiments of the present disclosure may be used with various other devices, systems, components, etc. other than vehicles. For example, embodiments of the present disclosure may be used with consumer electronics, industrial power systems and / or equipment, etc.

[0057]

[0060] Furthermore, the present disclosure includes embodiments according to the following clauses.

[0058]

[0061] Article 1. 1. A portable monitoring system comprising:

[0062] a control unit including one or more processors configured to access one or more technical specifications associated with a maintenance event for an aircraft, the maintenance event including one or more steps configured to be completed by an operator to complete the maintenance event; and

[0063] one or more sensors configured to acquire data during the maintenance event of the aircraft, the data being associated with one or more of the operator's activities during the maintenance event or one or more characteristics of the aircraft, the data comprising two or more different types of modalities;

[0064] the control unit is configured to monitor the maintenance event based at least in part on the data acquired by the one or more sensors during the maintenance event and at least one of the one or more technical specifications.

[0059]

[0065] Article 2. 2. The portable monitoring system of claim 1, wherein the two or more different types of modalities include one or more of video, text, audio, image, or touch.

[0060]

[0066] Article 3. 3. The portable monitoring system of claim 1 or 2, wherein the control unit is configured to inspect the completion of the one or more steps of the maintenance event by the operator based on each of the two or more different types of modalities of the data.

[0061]

[0067] Article 4. 4. The portable monitoring system of any one of clauses 1 to 3, further comprising an output device operably coupled to the control unit, the control unit configured to communicate with the operator during the maintenance event via the output device.

[0062]

[0068] Article 5. 5. The portable monitoring system of claim 4, wherein the control unit is configured to communicate one or more instructions to the operator for the operator to complete one or more of the one or more steps of the maintenance event.

[0063]

[0069] Article 6. 6. The portable monitoring system of any one of clauses 1 to 5, wherein the control unit is configured to confirm the completion of the one or more steps of the maintenance event.

[0064]

[0070] Article 7. 7. The portable monitoring system of any one of clauses 1 to 6, wherein the control unit is configured to identify at least one of the one or more steps of the maintenance event as having been completed incorrectly by comparing the at least one completion threshold of the one or more technical specifications.

[0065]

[0071] Article 8. 8. The portable monitoring system of clause 7, further comprising an output device operably coupled to the control unit, the control unit configured to communicate a notification to the operator in response to identifying at least one of the one or more steps as having been completed in error.

[0066]

[0072] Article 9. 9. The portable monitoring system of claim 8, wherein the notification is configured to include a recommendation of corrective action for the operator to complete to correct the at least one of the one or more steps that was completed in error.

[0067]

[0073] Article 10. The portable monitoring system of clause 8, wherein the at least one of the one or more steps that is erroneously completed is a first step, and the control unit is configured to communicate the notification to the operator in response to the operator erroneously completing the first step before the operator starts a subsequent second step.

[0068]

[0074] Article 11. A portable monitoring system as described in any one of clauses 1 to 10, wherein the one or more sensors and the control unit are operably coupled to a body, and the body is configured to be one or more of worn by or coupled to the operator during the maintenance event.

[0069]

[0075] Article 12. 12. The portable monitoring system of any one of clauses 1 to 11, wherein the control unit is an artificial intelligence (AI) or machine learning system.

[0070]

[0076] Article 13.

[0077] accessing one or more technical specifications associated with one or more maintenance events of the aircraft;

[0078] acquiring data using one or more sensors of a portable monitoring system during the maintenance event of the aircraft, the data being associated with one or more of operator activities during the maintenance event or one or more characteristics of the aircraft, the data including two or more different types of modalities; and

[0079] monitoring the maintenance event based at least in part on the data acquired by the one or more sensors during the maintenance event and at least one of the one or more technical specifications.

[0071]

[0080] Article 14. 14. The method of clause 13, further comprising verifying completion of one or more steps of the maintenance event based on each of the two or more different types of modalities of the data.

[0072]

[0081] Article 15.

[0082] comparing the one or more technical specifications to the at least one completion threshold to identify at least one of the one or more steps of the maintenance event as having been completed in error; and

[0083] 15. The method of clause 13 or 14, further comprising, in response to identifying the at least one of the one or more steps as having been completed in error, communicating a notification to the operator.

[0073]

[0084] Article 16. 1. A wearable or portable monitoring system comprising:

[0085] a body configured to be one or more of: worn by an operator or coupled to said operator during a maintenance or inspection event of the aircraft;

[0086] a control unit including one or more processors operably coupled to the body, the control unit configured to wirelessly access one or more technical specifications associated with the aircraft or one or more of the maintenance or inspection events of the aircraft, wherein one or more steps of the maintenance or inspection events are included in at least one of the one or more technical specifications; and

[0087] one or more sensors operably coupled to the body configured to acquire data during the maintenance or inspection event, the data relating to one or more of the operator's activities during the maintenance or inspection event or one or more characteristics of the aircraft, the data comprising two or more different types of modalities;

[0088] a wearable or portable monitoring system, wherein the control unit is configured to monitor the maintenance or inspection event based at least in part on the data acquired by the one or more sensors during the maintenance or inspection event and the at least one of the one or more technical specifications.

[0074]

[0089] Article 17. 17. The wearable or portable monitoring system of clause 16, wherein the control unit is configured to verify completion of the one or more steps of the maintenance or inspection event by the operator based on each of the two or more different types of modalities of the data.

[0075]

[0090] Article 18. 18. The wearable or portable monitoring system of clause 16 or 17, further comprising an output device operably coupled to the body, wherein the control unit is configured to communicate with the operator during the maintenance or inspection event via the output device.

[0076]

[0091] Article 19. further comprising an output device operably coupled to the body;

[0092] the control unit is configured to identify at least one of the one or more steps of the maintenance or inspection event as having been completed in error by comparing the at least one completion threshold of the one or more technical specifications;

[0093] 19. The wearable or portable monitoring system of any one of clauses 16 to 18, wherein the control unit is configured to communicate a notification to the operator in response to identifying at least one of the one or more steps as having been completed in error.

[0077]

[0094] Article 20. 20. The wearable or portable monitoring system of claim 19, wherein the at least one of the one or more steps that is erroneously completed is a first step, and the control unit is configured to communicate the notification to the operator in response to the operator erroneously completing the first step before the operator starts a subsequent second step.

[0078]

[0095] As described herein, embodiments of the present disclosure provide systems and methods for a hands-free communication system between systems onboard an aircraft and / or systems offboard an aircraft. An aircraft hands-free communication system may enable a pilot to directly communicate with an operator offboard the aircraft and to update and / or enter information or data stored in one or more systems onboard the aircraft without having to physically engage one or more knobs, switches, buttons, touchscreens, etc. on the aircraft.

[0079]

[0096] For purposes of describing the embodiments of the present disclosure, various spatial and directional terms may be used, such as top, bottom, lower, center, sideways, horizontal, vertical, front, etc., but it should be understood that such terms are used solely with reference to the orientations shown in the drawings. These orientations may be flipped, rotated, or otherwise changed so that top becomes bottom, bottom becomes top, horizontal becomes vertical, etc.

[0080]

[0097] As used herein, a structure, limitation, or element that is "configured to" perform an task or operation is structurally shaped, configured, or adapted specifically to correspond to the task or operation. For clarity and to avoid doubt, an object that can merely be modified to perform a task or operation is not "configured / set up to" perform a task or operation as used herein.

[0081]

[0098] It should be understood that the above description is intended to be illustrative, not limiting. For example, the above-described examples (and / or aspects thereof) can be used in combination with each other. In addition, many modifications can be made to adapt a particular situation or material to the teachings of the various embodiments of the present disclosure without departing from the scope of the present disclosure. While the dimensions and types of materials described herein are intended to define aspects of the various embodiments of the present disclosure, the examples are by no means limiting, but are illustrative examples. Many other examples will be apparent to those skilled in the art upon reviewing the above description. The scope of the various embodiments of the present disclosure should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the accompanying claims and the detailed description herein, the words "including" and "in which" are used as the plain English equivalents of the words "comprising" and "wherein," respectively. Furthermore, terms such as "first," "second," and "third" are used merely as labels and are not intended to impose numerical requirements on their objects. Moreover, the limitations of the following claims are not written in means-plus-function form, and are not intended to be construed under 35 U.S.C. §112(f) unless such claim limitations expressly use the phrase "means for," followed by a statement of function lacking further structure.

[0082]

[0099] The description herein uses examples to disclose various embodiments of the present disclosure, including the best mode, and to enable any person skilled in the art to practice various embodiments of the present disclosure, including making and using any device or system and practicing any methods incorporated therein. The patentable scope of various examples of the present disclosure is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if the examples have structural elements that do not differ from the literal language of the claims, or if the examples include equivalent structural elements that differ only insignificantly from the literal language of the claims.

Claims

1. A portable monitoring system (102), comprising: a control unit (104) including one or more processors configured to access one or more technical specifications (122) associated with a maintenance event for an aircraft (101), the maintenance event including one or more steps configured to be completed by an operator to complete the maintenance event; and one or more sensors (108) configured to acquire data during the maintenance event of the aircraft (101), the data being associated with one or more of the operator's activities during the maintenance event or one or more characteristics of the aircraft (101), the data comprising two or more different types of modalities; The control unit (104) is configured to monitor the maintenance event based at least in part on the data acquired by the one or more sensors (108) during the maintenance event and at least one of the one or more technical specifications (122).

2. The portable monitoring system (102) of claim 1, wherein the two or more different types of modalities include one or more of video, text, audio, image, or touch.

3. 2. The portable monitoring system of claim 1, wherein the control unit is configured to verify completion of the one or more steps of the maintenance event by the operator based on each of the two or more different types of modalities of the data.

4. 2. The portable monitoring system of claim 1, further comprising an output device operably coupled to the control unit, the control unit configured to communicate with the operator during the maintenance event via the output device.

5. 5. The portable monitoring system of claim 4, wherein the control unit is configured to communicate one or more instructions to the operator for the operator to complete one or more of the one or more steps of the maintenance event.

6. The portable monitoring system (102) of any preceding claim, wherein the control unit (104) is configured to verify the completion of the one or more steps of the maintenance event.

7. 2. The portable monitoring system of claim 1, wherein the control unit is configured to identify at least one of the one or more steps of the maintenance event as having been completed in error by comparing the at least one completion threshold of the one or more technical specifications.

8. 8. The portable monitoring system of claim 7, further comprising an output device operatively coupled to the control unit, wherein the control unit is configured to communicate a notification to the operator via the output device in response to identifying the at least one of the one or more steps as having been completed in error.

9. 10. The portable monitoring system (102) of claim 8, wherein the notification is configured to include a recommendation of corrective action for the operator to complete to correct the at least one of the one or more steps that was completed in error.

10. 9. The portable monitoring system (102) of claim 8, wherein the at least one of the one or more steps that is erroneously completed is a first step, and the control unit (104) is configured to communicate the notification to the operator in response to the operator erroneously completing the first step before the operator begins a subsequent second step.

11. 2. The portable monitoring system (102) of claim 1, wherein the one or more sensors (108) and the control unit (104) are operably coupled to a body, the body being configured to be one or more of worn by the operator or coupled to the operator during the maintenance event.

12. The portable monitoring system (102) of claim 1, wherein the control unit (104) is an artificial intelligence (AI) or machine learning system.

13. accessing one or more technical specifications (122) associated with one or more maintenance events of the aircraft (101); acquiring data using one or more sensors of a portable monitoring system during a maintenance event on the aircraft, the data being associated with one or more of operator activities during the maintenance event or one or more characteristics of the aircraft, the data including two or more different types of modalities; and monitoring the maintenance event based at least in part on the data acquired by the one or more sensors (108) during the maintenance event and at least one of the one or more technical specifications (122).

14. The method of claim 13 , further comprising verifying completion of one or more steps of the maintenance event based on each of the two or more different types of modalities of the data.

15. comparing the at least one completion threshold of the one or more technical specifications (122) to identify that at least one of the one or more steps of the maintenance event was completed in error; and The method of claim 13 , further comprising, in response to identifying the at least one of the one or more steps as having been completed in error, communicating a notification to the operator.

16. A wearable or portable monitoring system (102), comprising: a body configured to be one or more of: worn by an operator or coupled to said operator during a maintenance or inspection event of the aircraft (101); a control unit (104) including one or more processors operably coupled to the body, the control unit (104) configured to wirelessly access one or more technical specifications (122) associated with the aircraft (101) or one or more of the maintenance or inspection events of the aircraft (101), wherein one or more steps of the maintenance or inspection events are included in at least one of the one or more technical specifications (122); and one or more sensors (108) operably coupled to the body and configured to acquire data during the maintenance or inspection event, the data relating to one or more of the operator's activities during the maintenance or inspection event or one or more characteristics of the aircraft (101), the data comprising two or more different types of modalities; The control unit (104) is configured to monitor the maintenance or inspection event based at least in part on the data acquired by the one or more sensors (108) during the maintenance or inspection event and the at least one of the one or more technical specifications (122).

17. 17. The portable monitoring system of claim 16, wherein the control unit is configured to verify completion of the one or more steps of the maintenance or inspection event by the operator based on each of the two or more different types of modalities of the data.

18. 17. The wearable or portable monitoring system (102) of claim 16, further comprising an output device operably coupled to the body, wherein the control unit (104) is configured to communicate with the operator during the maintenance or inspection event via the output device.

19. further comprising an output device operably coupled to the body; the control unit (104) is configured to identify at least one of the one or more steps of the maintenance or inspection event as having been completed in error by comparing the at least one completion threshold of the one or more technical specifications (122); 17. The wearable or portable monitoring system of claim 16, wherein the control unit is configured to communicate a notification to the operator in response to identifying the at least one of the one or more steps as having been completed in error.

20. 20. The portable monitoring system (102) of claim 19, wherein the at least one of the one or more steps that is erroneously completed is a first step, and the control unit (104) is configured to communicate the notification to the operator in response to the operator erroneously completing the first step before the operator begins a subsequent second step.

Images