Unit disability state monitoring system based on non-contact sensor

Abstract

The invention relates to the technical field of aviation flight safety, in particular to a unit disability state monitoring system based on a non-contact sensor. Comprising a millimeter wave radar, an infrared thermal imaging sensor, an eye movement tracking sensor, a voice recognition module, a data acquisition and preprocessing module, a data analysis module, a data storage module, a safety communication module, a display and interaction module and an alarm module. The data acquisition and preprocessing module is used for realizing synchronous acquisition and preprocessing operation on multi-source sensor data; the data analysis module is used for realizing real-time analysis of multi-source sensor data; and the alarm module is used for issuing alarms of different levels based on abnormal levels when the analysis result output by the data analysis module is abnormal. According to the system, real-time and accurate monitoring of the health state of a cockpit pilot in various aircrafts is realized, and failure state early warning is provided, so that the flight safety is greatly improved, and the diversified requirements of future aircraft development are met.

Description

Technical Field

[0001] The embodiments of this application relate to the field of aviation flight safety technology, and in particular to a crew incapacity monitoring system based on non-contact sensors. Background Technology

[0002] With the continuous development of the global aviation industry, the types and application scenarios of aircraft are becoming increasingly diversified. From traditional commercial aircraft and general aviation aircraft to emerging electric vertical takeoff and landing aircraft (eVTOL), various types of aircraft, while meeting different transportation needs, place higher demands on the pilots' control capabilities and health.

[0003] In the traditional aviation field, commercial aircraft and general aviation aircraft are the most common types of aircraft. Commercial aircraft typically undertake long-haul, high-capacity passenger and cargo transportation missions, while general aviation aircraft are widely used in short-haul transportation, emergency rescue, agricultural and forestry operations, and other fields. The health status of pilots in these aircraft is crucial to flight safety. Fatigue, stress, and sudden health problems during flight can all affect the pilot's ability to control the aircraft, thus potentially threatening flight safety.

[0004] In recent years, eVTOL aircraft, as a representative of new types of aircraft, have received widespread attention. Electric-powered, vertical takeoff and landing (VTOL) aircraft are characterized by low noise and low emissions, and are considered an important component of future urban air mobility (UAM) and short-haul air transport. The high frequency and short-haul flights of eVTOL aircraft in urban environments, along with the high demands placed on pilots, further highlight the importance of health monitoring.

[0005] Currently, the commonly used methods for monitoring the health status of generator units mainly rely on contact-based physiological signal monitoring equipment, such as electrocardiogram (ECG) monitors and blood oxygen saturation monitors. Although these devices can provide relatively accurate physiological data, they have the following limitations in practical applications.

[0006] The first issue is the inconvenience and comfort of wearing them. Wearing contact sensors for extended periods can cause discomfort for pilots, especially during long-haul flights or in complex operating environments. The sensors may fall off or generate abnormal data due to factors such as sweat and friction.

[0007] Secondly, there is psychological stress and operational interference. When pilots wear these devices for extended periods, it may increase their psychological burden. During stressful flight missions, psychological stress may be further aggravated, which can adversely affect the pilot's operation.

[0008] Thirdly, there is the issue of data continuity and accuracy. Contact sensors may experience data interruptions or distortion during flight due to poor contact or environmental changes, especially in emergencies, potentially failing to reflect the pilot's true health status in a timely manner.

[0009] Given the aforementioned challenges, non-contact sensing technology has gradually become a research hotspot in the field of health status monitoring. Using technologies such as radar, infrared, and laser, non-contact sensors can remotely monitor physiological parameters such as heart rate, respiration, and body temperature without physical contact with the pilot. This technology offers significant advantages in both traditional aircraft and novel eVTOL aircraft.

[0010] Firstly, it improves comfort and reduces interference. Non-contact sensors do not need to be directly attached to the pilot's skin, reducing discomfort and psychological stress caused by wearing sensors, and do not interfere with the pilot's operation, making them particularly suitable for the high-frequency takeoff and landing operations of eVTOL aircraft.

[0011] Secondly, it enhances real-time performance and versatility. Non-contact sensors can monitor multiple physiological indicators of pilots in real time, providing a more comprehensive health assessment through data fusion technology, making them suitable for the complex operating environments of various aircraft.

[0012] Finally, there's the challenge of adapting to complex environments and improving reliability. The rapid changes in aircraft flight attitude and the complexity of the external environment place higher demands on the sensors' anti-interference capabilities. Non-contact sensing technology, through advanced signal processing algorithms, can maintain high monitoring accuracy and reliability in complex environments.

[0013] Despite the great potential of non-contact sensing technology in the field of health monitoring, its application in monitoring pilot disability status still faces some challenges, especially the issue of adaptability in various aircraft environments.

[0014] Therefore, developing a crew incapacity monitoring system that can adapt to various types of aircraft, including commercial aircraft, general aviation aircraft, and eVTOL aircraft, has become crucial to ensuring future flight safety. Summary of the Invention

[0015] In view of this, embodiments of this application propose a crew incapacity monitoring system based on non-contact sensors, which aims to achieve real-time and accurate monitoring of the health status of pilots in various aircraft through advanced sensing technology and intelligent algorithms, and provide incapacity warnings, thereby significantly improving flight safety and meeting the diverse needs of future aircraft development.

[0016] To achieve the above objectives, embodiments of this application propose a unit failure status monitoring system based on non-contact sensors. The system includes: millimeter-wave radar, infrared thermal imaging sensor, eye-tracking sensor, voice recognition module, data acquisition and preprocessing module, data analysis module, data storage module, secure communication module, display and interaction module, and alarm module. Millimeter-wave radar is a frequency-modulated continuous-wave radar used to monitor the heart rate, respiratory rate, presence, identity, gesture recognition, and posture of designated targets through radar signals. Infrared thermal imaging sensors are small, uncooled infrared detectors used to monitor the body temperature of people within the sensor's field of view in real time. The eye-tracking sensor is an embedded, hidden eye-tracking sensor used to monitor the pilot's eye movement parameters in real time, including blink frequency, duration of eye closure, and direction of gaze. The speech recognition module is used to collect the pilot's speech and locate the sound source; The data acquisition and preprocessing module is equipped with multiple different types of data acquisition interfaces to realize the synchronous acquisition and preprocessing of multi-source sensor data, including millimeter-wave radar data, infrared thermal imaging data, eye-tracking sensor data, and voice data. The data analysis module is equipped with a high-performance general-purpose computing unit and a high-performance AI computing unit to enable real-time analysis of multi-source sensor data; The data storage module is equipped with a large-capacity storage unit for real-time storage of multi-source sensor data, analysis results output by the data analysis module, and system logs; The secure communication module is equipped with an aviation Ethernet switch and an avionics security gateway to enable one-way secure communication between system data and the aircraft's onboard network. The display and interaction module is equipped with a touch screen to receive system status data and provide feedback through touch and button interactions. The alarm module is equipped with an audio alarm unit, a visual alarm unit, and a vibration alarm unit, which are used to issue alarms of different levels based on the anomaly level when the analysis results output by the data analysis module are abnormal.

[0017] To achieve the above objectives, embodiments of this application also propose a method for monitoring unit failure status based on non-contact sensors, implemented based on a unit failure status monitoring system based on non-contact sensors as described above. The method includes: The data acquisition and preprocessing module collects multi-source sensor data in real time, including millimeter-wave radar data, infrared thermal imaging data, eye-tracking sensor data, face image data, and voice data, and performs time synchronization, resampling, filtering, and noise reduction processing on the multi-source sensor data. The data acquisition and preprocessing module outputs processed multi-source sensor data, which is then analyzed by the data analysis module based on millimeter-wave radar data to detect the presence of flight crew. If no flight crew is detected or the number of flight crew members present is less than the specified number, it is assumed that the flight crew has left their posts and the aircraft is in an unmanned state, triggering an alarm. If the presence of flight crew members is detected and the required number is met, the data analysis module will perform real-time analysis of the crew's status characteristics, including crew identity, heart rate, respiratory rate, body temperature, blood oxygen saturation, eye movement characteristics, and gesture characteristics. Based on the crew identification results, the database stores the individual baseline data, and the analysis results output by the data analysis module are compared with the individual baseline data. The comparison method is either threshold method or trend analysis method, which enables the identification of abnormal heart rate, abnormal respiratory rate, abnormal body temperature, abnormal blood oxygen saturation, abnormal eye movement, and abnormal sitting posture of crew members. If the system detects one abnormal status item for the crew, a minor alarm will be triggered. If the system detects two abnormal status items for the crew, a medium alarm will be triggered. If the system detects no less than three abnormal status items for the crew, a severe alarm will be triggered. A mild system alarm is a visual flashing alarm; a moderate system alarm is a visual flashing alarm plus an audible alarm; and a severe system alarm is a visual flashing alarm plus an audible alarm plus a seat vibration alarm.

[0018] Optionally, the millimeter-wave radar uses a frequency band of 60GHz to 64GHz suitable for monitoring human vital signs. The millimeter-wave radar can be installed in a fixed, embedded manner, or in a concealed or exposed manner depending on the material of the cockpit interior.

[0019] Optionally, the designated target personnel are the flight crew members in the cockpit. When the aircraft is in single-pilot mode, the designated target personnel is one person, and when the aircraft is in two-pilot mode, the designated target personnel is two people.

[0020] Optionally, the heart rate monitoring, respiratory rate monitoring, presence detection, identity recognition, gesture recognition, and posture recognition of designated target personnel, achieved through radar signals, are all performed by an application residing in the data analysis module. For presence detection, the presence detection application residing in the data analysis module analyzes the millimeter-wave radar echo signals of the crew members in the cockpit to determine whether the crew members are in the cockpit and whether the number of crew members in the cockpit is the same as the prescribed number.

[0021] Optionally, for identity recognition, the identity recognition application residing in the data analysis module has multiple built-in identity recognition algorithms, including at least a speaker-based voice recognition algorithm, a face image matching-based identity recognition algorithm, and a cardiac radar signal-based identity recognition algorithm. During identity recognition, the collected millimeter-wave radar data is preprocessed, including resampling, filtering, and noise reduction, to eliminate irrelevant redundant interference and extract phase signals. Then, variational mode decomposition is performed on the phase signals containing interference to extract features and create identity recognition feature samples. Finally, the identity recognition feature samples are trained using two-dimensional principal component analysis and used for identity recognition.

[0022] Optionally, the detector pixels of the infrared thermal imaging sensor are not less than [number missing]. The detector's field of view is not less than Infrared thermal imaging sensors can achieve real-time and accurate temperature measurement of single or multiple points, with a measurement range of [missing information]. to Temperature measurement accuracy error is less than .

[0023] Optionally, the eye-tracking sensor is installed in an embedded, concealed manner, located below the panel of the flight display directly in front of the flight crew. The eye-tracking sensor also has a video image acquisition function, which can output video images within the sensor's field of view to the data acquisition and preprocessing module in real time.

[0024] Optionally, the data acquisition and preprocessing module integrates a GPS communication interface, which can perform time synchronization via GNSS. The data synchronization acquisition scheme of the data acquisition and preprocessing module uses GPS time as the reference time and adopts the PTP clock synchronization protocol to complete the time synchronization acquisition between the sensors.

[0025] Optionally, the high-performance AI computing unit includes GPU, NPU, and FPGA, with a computing power of no less than 6 TOPS.

[0026] The unit failure status monitoring system proposed in this application, based on non-contact sensors, offers the following advantages compared to traditional health status monitoring schemes based on non-contact sensors.

[0027] First, it significantly improves the accuracy and real-time performance of monitoring. This application utilizes advanced non-contact sensing technology to achieve real-time monitoring of pilots' physiological parameters such as heart rate, respiration, and body temperature. Through the fusion of data from multiple sensors and intelligent algorithm processing, the system can maintain high-precision monitoring in changing flight environments, promptly detect abnormal pilot health conditions, and issue timely alarms in the event of pilot incapacitation (crew incapacitation), effectively improving flight safety.

[0028] Secondly, it improves pilot comfort and operational freedom. Unlike traditional contact sensors, the non-contact sensor used in this application does not require direct contact with the pilot's skin, avoiding the discomfort and psychological stress that may result from wearing sensors. This not only improves pilot comfort but also ensures that the pilot can maintain optimal performance during long-duration flights or high-frequency operations, free from interference from external devices.

[0029] Third, it is applicable to a variety of aircraft types. The system design of this application is highly adaptable and applicable to a variety of aircraft types, including commercial aircraft, general aviation aircraft, and eVTOL aircraft. Whether in traditional large aircraft or emerging urban air transportation vehicles, the system can operate stably and reliably, meeting the needs of different aircraft for monitoring the health status of crew members under different flight environments. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0031] Figure 1 This is a schematic diagram of a unit failure status monitoring system based on a non-contact sensor provided in one embodiment of this application; Figure 2 This is a flowchart of a unit failure status monitoring method based on non-contact sensors provided in another embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0033] One embodiment of this application proposes a unit failure status monitoring system based on non-contact sensors. The implementation details of the unit failure status monitoring system based on non-contact sensors proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding only and are not necessary for implementing this solution.

[0034] The specific components of the unit failure status monitoring system based on non-contact sensors proposed in this embodiment can be as follows: Figure 1 As shown, it includes: millimeter-wave radar 11, infrared thermal imaging sensor 12, eye-tracking sensor 13, voice recognition module 14, data acquisition and preprocessing module 21, data analysis module 22, data storage module 31, secure communication module 23, display and interaction module 32, and alarm module 24.

[0035] Millimeter-wave radar 11 is a frequency modulated continuous wave radar (FMCW), used to achieve functions such as heart rate monitoring, respiratory rate monitoring, presence detection, identity recognition, gesture recognition, and posture recognition of designated target personnel through radar signals (these functions are implemented in data analysis module 22).

[0036] The infrared thermal imaging sensor 12 is a small, uncooled infrared detector used to monitor the body temperature of people within the sensor's field of view in real time.

[0037] The eye-tracking sensor 13 is an embedded, hidden eye-tracking sensor used to monitor the pilot's eye movement parameters in real time. The eye movement parameters include at least blink frequency, duration of eye closure, and direction of gaze.

[0038] The speech recognition module 14 is a linear speech microphone array used to realize functions such as real-time speech acquisition and sound source localization for the pilot.

[0039] The data acquisition and preprocessing module 21 is equipped with multiple different types of data acquisition interfaces to realize the synchronous acquisition and preprocessing of multi-source sensor data, including millimeter-wave radar data (acquired from millimeter-wave radar 11), infrared thermal imaging data (acquired from infrared thermal imaging sensor 12), eye-tracking sensor data (acquired from eye-tracking sensor 13), and voice data (acquired from voice recognition module 14).

[0040] The data analysis module 22 is equipped with a high-performance general-purpose computing unit and a high-computing-power AI computing unit to realize real-time analysis of multi-source sensor data.

[0041] The data storage module 31 is equipped with a large-capacity storage unit for real-time storage of various data, including multi-source sensor data, analysis results output by the data analysis module, and system logs.

[0042] The secure communication module 23 is equipped with an aviation Ethernet switch and an avionics security gateway to enable one-way secure communication between system data and the aircraft’s onboard networks (such as the aircraft control domain, airline information service domain, and other network domains).

[0043] The display and interaction module 32 is equipped with a touch screen to receive system status data and provide feedback on interactive information through touch and button interactions.

[0044] The alarm module 24 is equipped with an audio alarm unit, a visual alarm unit, and a vibration alarm unit, which are used to issue alarms of different levels based on the anomaly level when the analysis results output by the data analysis module are abnormal.

[0045] The following is a detailed description of each component of the unit failure status monitoring system based on non-contact sensors proposed in this embodiment.

[0046] In one example, the millimeter-wave radar operates in the 60GHz to 64GHz frequency band, suitable for monitoring human vital signs. The millimeter-wave radar is installed in a fixed, embedded manner, and can be either concealed or exposed depending on the interior material of the cockpit.

[0047] In one example, the designated target personnel are the flight crew members in the cockpit. When the aircraft is in single-pilot mode (equipped with one pilot), the designated target personnel number is one person, and when the aircraft is in two-pilot mode (equipped with two pilots), the designated target personnel number is two people.

[0048] In one example, the heart rate monitoring, respiratory rate monitoring, presence detection, identity recognition, gesture recognition, and posture recognition of a designated target person, achieved through radar signals, are all performed by applications residing in the data analysis module. That is, the data analysis module is configured to have applications for heart rate monitoring, respiratory rate monitoring, presence detection, identity recognition, gesture recognition, and posture recognition residing in it.

[0049] For presence detection, the presence detection application residing in the data analysis module analyzes the millimeter-wave radar echo signals of the crew members in the cockpit to determine whether the crew members are in the cockpit and whether the number of crew members in the cockpit is the same as the prescribed number. If the presence of the crew members is not detected, or the number of crew members detected is less than the prescribed number, it is considered that the crew members have left their posts and the aircraft is in an unmanned state, requiring the triggering of an alarm.

[0050] For identity recognition, the identity recognition application residing in the data analysis module incorporates various identity recognition algorithms, including but not limited to those based on speaker voice, face image matching, and cardiac radar signals. During identity recognition, the acquired millimeter-wave radar data undergoes preprocessing including resampling, filtering, and noise reduction to eliminate irrelevant redundant interference and extract phase signals. Then, variational mode decomposition is performed on the interfering phase signals to extract features and create identity recognition feature samples. Finally, these feature samples are trained using two-dimensional principal component analysis and used for identity recognition.

[0051] In one example, the detector pixels of the infrared thermal imaging sensor are no less than [number missing]. The detector's field of view is not less than Infrared thermal imaging sensors can achieve real-time and accurate temperature measurement of single or multiple points, with a measurement range of [missing information]. to Temperature measurement accuracy error is less than .

[0052] In one example, the eye-tracking sensor is installed in a recessed, concealed configuration, positioned below the flight display panel directly in front of the flight crew. The eye-tracking sensor also features video image acquisition capabilities, enabling it to output real-time video images within its field of view to the data acquisition and preprocessing module.

[0053] In one example, the data acquisition and preprocessing module integrates a GPS communication interface, which can perform time synchronization via GNSS. The data synchronization acquisition scheme of the data acquisition and preprocessing module uses GPS time as the reference time and adopts the PTP clock synchronization protocol to complete the time synchronization acquisition between various sensors.

[0054] In one example, the high-performance AI computing unit configured in the data analysis module includes, but is not limited to, GPUs, NPUs, FPGAs, etc., with a computing power of no less than 6 TOPS.

[0055] In one example, the data analysis module searches for individual baseline data stored in the database based on the crew identification results and compares the analysis results with the individual baseline data. Comparison methods include thresholding and trend analysis to identify abnormalities in crew members' heart rate, respiratory rate, body temperature, blood oxygen saturation, eye movement, and posture. If the system detects one abnormality (such as abnormal heart rate), a mild alarm is triggered. If the system detects two abnormalities (such as abnormal heart rate and abnormal body temperature), a moderate alarm is triggered. If the system detects at least three abnormalities (such as abnormal heart rate, abnormal body temperature, and abnormal eye movement simultaneously), a severe alarm is triggered. A mild alarm is a visual flashing alarm; a moderate alarm is a visual flashing alarm combined with an audible alarm; and a severe alarm is a combination of a visual flashing alarm, an audible alarm, and a seat vibration alarm.

[0056] The unit failure status monitoring system based on non-contact sensors proposed in this embodiment has the following beneficial effects compared with the traditional health status monitoring scheme based on non-contact sensors.

[0057] First, it significantly improves the accuracy and real-time performance of monitoring. This embodiment utilizes advanced non-contact sensing technology to achieve real-time monitoring of pilots' physiological parameters such as heart rate, respiration, and body temperature. Through the fusion of data from multiple sensors and intelligent algorithm processing, the system can maintain high-precision monitoring in changing flight environments, promptly detect abnormal pilot health conditions, and issue timely alarms in the event of pilot incapacitation (crew incapacitation), effectively improving flight safety.

[0058] Secondly, it improves pilot comfort and operational freedom. Unlike traditional contact sensors, the non-contact sensor used in this embodiment does not require direct contact with the pilot's skin, avoiding the discomfort and psychological stress that may result from wearing sensors. This not only improves pilot comfort but also ensures that the pilot can maintain optimal condition during long-duration flights or high-frequency operations, unaffected by external equipment interference.

[0059] Third, it is applicable to a variety of aircraft types. The system design in this embodiment is highly adaptable and suitable for various aircraft types, including commercial aircraft, general aviation aircraft, and eVTOL aircraft. Whether in traditional large aircraft or emerging urban air transportation vehicles, the system can operate stably and reliably, meeting the needs of different aircraft for monitoring the health status of crew members under different flight environments.

[0060] It is worth noting that all modules involved in this embodiment are logical modules. In practical applications, a logical module can be a physical module, a part of a physical module, or an organic combination of multiple physical modules. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce modules that are not closely related to solving the technical problems proposed in this application. However, this does not mean that other modules are absent from this embodiment.

[0061] Another embodiment of this application proposes a unit failure status monitoring method based on non-contact sensors, which is implemented based on a unit failure status monitoring system based on non-contact sensors as described in the above system embodiment. The implementation details of the unit failure status monitoring method based on non-contact sensors proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this example.

[0062] Figure 2 This is a flowchart illustrating a unit failure status monitoring method based on non-contact sensors proposed in this embodiment, including: Step 41: The data acquisition and preprocessing module acquires multi-source sensor data in real time, including millimeter-wave radar data, infrared thermal imaging data, eye-tracking sensor data, face image data, and voice data, and performs time synchronization, resampling, filtering, and noise reduction processing on the multi-source sensor data.

[0063] Step 42: The data acquisition and preprocessing module outputs the processed multi-source sensor data. The data analysis module then uses the millimeter-wave radar data to detect the presence of the flight crew. If no flight crew is detected or the number of flight crew members present is less than the specified number, it is assumed that the flight crew has left their posts and the aircraft is in an unmanned state, triggering an alarm.

[0064] Step 43: If the presence of flight crew members is detected and the required number is met, the data analysis module will perform real-time analysis of the crew's status characteristics, including crew identity, heart rate, respiratory rate, body temperature, blood oxygen saturation, eye movement characteristics, and gesture characteristics.

[0065] Step 44: Based on the crew identification results, search for the personal baseline data stored in the database, and compare the analysis results output by the data analysis module with the personal baseline data. The comparison method is the threshold method or the trend analysis method to realize the identification of abnormal heart rate, abnormal respiratory rate, abnormal body temperature, abnormal blood oxygen saturation, abnormal eye movement, and abnormal sitting posture of crew members.

[0066] Step 45: If the system detects one abnormal status item for the crew members, a minor alarm is triggered. If the system detects two abnormal status items for the crew members, a medium alarm is triggered. If the system detects no less than three abnormal status items for the crew members, a severe alarm is triggered.

[0067] Step 46: A mild system alarm is a visual flashing alarm; a moderate system alarm is a visual flashing alarm plus an audible alarm; and a severe system alarm is a visual flashing alarm plus an audible alarm plus a seat vibration alarm.

[0068] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.

[0069] It is not difficult to see that this embodiment is a method embodiment corresponding to the above system embodiment, and this embodiment can be implemented in conjunction with the above system embodiment. The relevant technical details and technical effects mentioned in the above system embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above system embodiment.

[0070] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables a unit failure status monitoring method based on non-contact sensors as described in the above method embodiments.

[0071] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0072] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A non-contact sensor based unit incapacitation monitoring system, characterized by, The system comprises a millimeter wave radar, an infrared thermal imaging sensor, an eye movement tracking sensor, a voice recognition module, a data acquisition and preprocessing module, a data analysis module, a data storage module, a secure communication module, a display and interaction module, and an alarm module. The millimeter wave radar is a frequency-modulated continuous wave radar, which is used to realize heart rate monitoring, respiratory rate monitoring, presence detection, identity recognition, gesture recognition, and posture recognition of the specified target personnel through radar signals. The infrared thermal imaging sensor is a small infrared non-cryogenic detector, which is used to realize real-time monitoring of the body temperature of personnel within the sensor's field of view. The eye movement tracking sensor is an embedded hidden eye movement tracking sensor, which is used to realize real-time monitoring of the eye movement parameters of the pilot, including blink frequency, closed eye duration, and gaze direction. The voice recognition module is used to realize voice acquisition and sound source positioning of the pilot. The data acquisition and preprocessing module is provided with multiple different types of data acquisition interfaces, which are used to realize synchronous acquisition and preprocessing of multi-source sensor data, including millimeter wave radar data, infrared thermal imaging data, eye movement tracking sensor data, and voice data. The data analysis module is configured with high-performance general-purpose computing units and high-performance AI computing units, which are used to realize real-time analysis of multi-source sensor data. The data storage module is configured with a large-capacity storage unit, which is used to store multi-source sensor data, analysis results output by the data analysis module, and system logs in real time. The secure communication module is configured with an aviation Ethernet switch and an avionics security gateway, which are used to realize one-way secure communication between system data and the aircraft's onboard network. The display and interaction module is configured with a touchable display, which is used to receive system state data and feedback interaction information through touch interaction and button interaction. The alarm module is configured with an audio alarm unit, a visual alarm unit, and a vibration alarm unit, which are used to issue different levels of alarms based on the abnormality level when the analysis result output by the data analysis module is abnormal.

2. The system of claim 1, wherein, The frequency band of the millimeter wave radar is 60GHz to 64GHz, which is suitable for human vital sign monitoring. The millimeter wave radar is installed in a hidden or exposed manner according to the different materials of the cockpit interior.

3. The system of claim 1, wherein, The specified target personnel are the flight crew in the cockpit. When the aircraft is in single-pilot mode, the specified number of target personnel is one, and when the aircraft is in dual-pilot mode, the specified number of target personnel is two.

4. The system of claim 3, wherein, The heart rate monitoring, respiratory rate monitoring, presence detection, identity recognition, gesture recognition, and posture recognition of the specified target personnel through radar signals are executed by the application program residing in the data analysis module. For presence detection, the presence detection application program residing in the data analysis module analyzes the millimeter wave radar echo signals of the cockpit crew to determine whether the cockpit crew is in the cockpit and whether the number of cockpit crew is the same as the specified number.

5. The system of claim 4, wherein, For identity recognition, the identity recognition application program residing in the data analysis module is built-in with multiple identity recognition algorithms, at least including an identity recognition algorithm based on speaker voice, an identity recognition algorithm based on face image matching, and an identity recognition algorithm based on heart radar signal; In the process of identity recognition, the collected millimeter wave radar data is pre-processed including resampling, filtering, and noise reduction, irrelevant redundant interference is eliminated, and phase signals are extracted, then the phase signals containing interference are subjected to variational mode decomposition, features are extracted, and identity recognition feature samples are made, finally the identity recognition feature samples are trained by two-dimensional principal component analysis method and identity recognition is performed.

6. The system of claim 1, wherein, The detector pixel of the infrared thermal imaging sensor is not less than 10 microns , and the detector field of view angle is not less than 10 microns . The infrared thermal imaging sensor can realize real-time and accurate temperature measurement of single point and multiple points, the temperature measurement range is to , and the temperature measurement accuracy error is less than .

7. The system of claim 1, wherein, The eye tracking sensor is installed in an embedded hidden manner, and the installation position is below the panel of the flight display directly in front of the flight crew; The eye tracking sensor also has a video image acquisition function, which can output video images in the sensor field of view to the data acquisition and preprocessing module in real time.

8. The system of claim 1, wherein, The data acquisition and preprocessing module integrates a GPS communication interface, which can perform time service through GNSS, and the data synchronization acquisition scheme of the data acquisition and preprocessing module is to use the GPS time as the reference time and adopt the PTP clock synchronization protocol to complete the time synchronization acquisition between sensors.

9. The system of claim 1, wherein, The high-computing-power AI computing unit includes GPU, NPU, and FPGA, and its computing power is not less than 6TOPS.

10. A non-contact sensor based unit incapacitation monitoring method, implemented based on a non-contact sensor based unit incapacitation monitoring system as claimed in any one of claims 1 to 9, characterized in that, The method comprises: The data acquisition and preprocessing module acquires multi-source sensor data including millimeter wave radar data, infrared thermal imaging data, eye tracking sensor data, face image data, and voice data in real time, and performs time synchronization, resampling, filtering, and noise reduction processing on the multi-source sensor data; The data acquisition and preprocessing module outputs the processed multi-source sensor data, the data analysis module detects the presence of the flight crew according to the millimeter wave radar data, if no flight crew is detected or the number of the existing flight crew is less than the specified number, it is considered that the flight crew has left the post, the aircraft is in an uncontrolled state, and an alarm is triggered; If the presence of the flight crew is detected and the number meets the specified number, the data analysis module performs real-time analysis of the crew state features, including feature data of crew identity, heart rate, breathing rate, body temperature, blood oxygen saturation, eye movement characteristics, and hand gesture characteristics; According to the crew identity recognition result, the personal baseline data saved in the database is searched, and the analysis result output by the data analysis module is compared with the personal baseline data by threshold method or trend analysis method, to realize abnormal recognition of the crew's heart rate, breathing rate, body temperature, blood oxygen saturation, eye movement, and sitting posture; If the system monitors one state abnormality of the crew, a light system alarm is triggered, if the system monitors two state abnormalities of the crew, a medium system alarm is triggered, and if the system monitors not less than three state abnormalities of the crew, a heavy system alarm is triggered. The system light warning is a visual flashing warning, the system medium warning is a visual flashing warning plus a sound warning, and the system severe warning is a visual flashing warning plus a sound warning plus a seat vibration warning.

Images