Flight crew monitoring method and system and computer readable storage medium

Abstract

The invention discloses a flight crew monitoring method and system and a computer readable storage medium, and relates to the technical field of flight crew state monitoring. The monitoring method comprises the steps that monitoring equipment is controlled to monitor the state of a flight crew to obtain monitoring state information, and the monitoring equipment is configured to be not needed to be worn by the flight crew; and determining an alarm strategy based on the monitoring state information in combination with the flight state information of the aircraft. By means of the mode, the state of the flight crew can be monitored to guarantee flight safety, the flight crew does not need to wear monitoring equipment, the time consumed by the flight crew to wear the monitoring equipment can be saved, and the psychological burden of the flight crew can be reduced.

Description

Technical Field

[0001] The present application relates to the technical field of flight crew status monitoring, and in particular to a flight crew monitoring method and system, and a computer-readable storage medium. Background Art

[0002] Safety is the primary prerequisite for the development of air transportation. Pilots, as the direct operators of aircraft, play a crucial role in aviation safety. Modern aircraft operations rely heavily on the crew's judgment, control, and coordination. Therefore, continuous monitoring of crew members' driving behavior, as well as their physiological and psychological state, is crucial to ensuring flight safety. Summary of the Invention

[0003] The present application provides a flight crew monitoring method and system, and a computer-readable storage medium, which can monitor the status of the flight crew to ensure flight safety. The flight crew does not need to wear monitoring equipment, which can save the flight crew the time spent wearing monitoring equipment and reduce the psychological burden of the flight crew.

[0004] The present application provides a flight crew monitoring method, the monitoring method comprising: controlling a monitoring device to monitor the status of the flight crew to obtain monitoring status information, wherein the monitoring device is configured to not require the flight crew to wear; and determining an alarm strategy based on the monitoring status information and in combination with the flight status information of the aircraft.

[0005] In one embodiment of the present application, the monitoring equipment includes a camera device and a millimeter-wave radar; based on the monitoring status information and in combination with the flight status information of the aircraft, the step of determining the warning strategy includes: according to a data source weight allocation strategy, based on the data monitored by the camera device and / or the millimeter-wave radar, in combination with the flight status information, determining the warning strategy; wherein the data source weight allocation strategy includes at least one of the characteristics of the camera device, the facial occlusion of the flight crew, and the psychological acceptance of the camera device by the flight crew.

[0006] In one embodiment of the present application, according to the data source weight distribution strategy, based on the data monitored by the camera device and / or the millimeter-wave radar, and combined with the flight status information, the step of determining the warning strategy includes: determining the light intensity of the cockpit environment; when the light intensity of the cockpit environment is between the lower light intensity limit and the upper light intensity limit, determining the warning strategy based on the data monitored by the camera device, and when the light intensity of the cockpit environment is less than the lower light intensity limit or greater than the upper light intensity limit, determining the warning strategy based on the data monitored by the millimeter-wave radar; or when the light intensity of the cockpit environment is less than or equal to the upper light intensity limit, determining the warning strategy based on the data monitored by the camera device, and when the light intensity of the cockpit environment is greater than the upper light intensity limit, determining the warning strategy based on the data monitored by the millimeter-wave radar; or when the light intensity of the cockpit environment is greater than or equal to the lower light intensity limit, determining the warning strategy based on the data monitored by the camera device, and when the light intensity of the cockpit environment is less than the lower light intensity limit, determining the warning strategy based on the data monitored by the millimeter-wave radar.

[0007] In one embodiment of the present application, according to the data source weight distribution strategy, based on the data monitored by the camera equipment and / or the millimeter-wave radar, combined with the flight status information, the steps of determining the warning strategy include: determining the obscured parts and unobstructed parts of the flight crew's face; obtaining facial feature information of the flight crew based on the data obtained by monitoring the unobstructed parts with the camera equipment and the data obtained by monitoring the obscured parts with the millimeter-wave radar, so as to determine the warning strategy.

[0008] In one embodiment of the present application, according to the data source weight distribution strategy, based on the data monitored by the camera device and / or the millimeter-wave radar, combined with the flight status information, the steps of determining the alarm strategy include: determining that the camera device is in a stopped working state, and determining the alarm strategy based on the data monitored by the millimeter-wave radar, combined with the flight status information.

[0009] In one embodiment of the present application, the monitoring device includes a fiber optic sensor; and the step of controlling the monitoring device to monitor the status of the flight crew includes: controlling the fiber optic sensor to monitor physiological characteristic information of the flight crew.

[0010] In one embodiment of the present application, the monitoring device includes a cabin sound collection device; the step of controlling the monitoring device to monitor the status of the flight crew includes: controlling the cabin sound collection device to monitor the audio information in the cockpit environment to obtain cabin sound characteristic information.

[0011] In one embodiment of the present application, the monitoring status information includes at least one of physiological characteristic information, facial characteristic information, behavioral posture information, and cabin sound characteristic information of the flight crew; the step of determining the alarm strategy based on the monitoring status information and combined with the flight status information of the aircraft includes: determining the driving status of the flight crew based on the monitoring status information; when the flight crew is in an abnormal driving state, determining the alarm strategy in combination with the flight status information.

[0012] In one embodiment of the present application, the abnormal driving state is divided into a level one abnormal driving state and a level two abnormal driving state according to the degree of impact on flight safety, wherein the level one abnormal driving state has a lower degree of impact on flight safety than the level two abnormal driving state; when the flight crew is in the abnormal driving state, the step of determining the alarm strategy in combination with the flight status information includes: if the flight crew is in the level one abnormal driving state, determining a first alarm strategy; if the flight crew is in the level two abnormal driving state, determining a second alarm strategy; wherein the first alarm strategy is different from the second alarm strategy.

[0013] In one embodiment of the present application, the flight status information includes the flight phase of the aircraft, and the flight phase includes a critical phase and a non-critical phase; if the flight crew is in a level one abnormal driving state, the step of determining the first alarm strategy also includes: when the flight crew is in a level one abnormal driving state and the aircraft is in a critical phase, the first alarm strategy includes canceling the intervention measures corresponding to the level one abnormal driving state.

[0014] In one embodiment of the present application, the second warning strategy includes outputting warning information in image form and / or audio form, and communicating with a ground workstation to monitor the movement of the aircraft; wherein the warning information carries information that the flight crew is in a second-level abnormal driving state.

[0015] Accordingly, the present application also provides a monitoring system, comprising: a monitoring device for monitoring the status of a flight crew member, wherein the monitoring device is configured to not require the flight crew member to wear it; an airborne core network, which is communicatively connected to the monitoring device, and the airborne core network is configured to determine an alarm strategy based on the monitoring status information and in combination with the flight status information of the aircraft.

[0016] In one embodiment of the present application, the monitoring device includes a camera device and a millimeter-wave radar, and the data monitored by the camera device and the millimeter-wave radar are configured to be assigned different weights according to a data source weight allocation strategy to determine an alarm strategy, wherein the data source weight allocation strategy includes at least one of the characteristics of the camera device, the facial occlusion of the flight crew, and the psychological acceptance of the flight crew to the camera device.

[0017] In an embodiment of the present application, the face of the flight crew member includes an obstructed portion and an unobstructed portion, and the millimeter-wave radar is configured to monitor at least the obstructed portion.

[0018] In an embodiment of the present application, the monitoring device includes a fiber optic sensor configured to monitor physiological characteristic information of a flight crew member.

[0019] In one embodiment of the present application, the onboard core network includes: a flight alert module for obtaining the flight crew's driving status and flight status information, wherein the flight crew's driving status is obtained by analyzing monitoring status information; and a comprehensive processing module, which is configured to combine the flight status information to determine an alert strategy when the flight crew is in an abnormal driving state.

[0020] In one embodiment of the present application, the abnormal driving state is divided into a level one abnormal driving state and a level two abnormal driving state according to the degree of impact on flight safety, wherein the level one abnormal driving state has a lower degree of impact on flight safety than the level two abnormal driving state; the flight state information includes the flight phase of the aircraft, and the flight phase includes a critical phase and a non-critical phase; wherein, when the flight crew is in a level one abnormal driving state and the aircraft is in a critical phase, the alarm strategy includes canceling the intervention measures corresponding to the level one abnormal driving state.

[0021] Accordingly, the present application also provides a computer-readable storage medium, which stores a program. The program can be executed to implement the flight crew monitoring method as described in the above embodiment.

[0022] The beneficial effects of this application are as follows: Unlike existing technologies, this application provides a flight crew monitoring method and system, as well as a computer-readable storage medium. This application monitors the flight crew's status by controlling monitoring equipment to obtain monitoring status information, and determines an alert strategy based on the aircraft's flight status information. This allows monitoring of the flight crew's status to ensure flight safety. Furthermore, the monitoring equipment is configured so that the flight crew does not need to wear it. This reduces the time spent on wearing the monitoring equipment and reduces the psychological burden on the flight crew. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1This is a schematic structural diagram of an embodiment of the monitoring system of the present application;

[0025] Figure 2 It is a schematic diagram of an embodiment of a flight phase of the aircraft of the present application;

[0026] Figure 3 This is a flow chart of an embodiment of a method for monitoring flight crew members of the present application;

[0027] Figure 4 This is a flow chart of another embodiment of the flight crew monitoring method of the present application;

[0028] Figure 5 It is a schematic diagram of an embodiment of a computer-readable storage medium of the present application.

[0029] Description of reference numerals:

[0030] 10-Monitoring equipment; 11-Camera equipment; 12-Millimeter wave radar; 13-Fiber optic sensor; 14-Cabin sound acquisition equipment; 20-Airborne core network; 21-Flight warning module; 22-Integrated processing module; 30-Data analysis module; 31-State model library; 32-Network interface; 40-Cockpit information module; 41-Cockpit warning unit; 42-Cockpit communication unit; 43-Cockpit information recording unit; 50-Computer-readable storage medium; 51-Program. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. In addition, it should be understood that the specific implementation methods described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In the present application, unless otherwise specified, the directional words used, such as "up", "down", "left", and "right", generally refer to the up, down, left, and right of the device in actual use or working state, specifically the drawing direction in the accompanying drawings.

[0032] In this application, unless otherwise specified or limited, terms such as "connected," "connected," and "stacked" should be understood broadly. For example, they may refer to fixed connections, detachable connections, or integration; they may refer to direct connections or indirect connections through an intermediate medium; they may refer to internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of these terms in this application based on the specific circumstances.

[0033] This application provides a flight crew monitoring method and system, and a computer-readable storage medium, each of which is described in detail below. It should be noted that the order in which the following embodiments are described does not limit the preferred order of the embodiments of this application. Furthermore, in the following embodiments, the description of each embodiment has its own emphasis. For portions not detailed in one embodiment, please refer to the relevant descriptions of other embodiments.

[0034] See also Figure 1 , Figure 1 It is a structural diagram of an embodiment of the monitoring system of the present application.

[0035] In one embodiment, the monitoring system includes a monitoring device 10 and an onboard core network 20. The monitoring device 10 is communicatively coupled to the onboard core network 20. The monitoring device 10 is configured to monitor the status of flight crew members and is configured so that the flight crew members do not need to wear it. The onboard core network 20 is configured to obtain monitoring status information obtained by the monitoring device 10 and determine an alerting strategy based on the monitoring status information and the aircraft's flight status information.

[0036] Through the above-described approach, this embodiment controls the monitoring device 10 to monitor the flight crew's status to obtain monitoring status information. This information, combined with the aircraft's flight status information, determines an alerting strategy, enabling flight crew status monitoring to ensure flight safety. Furthermore, the monitoring device 10 is configured to be wearable by the flight crew. This eliminates the need for the flight crew to wear the monitoring device 10, saving the flight crew time and reducing the psychological burden on the flight crew. The monitoring status information includes at least one of the flight crew's physiological characteristics, facial features, behavioral and posture information, and cabin acoustic characteristics. This embodiment monitors various factors that influence the flight crew's driving state and, based on the fusion of these multiple feature information, comprehensively processes and analyzes the feature information to accurately assess the flight crew's driving state. Furthermore, the alerting strategy is determined based on the aircraft's flight status information. The specific assessment process is described in detail below. This embodiment enables rapid, effective, and seamless assessment of the flight crew's status and is suitable for monitoring flight crew driving safety in flight scenarios, enabling the implementation of appropriate preventive measures or timely warning and intervention of abnormal driving conditions.

[0037] In one embodiment, the monitoring device 10 includes a camera 11 and a millimeter-wave radar 12. The data monitored by the camera 11 and the millimeter-wave radar 12 are configured to be assigned different weights according to a data source weight allocation strategy to determine an alarm strategy. Specifically, the data monitored by the camera 11 and the millimeter-wave radar 12 are assigned different weights according to the data source weight allocation strategy to determine the driving status of the flight crew, and then determine the alarm strategy in combination with the flight status information of the aircraft. Among them, the data source weight allocation strategy includes at least one of the characteristics of the camera 11, the facial occlusion of the flight crew, and the psychological acceptance of the camera 11 by the flight crew. In other words, this embodiment assigns different weights to different data sources (including the camera 11 and the millimeter-wave radar 12, etc.) according to at least one of the characteristics of the camera 11, the facial occlusion of the flight crew, and the psychological acceptance of the camera 11 by the flight crew.

[0038] In one embodiment, based on the characteristics of the camera device 11 and in combination with the lighting conditions of the aircraft cockpit environment, the data monitored by the camera device 11 and the millimeter wave radar 12 are assigned different weights to determine the warning strategy.

[0039] In the case where the camera device 11 is in the form of an RGB camera or the like, when the cockpit environment has good lighting conditions, the camera device 11 can obtain high-quality image information to ensure the accuracy of the monitoring results. Define the lower limit value of light intensity and the upper limit value of light intensity. When the light intensity of the cockpit environment is between the lower limit value of light intensity and the upper limit value of light intensity, it means that the current cockpit environment has good lighting conditions. Specifically, when the light intensity of the cockpit environment is between the lower limit value of light intensity and the upper limit value of light intensity, the alarm strategy is determined based on the data monitored by the camera device 11. At this time, the cockpit environment has good lighting conditions, the camera device 11 can obtain high-quality image information, and the technology for monitoring the status of the flight crew through the camera device 11 is relatively mature, which can ensure the accuracy of the monitoring results. When the light intensity of the cockpit environment is less than the lower limit value of light intensity or greater than the upper limit value of light intensity, the alarm strategy is determined based on the data monitored by the millimeter wave radar 12. At this time, the lighting conditions in the cockpit environment are poor, and the camera equipment 11 cannot obtain high-quality image information. The millimeter-wave radar 12 is less affected by the lighting conditions and can obtain monitoring status information relatively accurately, thereby ensuring the accuracy of the monitoring results.

[0040] For example, when the light intensity of the cockpit environment is between 500 and 10,000 lux, for example, it is normal daytime outside the cockpit or there is a good lighting environment inside the cockpit, then the cockpit environment has good lighting conditions. When the light intensity of the cockpit environment is lower than 500 lux, for example, it is night outside the cockpit, rainy and foggy weather, and other low-light environments. When the light intensity of the cockpit environment is higher than 10,000 lux, for example, strong direct sunlight, overexposure risk scenes, etc. The lower limit value of the light intensity can be the above-mentioned 500 lux, etc., and the upper limit value of the light intensity can be the above-mentioned 10,000 lux, etc. Of course, in other embodiments of the present application, the lower limit value of the light intensity and the upper limit value of the light intensity can be reasonably set according to the specific conditions of the camera device 11, and are not limited here.

[0041] If the camera 11 is an infrared camera, a thermal imaging camera, or the like, the camera 11 can obtain high-quality image information to ensure the accuracy of the monitoring results, even when the cockpit environment has good lighting conditions and / or the light intensity is low (for example, when the cockpit is outside at night, in rainy or foggy weather, or in other low-light environments). Specifically, when the light intensity in the cockpit environment is less than or equal to the upper limit of light intensity, the alarm strategy is determined based on the data monitored by the camera 11. When the light intensity in the cockpit environment is greater than the upper limit of light intensity, the alarm strategy is determined based on the data monitored by the millimeter-wave radar 12.

[0042] If the camera 11 is an HDR camera, a polarized camera, or the like, and the cockpit environment has good lighting conditions and / or high light intensity (e.g., strong direct sunlight, overexposure risk scenarios), the camera 11 can obtain high-quality image information to ensure the accuracy of the monitoring results. Specifically, when the light intensity in the cockpit environment is greater than or equal to the lower light intensity limit, the alarm strategy is determined based on the data monitored by the camera 11. When the light intensity in the cockpit environment is less than the lower light intensity limit, the alarm strategy is determined based on the data monitored by the millimeter-wave radar 12.

[0043] Of course, in other embodiments of the present application, corresponding weights can be assigned to the data from the camera device 11 and the millimeter-wave radar 12 in the data used to evaluate the driving status of the flight crew based on the quality of the data monitored by the camera device 11 and the millimeter-wave radar 12, and this is not limited here.

[0044] In one embodiment, based on the crew member's facial obstruction, the data monitored by the camera 11 and the millimeter-wave radar 12 are assigned different weights to determine an alert strategy. Specifically, the crew member's face includes both obscured and unobstructed areas, and the millimeter-wave radar 12 is configured to monitor at least the obscured area.

[0045] The obscured areas may be those caused by the flight crew wearing sunglasses, masks, etc., while the rest of the flight crew's face is the unobstructed areas. In this embodiment, the obscured areas can be monitored by the millimeter-wave radar 12, and the unobstructed areas can be monitored by the camera 11. Based on the data obtained by the millimeter-wave radar 12 monitoring the obscured areas and the data obtained by the camera 11 monitoring the unobstructed areas, the flight crew's driving status can be determined, and then, combined with the aircraft's flight status information, an alert strategy can be determined. On the one hand, the technology used by the camera 11 to monitor the flight crew's status is relatively mature and can ensure the accuracy of the monitoring results. On the other hand, the camera 11 cannot image the obscured areas, so the millimeter-wave radar 12 is required to monitor the obscured areas. In this embodiment, the camera 11 and the millimeter-wave radar 12 cooperate to monitor the flight crew's status based on the obstruction of the flight crew's face, ensuring the accuracy of the monitoring results.

[0046] It is understandable that if the faces of the flight crew members are not obscured, different weights can be assigned to the data monitored by the camera device 11 and the millimeter-wave radar 12 according to the quality of the data monitored by the camera device 11 and the millimeter-wave radar 12 to determine the driving status of the flight crew members, and then combined with the flight status information of the aircraft to determine the warning strategy.

[0047] In one embodiment, based on the flight crew's psychological acceptance of the camera device 11, that is, based on whether the camera device 11 is in a usable state, the data monitored by the camera device 11 and the millimeter wave radar 12 are assigned different weights to determine the warning strategy.

[0048] In the case where the flight crew has a low psychological acceptance of the camera device 11, the flight crew may manually turn off the camera device 11, causing the camera device 11 to be in a stopped state. At this time, the camera device 11 is unavailable, so the alarm strategy is determined based on the data monitored by the millimeter-wave radar 12 to ensure that the status of the flight crew can be monitored in real time. In the case where the flight crew has a high psychological acceptance of the camera device 11, the camera device 11 is often in a working state. At this time, the camera device 11 is available, and the driving status of the flight crew can be determined based on the data monitored by the camera device 11 and the millimeter-wave radar 12 in the manner described in the above embodiment, and then the alarm strategy is determined in combination with the flight status information of the aircraft.

[0049] It should be noted that the data obtained by the camera 11 and millimeter-wave radar 12 from monitoring the flight crew includes facial feature information and / or behavioral posture information, which assists in assessing the flight crew's piloting status. Facial feature information may include: lip status, such as lip tightness, lip opening, and lip breathing; and / or eyebrow status, such as the distance between eyebrows, average frequency of eyebrow convergence, and eyebrow position; and / or eye status, such as whether the eyes are closed, average blink frequency, and eye gaze direction. Behavioral posture information may include at least one of body posture, hand gestures, and head movements.

[0050] In one embodiment, the monitoring device 10 further includes a fiber optic sensor 13 configured to monitor physiological characteristics of the flight crew. The fiber optic sensor 13 collects and analyzes physiological characteristics that affect the flight crew's driving state, such as their health, fatigue, mood, and attention span, to accurately assess their driving status and ensure flight safety. Physiological characteristics may include at least one of respiratory rate and heart rate, among others. The fiber optic sensor 13 may be built into a seat cushion or seat back, for example.

[0051] In one embodiment, the monitoring device 10 further includes a cockpit sound acquisition device 14, which is configured to monitor audio information in the cockpit environment to obtain cockpit sound characteristic information. This embodiment uses cockpit sound characteristic information to assist in assessing the flight crew's driving status and thereby ensure flight safety. The cockpit sound characteristic information may include at least one of the following audio information types: sounds emitted by cockpit equipment and ATC (Air Traffic Control) voices.

[0052] In one embodiment, the monitoring system further includes a data analysis module 30. The data analysis module 30 includes a state model library 31 and a network interface 32. The state model library 31 is connected to the monitoring device 10. The state model library 31 is used to receive the monitoring state information obtained by the monitoring device 10, and compare the received monitoring state information with the data model in the state model library 31 to determine the driving state of the flight crew, and then evaluate whether the flight crew is in an abnormal driving state. The state model library 31 is also connected to the airborne core network 20 through the network interface 32, and the information that the flight crew is in an abnormal driving state is transmitted to the airborne core network 20 through the network interface 32, and then the airborne core network 20 determines the alarm strategy in combination with the flight state information of the aircraft.

[0053] For example, the monitoring status information may include physiological characteristics, facial characteristics, behavioral posture information, and cabin voice characteristics of the flight crew. The collected physiological characteristics, facial characteristics, behavioral posture information, and cabin voice characteristics information are compared with the data models in the state model library 31 to obtain information such as the crew's health status, fatigue status, emotional state, and behavioral movements, so as to assess whether the flight crew is in an abnormal driving state. If the flight crew is in an abnormal driving state, intervention measures are taken. Abnormal driving states include at least one of the following abnormal states:

[0054] ① Abnormal health conditions, such as cardiac arrest or other conditions that affect the normal actions of the flight crew;

[0055] ② Abnormal fatigue state, such as flight crew members falling asleep or other conditions that affect the flight crew members' performance of their duties;

[0056] ③ Abnormal emotional state, such as extreme emotional instability of the flight crew, which affects the flight crew's rational decision-making;

[0057] ④ Abnormal behavior, such as behavioral errors by the flight crew that affect the flight crew’s status in performing their mission.

[0058] In one embodiment, the airborne core network 20 includes a flight alert module 21 and a comprehensive processing module 22. The flight alert module 21 is configured to obtain the flight crew's driving status and flight status information, where the flight crew's driving status is obtained by analyzing monitoring status information. The comprehensive processing module 22 is configured to combine the flight status information to determine an alert strategy when the flight crew is in an abnormal driving state.

[0059] The abnormal driving state is divided into at least a first-level abnormal driving state and a second-level abnormal driving state according to the degree of impact on flight safety. Among them, the first-level abnormal driving state has a lower degree of impact on flight safety than the second-level abnormal driving state. For example, the first-level abnormal driving state slightly affects flight safety, and the adverse effects can be eliminated in a relatively short time by reminding the flight crew to make corrections, such as abnormal driving states such as flight crew misoperation and failure to conduct approach briefings; the second-level abnormal driving state affects flight safety, and the adverse effects can be eliminated within a period of time through a series of intervention measures (such as limiting some of the flight crew's authority, and then restoring their authority after the flight crew returns to normal driving status), such as abnormal driving states such as flight crew slight emotional instability and flight crew falling asleep. Of course, in other embodiments of the present application, more levels of abnormal driving states can be divided according to actual needs, and this is not limited here.

[0060] In one embodiment, the monitoring system further includes a cockpit information module 40, which is communicatively connected to the onboard core network 20. The cockpit information module 40 is configured to intervene in the status of the flight crew based on the warning strategy determined by the onboard core network 20. The cockpit information module 40 includes a cockpit warning unit 41, a cockpit communication unit 42, and a cockpit information recording unit 43. The cockpit warning unit 41 may include a display device and / or an audio device, the display device may output warning information in the form of images, and the audio device may output warning information in the form of audio. The cockpit communication unit 42 may communicate with a ground workstation to monitor the movement of the aircraft. The cockpit information recording unit 43 is configured to record status information of the flight crew.

[0061] In the above embodiment, the abnormal driving state is divided into a first-level abnormal driving state and a second-level abnormal driving state. If the flight crew is in the first-level abnormal driving state, a first warning strategy is determined. If the flight crew is in the second-level abnormal driving state, a second warning strategy is determined. If the flight crew is in the first-level abnormal driving state, the first warning strategy may include the cockpit warning unit 41 outputting a warning message in the form of images and / or audio to prompt the flight crew to make corrections. If the flight crew is in the second-level abnormal driving state, the second warning strategy may include the cockpit warning unit 41 outputting a warning message in the form of images and / or audio to prompt the flight crew to make corrections, and the cockpit communication unit 42 communicating with the ground workstation to conduct ATC (Air Traffic Control) communication to monitor the aircraft's movements until the flight crew returns to normal driving status or the abnormal driving state is reduced to the first-level abnormal driving state.

[0062] Please also refer to Figure 2 The flight status information includes the flight phase of the aircraft, which includes critical phases and non-critical phases. For example, the flight phase of an aircraft generally includes 10 phases, and the starting and ending points of each phase are: starting the power supply, starting the first engine, the first engine reaching takeoff power, the airspeed reaching 80kt, taking off, climbing to 1500ft, descending to 800ft, touching down, the airspeed dropping to 80kt, shutting down the last engine, and 5 minutes later. Among them, the critical phases include phases 3, 4, 5, 7, and 8. According to the history of aviation operations to date, 70%-80% of air accidents occur in critical phases. In order to reasonably allocate the attention of the flight crew and reduce the workload of the flight crew, when the flight crew is in a level 1 abnormal driving state and the aircraft is in a critical phase, the warning strategy includes canceling the intervention measures corresponding to the level 1 abnormal driving state. The embodiment of the present application determines different warning strategies after judging and voting based on different flight phases and aircraft states.

[0063] See also Figure 3 , Figure 3 It is a flowchart of an embodiment of the flight crew monitoring method of the present application.

[0064] S101: Controlling a monitoring device to monitor the status of a flight crew member to obtain monitoring status information, wherein the monitoring device is configured to be unnecessary for the flight crew member to wear.

[0065] In this embodiment, the flight crew's status is monitored by controlling the monitoring device 10 to obtain monitoring status information. This monitoring status information is combined with the aircraft's flight status information to determine an alerting strategy, enabling monitoring of the flight crew's status to ensure flight safety. Furthermore, the monitoring device 10 is configured to be wearable by the flight crew. This eliminates the need for the flight crew to wear the monitoring device 10, saving them time and reducing their psychological burden.

[0066] S102: Determine an alert strategy based on the monitoring status information and in combination with the aircraft's flight status information.

[0067] In this embodiment, after the monitoring status information is obtained by the monitoring device 10, an alarm strategy is determined in combination with the flight status information of the aircraft. The alarm strategy can intervene in the status of the flight crew and monitor the status of the flight crew to ensure flight safety.

[0068] See also Figure 4 , Figure 4 It is a flowchart of another embodiment of the flight crew monitoring method of the present application.

[0069] S201: Controlling monitoring equipment to monitor the status of flight crew members to obtain monitoring status information.

[0070] In this embodiment, the flight crew's status is monitored by controlling the monitoring device 10 to obtain monitoring status information. This monitoring status information is combined with the aircraft's flight status information to determine an alerting strategy, enabling monitoring of the flight crew's status to ensure flight safety. Furthermore, the monitoring device 10 is configured to be wearable by the flight crew. This eliminates the need for the flight crew to wear the monitoring device 10, saving them time and reducing their psychological burden.

[0071] Monitoring status information includes physiological characteristics, facial features, behavioral posture information, and cabin acoustic characteristics of the flight crew. Facial and behavioral posture information is obtained through the camera 11 and millimeter-wave radar 12. Facial characteristic information may include: lip status, such as lip tightness, lip opening, and lip breathing movements; eyebrow status, such as the distance between eyebrows, average eyebrow convergence frequency, and eyebrow position; and eye status, such as whether the eyes are closed, average blink frequency, and eye gaze direction. Behavioral posture information may include body posture, gestures, and head movements. Physiological characteristic information is obtained through the fiber optic sensor 13. Physiological characteristic information may include respiratory rate and heart rate. Cabin acoustic characteristics information is obtained through the cabin acoustic acquisition device 14. Cabin acoustic characteristics information may include sounds emitted by equipment in the cockpit and ATC voice.

[0072] S202: Compare the monitoring status information with the data model in the status model library to determine the driving status of the flight crew.

[0073] In this embodiment, the data analysis module 30 receives monitoring status information from the monitoring device 10 and analyzes and processes the monitoring status information. Specifically, the flight crew's driving status is determined based on the monitoring status information. The monitoring status information is compared with the data models in the state model library 31 to determine the flight crew's driving status. The monitored physiological characteristics, facial features, behavioral posture information, and cabin voice characteristics are compared with the data models in the state model library 31 to obtain the flight crew's health status, fatigue status, emotional state, and behavioral actions, etc., based on which the flight crew's driving status is determined.

[0074] Furthermore, according to a data source weighting strategy, the flight crew's driving status is determined based on data monitored by the camera 11 and / or millimeter-wave radar 12, and an alert strategy is then determined in combination with the flight status information. The data source weighting strategy includes at least one of the characteristics of the camera 11, the crew's facial obstruction, and the crew's psychological acceptance of the camera 11.

[0075] The data monitored by the camera device 11 and the millimeter-wave radar 12 are configured to be assigned different weights based on a data source weight allocation strategy to determine an alert strategy. The data source weight allocation strategy includes at least one of the characteristics of the camera device 11, the facial obstruction of the flight crew, and the psychological acceptance of the camera device 11 by the flight crew. In other words, this embodiment assigns different weights to different data sources (including the camera device 11 and the millimeter-wave radar 12, etc.) based on at least one of the characteristics of the camera device 11, the facial obstruction of the flight crew, and the psychological acceptance of the camera device 11 by the flight crew.

[0076] Based on the characteristics of the camera device 11 and in combination with the lighting conditions of the aircraft cockpit environment, the data monitored by the camera device 11 and the millimeter-wave radar 12 are assigned different weights to determine the alarm strategy. In the case where the camera device 11 is in the form of an RGB camera, etc., the light intensity of the cockpit environment is determined. When the light intensity of the cockpit environment is between the lower light intensity limit and the upper light intensity limit, the alarm strategy is determined based on the data monitored by the camera device 11. When the light intensity of the cockpit environment is less than the lower light intensity limit or greater than the upper light intensity limit, the alarm strategy is determined based on the data monitored by the millimeter-wave radar 12. In the case where the camera device 11 is in the form of an infrared camera, a thermal imaging camera, etc., the light intensity of the cockpit environment is determined. When the light intensity of the cockpit environment is less than or equal to the upper light intensity limit, the alarm strategy is determined based on the data monitored by the camera device 11. When the light intensity of the cockpit environment is greater than the upper light intensity limit, the alarm strategy is determined based on the data monitored by the millimeter-wave radar 12. If the camera 11 is an HDR camera, a polarization camera, or the like, the cockpit ambient light intensity is determined. When the cockpit ambient light intensity is greater than or equal to the lower limit, an alert strategy is determined based on the data monitored by the camera 11. When the cockpit ambient light intensity is less than the lower limit, an alert strategy is determined based on the data monitored by the millimeter-wave radar 12. Of course, the data from the camera 11 and the millimeter-wave radar 12 can also be weighted in the data used to assess the flight crew's driving status based on the quality of the data monitored by the camera 11 and the millimeter-wave radar 12, but this is not limited here.

[0077] Based on the occlusion of the flight crew's face, the data monitored by the camera device 11 and the millimeter-wave radar 12 are assigned different weights to determine the warning strategy. In this embodiment, the millimeter-wave radar 12 can monitor the blocked part and the camera device 11 can monitor the unblocked part. Based on the data obtained by the millimeter-wave radar 12 monitoring the blocked part and the data obtained by the camera device 11 monitoring the unblocked part, the flight crew's driving status is determined, and then the warning strategy is determined in combination with the flight status information of the aircraft. If the flight crew's face is not blocked, different weights can be assigned to the data monitored by the camera device 11 and the millimeter-wave radar 12 based on the quality of the data monitored by the camera device 11 and the millimeter-wave radar 12 to determine the flight crew's driving status, and then the warning strategy is determined in combination with the flight status information of the aircraft.

[0078] Based on the flight crew's psychological acceptance of the camera device 11, that is, based on whether the camera device 11 is in an available state, the data monitored by the camera device 11 and the millimeter-wave radar 12 are assigned different weights to determine the alarm strategy. In the case that the flight crew has a low psychological acceptance of the camera device 11, the flight crew may manually turn off the camera device 11, causing the camera device 11 to be in a stopped state. At this time, the camera device 11 is not available, so the alarm strategy is determined based on the data monitored by the millimeter-wave radar 12 to ensure that the status of the flight crew can be monitored in real time. In the case that the flight crew has a high psychological acceptance of the camera device 11, the camera device 11 is often in a working state. At this time, the camera device 11 is available, and the driving status of the flight crew can be determined based on the data monitored by the camera device 11 and the millimeter-wave radar 12 in the manner described in the above embodiment, and then the alarm strategy is determined in combination with the flight status information of the aircraft.

[0079] S203: Determine whether the flight crew is in an abnormal driving state.

[0080] In this embodiment, if the flight crew is in an abnormal driving state, step S204 is executed; if the flight crew is not in an abnormal driving state, step S207 is executed.

[0081] The collected physiological characteristic information, facial characteristic information, behavioral posture information, and cabin voice characteristic information are compared with the data models in the state model library 31 to obtain information such as the crew member's health status, fatigue status, emotional state, and behavioral movements, so as to assess whether the flight crew member is in an abnormal driving state. If the flight crew member is in an abnormal driving state, intervention measures are taken. Abnormal driving state includes at least one of the following abnormal states:

[0082] ① Abnormal health conditions, such as cardiac arrest or other conditions that affect the normal actions of the flight crew;

[0083] ② Abnormal fatigue state, such as flight crew members falling asleep or other conditions that affect the flight crew members' performance of their duties;

[0084] ③ Abnormal emotional state, such as extreme emotional instability of the flight crew, which affects the flight crew's rational decision-making;

[0085] ④ Abnormal behavior, such as behavioral errors by the flight crew that affect the flight crew’s status in performing their mission.

[0086] S204: Communicate with the flight warning module.

[0087] In this embodiment, after determining that the flight crew is in an abnormal driving state, the flight warning module 21 communicates with the flight warning module 21 and transmits information indicating that the flight crew is in an abnormal driving state to the flight warning module 21. Furthermore, the flight warning module 21 obtains the aircraft's flight status information, which includes the flight phase and the status of various aircraft systems.

[0088] S205: Determine an alert strategy based on the flight status information.

[0089] In this embodiment, the comprehensive processing module 22 comprehensively analyzes and processes the abnormal driving status of the flight crew and the various system information of the aircraft, and determines the warning strategy in combination with the flight phase to intervene in the status of the flight crew and monitor the status of the flight crew to ensure flight safety.

[0090] Abnormal driving conditions are categorized into at least Level 1 and Level 2 based on their impact on flight safety. Level 1 abnormal driving conditions have a lower impact on flight safety than Level 2. For example, a Level 1 abnormal driving condition has a minor impact on flight safety, and the adverse impact can be quickly eliminated by alerting the flight crew to make corrections. Examples include abnormal driving conditions such as flight crew misoperation or failure to conduct an approach briefing. Level 2 abnormal driving conditions impact flight safety, and the adverse impact can be eliminated over a period of time through a series of intervention measures (such as restricting some flight crew privileges and restoring them after the flight crew returns to normal driving conditions). Examples include abnormal driving conditions such as slight emotional instability or falling asleep. If the flight crew is in a Level 1 abnormal driving condition, a first alert strategy is determined. If the flight crew is in a Level 2 abnormal driving condition, a second alert strategy is determined. The first alert strategy is different from the second alert strategy.

[0091] S206: Intervene in the flight crew's status according to the warning strategy.

[0092] In this embodiment, according to the warning strategy, the intervention in the status of the flight crew is specifically: if the flight crew is in a level one abnormal driving state, the first warning strategy may include the cockpit warning unit 41 outputting warning information in image form and / or audio form, prompting the flight crew to make corrections; if the flight crew is in a level two abnormal driving state, the second warning strategy may include the cockpit warning unit 41 outputting warning information in image form and / or audio form, the warning information carries information that the flight crew is in a level two abnormal driving state, prompting the flight crew to make corrections, and the cockpit communication unit 42 communicates with the ground workstation to conduct ATC communication to monitor the aircraft's movements until the flight crew returns to a normal driving state or is downgraded to a level one abnormal driving state.

[0093] Furthermore, flight phases include critical phases and non-critical phases. If the flight crew is in a Level 1 abnormal flight state and the aircraft is in a critical phase, the first alert strategy includes canceling the intervention measures corresponding to the Level 1 abnormal flight state. This effectively allocates the flight crew's attention and reduces their workload.

[0094] S207: Record the flight crew status information.

[0095] It should be noted that the advantages of this embodiment are: First, the detection method based on facial features and behavioral posture features of the millimeter-wave radar 12 can be unrestricted by lighting conditions and non-metallic facial obstructions (such as sunglasses, masks, etc.); second, the equipment of the monitoring system is deployed in the cockpit environment, reducing the intervention of the flight crew in the detection phase and interfering with the flight crew's work as much as possible; third, by interconnecting with the airborne avionics network system to obtain flight information, an early warning trigger mechanism and logic are proposed to adapt to the flight scenario; fourth, the fiber optic seat cushion can be integrated with the seat, and the millimeter-wave radar 12 is compact, making the entire system light and easy to integrate with the existing cockpit system; fifth, a multi-source data fusion logic is proposed, which can give full play to the advantages of different sensors and equipment and avoid their disadvantages as much as possible; sixth, based on the various flight crew states that affect the driving state, a high-dimensional driving state classification logic is proposed.

[0096] See also Figure 5 , Figure 5 It is a schematic diagram of an embodiment of a computer-readable storage medium of the present application.

[0097] In one embodiment, the computer-readable storage medium 50 stores a program 51 , which can be executed to implement the flight crew monitoring method described in the above embodiments.

[0098] The functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a computer-readable storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned computer-readable storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0099] The above is a detailed introduction to the flight crew monitoring method and system, and the computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. At the same time, for those skilled in the art, based on the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for monitoring flight crew members, characterized in that: The monitoring method comprises: controlling a monitoring device to monitor a status of a flight crew member to obtain monitoring status information, wherein the monitoring device is configured to be unnecessary for the flight crew member to wear; Based on the monitoring status information and in combination with the flight status information of the aircraft, an alarm strategy is determined.

2. The monitoring method according to claim 1, characterized in that The monitoring equipment includes a camera and a millimeter-wave radar; The step of determining an alert strategy based on the monitoring status information and in combination with the flight status information of the aircraft includes: Determining an alert strategy based on the data source weight distribution strategy, the data monitored by the camera device and / or the millimeter-wave radar, and the flight status information; The data source weight allocation strategy includes at least one of the characteristics of the camera device, facial occlusion of the flight crew, and psychological acceptance of the flight crew to the camera device.

3. The monitoring method according to claim 2, characterized in that: The step of determining the warning strategy according to the data source weight distribution strategy, based on the data monitored by the camera device and / or the millimeter wave radar, and combined with the flight status information includes: Determine the lighting intensity of the cockpit environment; When the light intensity of the cockpit environment is between a lower light intensity limit and an upper light intensity limit, determining an alarm strategy based on data monitored by the camera device; and when the light intensity of the cockpit environment is less than the lower light intensity limit or greater than the upper light intensity limit, determining an alarm strategy based on data monitored by the millimeter wave radar; or When the light intensity of the cockpit environment is less than or equal to the light intensity upper limit, determining the warning strategy based on the data monitored by the camera device, and when the light intensity of the cockpit environment is greater than the light intensity upper limit, determining the warning strategy based on the data monitored by the millimeter wave radar; or When the light intensity of the cockpit environment is greater than or equal to the lower limit of the light intensity, the alarm strategy is determined based on the data monitored by the camera device, and when the light intensity of the cockpit environment is less than the lower limit of the light intensity, the alarm strategy is determined based on the data monitored by the millimeter wave radar.

4. The monitoring method according to claim 2, characterized in that: The step of determining the warning strategy according to the data source weight distribution strategy, based on the data monitored by the camera device and / or the millimeter wave radar, and combined with the flight status information includes: Determine the obscured and unobscured areas of the flight crew members' faces; Based on the data obtained by the camera device monitoring the unobstructed part and the data obtained by the millimeter wave radar monitoring the obstructed part, facial feature information of the flight crew is obtained to determine an alarm strategy.

5. The monitoring method according to claim 2, characterized in that: The step of determining the warning strategy according to the data source weight distribution strategy, based on the data monitored by the camera device and / or the millimeter wave radar, and combined with the flight status information includes: It is determined that the camera device is in a stopped working state, and an alarm strategy is determined based on the data monitored by the millimeter wave radar in combination with the flight status information.

6. The monitoring method according to claim 1, characterized in that: The monitoring device includes an optical fiber sensor; The step of controlling the monitoring device to monitor the status of the flight crew comprises: The optical fiber sensor is controlled to monitor physiological characteristic information of the flight crew.

7. The monitoring method according to claim 1, characterized in that: The monitoring equipment includes cabin sound collection equipment; The step of controlling the monitoring device to monitor the status of the flight crew comprises: The cockpit sound collection device is controlled to monitor the audio information in the cockpit environment to obtain cockpit sound feature information.

8. The monitoring method according to claim 1, characterized in that: The monitoring status information includes at least one of physiological characteristic information, facial characteristic information, behavioral posture information, and cabin voice characteristic information of the flight crew; The step of determining an alert strategy based on the monitoring status information and in combination with the flight status information of the aircraft includes: determining a flight crew member's driving status based on the monitoring status information; When the flight crew is in an abnormal driving state, the warning strategy is determined in combination with the flight state information.

9. The monitoring method according to claim 8, characterized in that: The abnormal driving state is divided into a first-level abnormal driving state and a second-level abnormal driving state according to the degree of impact on flight safety, wherein the first-level abnormal driving state has a lower impact on flight safety than the second-level abnormal driving state; The step of determining the warning strategy in combination with the flight status information when the flight crew is in an abnormal driving state includes: If the flight crew is in the first level abnormal driving state, a first warning strategy is determined; if the flight crew is in the second level abnormal driving state, a second warning strategy is determined; wherein the first warning strategy is different from the second warning strategy.

10. The monitoring method according to claim 9, characterized in that: The flight status information includes the flight phase of the aircraft, and the flight phase includes a critical phase and a non-critical phase; If the flight crew is in the first level abnormal driving state, the step of determining the first warning strategy further includes: When the flight crew is in the first-level abnormal driving state and the aircraft is in the critical phase, the first warning strategy includes canceling the intervention measures corresponding to the first-level abnormal driving state.

11. The monitoring method according to claim 9, characterized in that: The second warning strategy includes outputting warning information in image form and / or audio form, and communicating with a ground workstation to monitor the aircraft movement; wherein the warning information carries information that the flight crew is in the second-level abnormal driving state.

12. A monitoring system, characterized in that: include: A monitoring device (10) for monitoring the status of a flight crew member, wherein the monitoring device (10) is configured to not be worn by the flight crew member; An onboard core network (20) is in communication with the monitoring device (10), and the onboard core network (20) is configured to determine an alert strategy based on the monitoring status information and in combination with the flight status information of the aircraft.

13. The monitoring system according to claim 12, characterized in that The monitoring device (10) includes a camera device (11) and a millimeter-wave radar (12), and the data monitored by the camera device (11) and the millimeter-wave radar (12) are configured to be assigned different weights according to a data source weight allocation strategy to determine an alarm strategy, wherein the data source weight allocation strategy includes at least one of the characteristics of the camera device (11), the facial occlusion of the flight crew, and the psychological acceptance of the flight crew to the camera device (11).

14. The monitoring system according to claim 13, characterized in that The face of the flight crew member includes an obstructed portion and an unobstructed portion, and the millimeter wave radar (12) is configured to monitor at least the obstructed portion.

15. The monitoring system according to claim 12, wherein: The monitoring device (10) includes a fiber optic sensor (13) configured to monitor physiological characteristic information of a flight crew member.

16. The monitoring system according to claim 12, wherein: The onboard core network (20) comprises: A flight warning module (21) is used to obtain the flight crew's driving status and the flight status information, wherein the flight crew's driving status is obtained by analyzing the monitoring status information; and A comprehensive processing module (22) is configured to combine the flight status information to determine a warning strategy when the flight crew is in an abnormal driving state.

17. The monitoring system according to claim 16, characterized in that The abnormal driving state is divided into a first-level abnormal driving state and a second-level abnormal driving state according to the degree of impact on flight safety, wherein the first-level abnormal driving state has a lower impact on flight safety than the second-level abnormal driving state; the flight state information includes the flight phase of the aircraft, and the flight phase includes a critical phase and a non-critical phase; In which, when the flight crew is in the first-level abnormal driving state and the aircraft is in the critical stage, the warning strategy includes canceling the intervention measures corresponding to the first-level abnormal driving state.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, which can be executed to implement the flight crew monitoring method according to any one of claims 1 to 11.

Images