Aircraft alarm output method and aircraft alarm control device

Abstract

The present invention provides an aircraft alarm output method and an aircraft alarm control device that can suppress the development of habituation to alarms among aircraft crew members. [Solution] This aircraft alarm output method includes the steps of: acquiring a crew image 40 including the crew 100 of the aircraft 90; estimating the skeletal state 42 of the crew 100 based on the acquired crew image 40; and, in the control to output an alarm to the crew 100 of the aircraft 90, performing control to suppress the alarm output based on the estimated skeletal state 42 of the crew 100.

Description

Technical Field

[0001] The present invention relates to an aircraft warning output method and an aircraft warning control device.

Background Art

[0002] Conventionally, an aircraft warning output method has been known (for example, Patent Document 1).

[0003] In the above Patent Document 1, a lidar device mounted on an aircraft or the like, which detects an obstacle and issues a warning, is disclosed. This lidar device includes a target processing unit that detects a linear obstacle such as a power transmission line in a monitoring area. When a linear obstacle is detected, the target processing unit outputs a warning indicating that fact by voice or the like.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Although not disclosed in the above Patent Document 1, for example, in rescue activities using an aircraft, etc., there may be cases where the aircraft continues to fly approaching a plurality of structures and natural shapes that may become obstacles to flight. In this case, each of the plurality of structures and natural shapes, etc. is detected as an obstacle, and a plurality of warnings based on each of the detected plurality of obstacles are output. Here, the inventor of the present application has found that a kind of habituation to the warnings may occur in the aircraft passengers due to the warnings being continuously output even for obstacles that the aircraft passengers are aware of. Therefore, it is desired to be able to suppress the occurrence of habituation to the warnings in the aircraft passengers.

[0006] This invention was made to solve the above-mentioned problems, and one of its objectives is to provide an aircraft alarm output method and an aircraft alarm control device that can suppress the development of habituation to alarms among aircraft crew members. [Means for solving the problem]

[0007] An aircraft warning output method comprising the steps of: acquiring crew images including the aircraft crew; estimating the skeletal state of the crew based on the acquired crew images; and, in control of outputting a warning to the aircraft crew, performing control to suppress the warning output based on the estimated skeletal state of the crew. The aircraft alarm control device also includes a camera unit that captures images of the crew, including the aircraft crew; a skeleton estimation unit that estimates the skeletal state of the crew based on the captured images of the crew; and an output control unit that, in the control of outputting an alarm to the aircraft crew, performs control to suppress the alarm output based on the estimated skeletal state of the crew. [Effects of the Invention]

[0008] In the above-described aircraft warning output method, the state of the crew member's skeleton is estimated using acquired crew member images. Based on the estimated state of the crew member's skeleton, it is possible to determine whether or not the crew member is currently in a state where they should be performing an action. Therefore, if it is determined that the crew member is currently in a state where they should be performing an action based on the estimated state of their skeleton, the warning output can be suppressed. In other words, if the aircraft crew member is aware of the cause of the warning, the warning output for that cause may be suppressed. As a result, it is possible to prevent the aircraft crew member from becoming accustomed to the warning. Furthermore, since the state of the crew member's skeleton is estimated using acquired crew member images, the crew member does not need to wear additional equipment such as inertial sensors to detect the position of various parts of their body. Therefore, the burden on the crew member does not increase compared to when additional equipment for detecting the position of various parts of the body is worn. Thus, it is possible to prevent the crew member from becoming accustomed to the warning without increasing the burden on the aircraft crew member. Furthermore, the above-mentioned aircraft warning control system estimates the state of the crew member's skeleton using acquired crew member images, and based on the estimated state of the crew member's skeleton, it can determine whether or not the crew member is currently in a state where they should be performing an action. Therefore, if it is determined that the crew member is currently in a state where they should be performing an action based on the estimated state of their skeleton, the warning output can be suppressed. In other words, if the aircraft crew member is aware of the cause of the warning, the output of the warning for that cause can be suppressed. As a result, it is possible to prevent the aircraft crew member from becoming accustomed to the warning. In addition, since the state of the crew member's skeleton is estimated using acquired crew member images, the crew member does not need to wear additional equipment such as inertial sensors to detect the position of each part of their body. Therefore, the burden on the crew member does not increase compared to when additional equipment for detecting the position of each part of the body is worn. Thus, it is possible to prevent the crew member from becoming accustomed to the warning without increasing the burden on the aircraft crew member. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing an example of an aircraft in flight equipped with the warning control device according to this embodiment. [Figure 2] This is a block diagram showing an example of an alarm control device according to this embodiment. [Figure 3] This is a schematic diagram showing an example of a pilot image. [Figure 4] This is a schematic diagram illustrating the information that triggers alarms. [Figure 5] This is a schematic diagram showing an example of a skeletal structure image. [Figure 6] This figure illustrates the generation of a pre-trained model and the acquisition of skeletal state images using the pre-trained model. [Figure 7] This is a schematic diagram illustrating an example of how pilot posture is obtained from skeletal state images. [Figure 8] This is a schematic diagram illustrating another example of acquiring the pilot's posture in skeletal state imaging. [Figure 9] This is a flowchart illustrating the alarm output control process performed by the control unit. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments embodying the present invention will be described based on the drawings.

[0011] The alarm control device 1 according to this embodiment will be described with reference to Figures 1 and 2.

[0012] (Overview of alarm control system) As shown in Figure 1, the alarm control device 1 (see Figure 2) is installed in an aircraft 90, which includes fixed wings or rotorcraft. The alarm control device 1 is installed in an aircraft 90, which includes rotorcraft, for example, a helicopter, which is used for rescue operations. The aircraft 90 is also equipped with an alarm information generation device 2 and an alarm device 3.

[0013] In a rescue operation using an aircraft 90, a warning is issued from the warning device 3 under the control of the warning control device 1 to make the pilot 100 (see Figure 3) aware of the target of rescue 80, such as distressed persons or injured persons who need to be rescued. In addition, in a rescue operation using an aircraft 90, a warning is issued from the warning device 3 under the control of the warning control device 1 to make the pilot 100 aware of structures 81, natural features 82, and disasters 83 that may obstruct flight. The pilot 100 is an example of a "crew member" in the claims.

[0014] Regarding the rescue target 80 and the warning target 51 as an obstacle to flight (refer to FIG. 4) described above, a plurality of warnings based on each of the plurality of warning targets 51 are successively output, and thus there is a possibility that the pilot 100 of the aircraft 90 may become accustomed to the warnings. Therefore, in order to suppress the pilot 100 of the aircraft 90 from becoming accustomed to the warnings, the warning control device 1 is configured to perform control not to output a warning in a predetermined case in the control of outputting a warning to the pilot 100 of the aircraft 90. Details will be described later, but the warning control device 1 performs control not to output a warning based on the estimated state 42 of the skeleton of the pilot 100 (refer to FIG. 5).

[0015] (Configuration of Warning Control Device) As shown in FIG. 2, the warning control device 1 includes an in-cabin image capturing unit 10, a warning control device side communication unit 11, a storage unit 12, and a control unit 13. The warning control device 1 is provided in the aircraft 90. Note that the in-cabin image capturing unit 10 is an example of the "image capturing unit" in the claims.

[0016] The in-cabin image capturing unit 10 is configured to acquire a pilot image 40 (refer to FIG. 3) including the pilot 100 who operates the aircraft 90. The in-cabin image capturing unit 10 is, for example, an optical camera. The in-cabin image capturing unit 10 acquires a pilot image 40 including at least the head and arms of the pilot 100 sitting in the pilot seat and the operating device 91 of the aircraft 90 arranged around the pilot 100. One in-cabin image capturing unit 10 is provided in the cockpit of the aircraft 90. In the present embodiment, the in-cabin image capturing unit 10 is provided to the left of the pilot 100. Note that the pilot image 40 is an example of the "passenger image" in the claims.

[0017] In addition, in the present embodiment, although one in-cabin image capturing unit 10 is provided in the cockpit, a plurality of in-cabin image capturing units 10 may be provided in the cockpit. In this case, each of the plurality of in-cabin image capturing units 10 is installed at a position where images with different shooting ranges and different shooting directions with respect to the pilot 100 can be acquired. Then, based on the plurality of pilot images 40 acquired by each of the plurality of in-cabin image capturing units 10, the state 42 of the skeleton of the pilot 100 (see FIG. 5), which will be described later, is estimated.

[0018] The warning control device side communication unit 11 is configured to communicate with the warning information generation device side communication unit 23 of the aircraft 90, which will be described later. The warning control device side communication unit 11 receives warning information 50 from the warning information generation device side communication unit 23, which will be described later. The warning control device side communication unit 11 receives the warning information 50 from the warning information generation device side communication unit 23 based on the generation timing of the warning information 50 by the warning information generation device 2. Note that the reception timing of the warning information 50 from the warning information generation device side communication unit 23 by the warning control device side communication unit 11 is not particularly limited, and it may be configured to receive the warning information 50 from the warning information generation device side communication unit 23 every control cycle.

[0019] In addition, the warning control device side communication unit 11 is configured to communicate with the warning device side communication unit 31 of the warning device 3, which will be described later. The warning control device side communication unit 11 transmits a control signal for causing the warning unit 30 of the warning device 3 to output a warning to the warning device side communication unit 31.

[0020] The warning control device side communication unit 11 is configured to perform wireless communication with each of the warning information generation device side communication unit 23 and the warning device side communication unit 31. As the wireless communication, for example, Bluetooth (registered trademark) communication, Wi-Fi communication, or the like is used. Note that the wireless communication method between the warning control device side communication unit 11 and each of the warning information generation device side communication unit 23 and the warning device side communication unit 31 is not particularly limited. In addition, the warning control device side communication unit 11 may be configured to perform wired communication with each of the warning information generation device side communication unit 23 and the warning device side communication unit 31.

[0021] The memory unit 12 stores various programs executed by the control unit 13. The memory unit 12 also stores the trained model 4. As will be described in detail later, as shown in Figure 5, the control unit 13 uses the trained model 4 as input to the pilot image 40 acquired by the in-flight image acquisition unit 10, and obtains the skeletal state 42 of the pilot 100 as an output result. Based on the skeletal state 42 of the pilot 100 estimated using the trained model 4, it performs control to prevent the output of an alarm. The memory unit 12 includes, for example, a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0022] As shown in Figure 2, the control unit 13 is configured to control the aircraft 90's pilot 100 to output a warning. The control unit 13 is also configured to estimate the skeletal state 42 (see Figure 5) of the pilot 100 based on the pilot image 40 (see Figure 3) acquired by the in-flight image acquisition unit 10. The control unit 13 is also configured to control the aircraft so as not to output a warning based on the estimated skeletal state 42 of the pilot 100. Specifically, the control unit 13 is configured to control the aircraft so as not to output a warning if it estimates, based on the estimated skeletal state 42 of the pilot 100, that the pilot 100's attitude 44 is in an attitude that recognizes the warning target 51 included in the warning information 50.

[0023] The control unit 13 includes a processor (arithmetic processing unit) such as a CPU (Central Processing Unit) or microcontroller, as well as ROM (Read Only Memory) and RAM (Random Access Memory). The control unit 13 is configured in software as a functional block realized by the processor executing various programs.

[0024] (Configuration of the alarm information generation device) The alarm information generation device 2 comprises a detection unit 20, an external image capture unit 21, an image processing unit 22, and an alarm information generation device side communication unit 23. The alarm information generation device 2 is installed on the aircraft 90.

[0025] The detection unit 20 is configured to acquire alarm information 50. The detection unit 20 includes, for example, a radar (RADAR: Radio Detecting and Ranging) or a LiDAR (LiDAR: Light Detection and Ranging).

[0026] The external image acquisition unit 21 is configured to acquire external images (not shown) of the aircraft 90, including images of the area in front of, to the side of, and behind the aircraft 90. The external image acquisition unit 21 includes, for example, multiple cameras.

[0027] The image processing unit 22 is configured to extract alarm targets 51 based on external images acquired by the external image acquisition unit 21, and to generate alarm information 50 based on the extracted alarm targets 51. The image processing unit 22 includes, for example, a processor (arithmetic processing unit) such as a GPU (Graphics Processing Unit).

[0028] The alarm information generation device side communication unit 23 is configured to communicate with the alarm control device side communication unit 11. The alarm information generation device side communication unit 23 transmits alarm information 50 to the alarm control device side communication unit 11.

[0029] (Alarm information) As shown in Figure 4, the alarm information 50 includes the alarm target 51, the severity level 52 of the alarm target 51, and visual information 53 indicating whether or not the alarm target 51 is visible to the pilot 100.

[0030] The warning targets 51 include, for example, those requiring rescue, such as distressed or injured persons (see Figure 1), as well as structures 81 (see Figure 1), natural formations 82 (see Figure 1), and disasters 83 (see Figure 1) that could obstruct flight.

[0031] When the alert target 51 is a rescue target 80, the importance level 52 of the alert target 51 refers, for example, to the confidence level (validity) of the rescue target 80 extracted based on external images acquired by the external image acquisition unit 21. The image processing unit 22 determines that the importance level 52 of the alert target 51 is high importance if the confidence level (validity) of the rescue target 80 extracted based on external images acquired by the external image acquisition unit 21 is above a predetermined standard. The image processing unit 22 also determines that the importance level 52 of the alert target 51 is low importance if the confidence level (validity) of the rescue target 80 extracted based on external images acquired by the external image acquisition unit 21 is below a predetermined standard.

[0032] Furthermore, when the alert target 51 is an obstacle to flight, the severity level 52 of the alert target 51 means, for example, the possibility that serious consequences may be caused by structures 81, natural formations 82, and hazards 83 extracted based on external images acquired by the external image acquisition unit 21. A high probability of serious consequences means, for example, that the distance from the aircraft 90 to the obstacle to flight is short, that the size (height) of the structures 81 and natural formations 82 as obstacles to flight is large (tall), and that a severe hazard 83 as an obstacle to flight has occurred. A low probability of serious consequences means, for example, that the distance from the aircraft 90 to the obstacle to flight is long, that the size (height) of the structures 81 and natural formations 82 as obstacles to flight is small (low), and that a minor hazard 83 as an obstacle to flight has occurred.

[0033] The image processing unit 22 determines that the importance level 52 of the warning target 51 is high importance if there is a high probability that the flight disruption extracted based on the external image acquired by the external image acquisition unit 21 will cause serious consequences. The image processing unit 22 also determines that the importance level 52 of the warning target 51 is low importance if there is a low probability that the flight disruption extracted based on the external image acquired by the external image acquisition unit 21 will cause serious consequences. The criteria and methods for determining whether the importance level 52 of the warning target 51 is high importance or low importance are not limited to the criteria and methods described above, and various criteria and methods may be used.

[0034] Furthermore, the image processing unit 22 determines whether the warning target 51 is visible to the pilot 100 based on the size of the warning target 51 extracted from the external image acquired by the external image acquisition unit 21 and the distance to the warning target 51, and acquires the visual information 53 which is the result of the determination. The method for determining whether the warning target 51 is visible to the pilot 100 is not particularly limited.

[0035] (Configuration of the alarm system) The alarm device 3 comprises an alarm unit 30 and an alarm device-side communication unit 31. For example, the alarm device 3 is attached to the pilot 100.

[0036] The alarm unit 30 is configured to output an alarm to the pilot 100. For example, the alarm unit 30 is a headset worn on the pilot 100's head. Specifically, the alarm unit 30 is the headphones included in the headset. The alarm unit 30 outputs an audible or voice alarm to the pilot 100 when the alarm device side communication unit 31 receives a control signal (electrical signal) from the alarm control device side communication unit 11 that causes the alarm to be output.

[0037] The alarm device side communication unit 31 is configured to communicate with the alarm control device side communication unit 11. The alarm device side communication unit 31 receives a control signal (electrical signal) from the alarm control device side communication unit 11 that causes the alarm unit 30 of the alarm device 3 to output an alarm.

[0038] (Each functional block of the control unit) The functional blocks included in the control unit 13 will now be described. The control unit 13, which consists of a CPU or microcontroller as hardware, includes the following as software (program) functional blocks: an alarm information acquisition unit 14, a skeleton estimation unit 15, a posture estimation unit 16, and an output control unit 17. The control unit 13 functions as the alarm information acquisition unit 14, the skeleton estimation unit 15, the posture estimation unit 16, and the output control unit 17 by executing the program stored in the memory unit 12. The alarm information acquisition unit 14, the skeleton estimation unit 15, the posture estimation unit 16, and the output control unit 17 may be individually configured by hardware with dedicated processors (processing circuits).

[0039] (Alarm information acquisition unit) The alarm information acquisition unit 14 is configured to acquire alarm information 50 from the alarm information generation device 2 via the alarm control device side communication unit 11. The control unit 13 outputs an alarm only when the alarm information 50 is acquired by the alarm information acquisition unit 14.

[0040] (Skeletal estimation section) Skeleton estimation by the skeleton estimation unit 15 (see Figure 2) will be explained with reference to Figures 3, 5, and 6. The skeleton estimation unit 15 is configured to estimate the state of the pilot's skeleton 42 (see Figure 5) in the pilot image 40 (see Figure 3) acquired by the in-flight image acquisition unit 10 (see Figure 2).

[0041] When the alarm information acquisition unit 14 acquires alarm information 50 from the alarm information generation device 2, the skeletal estimation unit 15 acquires real-time pilot images 40 (see Figure 3) acquired by the in-flight image capture unit 10.

[0042] As shown in Figure 5, the skeleton estimation unit 15 performs a skeleton estimation process on the acquired pilot image 40 (see Figure 3) to estimate the state of the pilot's skeleton 42, which is skeletal data including the positions of the pilot's feature points 43 in the pilot image 40. The feature points 43 of the pilot 100 in the pilot image 40 include at least the joints of the pilot's arms, the knee joints of the legs, and the eyes and nose or mouth. As shown in Figure 6, in the skeleton estimation process, the skeleton estimation unit 15 uses the trained model 4 as an input image to obtain a skeleton state image 41 as an output, which includes the state of the pilot's skeleton 42 in the pilot image 40. That is, the skeleton estimation unit 15 inputs the pilot image 40 to the trained model 4 and estimates the state of the pilot's skeleton 42 using the skeleton state image 41 output by the trained model 4, which includes the state of the pilot's skeleton 42 in the pilot image 40.

[0043] (Pre-trained models and how to create them) The trained model 4 is generated by machine learning using a pre-acquired pilot image 4a for training input and a pre-acquired skeletal state image 4b ​​for training output, which includes skeletal data including the positions of feature points 43 of the pilot 100 in the pilot image 4a for training input. The trained model 4 is generated by a learning device 5 separate from the warning control device 1. The learning device 5 is a computer for machine learning, including, for example, a CPU, GPU, ROM, and RAM.

[0044] The learning device 5 uses pilot images 4a for teacher input as input and skeletal state images 4b for teacher output as output, and performs machine learning to generate a trained model 4. In other words, the learning device 5 uses multiple pilot images 4a for teacher input and multiple skeletal state images 4b for teacher output as training data (training set) to train the trained model 4 using machine learning. The machine learning method used is, for example, deep learning using a fully convolutional network (FCN). The created trained model 4 is provided to the alarm control device 1 via the network or recorded on a recording medium such as flash memory.

[0045] The skeleton estimation unit 15 uses the trained model 4 as an input image to take the acquired pilot image 40 as an input image and outputs a skeleton state image 41 which includes the state of the pilot's skeleton 42 in the pilot image 40, thereby estimating the state of the pilot's skeleton 42 of the pilot 100.

[0046] (Posture estimation section) Referring to Figure 7, the attitude estimation by the attitude estimation unit 16 (see Figure 2) will be explained. The attitude estimation unit 16 is configured to acquire the attitude 44 of the pilot 100 based on the state of the pilot's skeleton 42 (see Figure 5) in the skeleton state image 41 estimated by the skeleton estimation unit 15. Specifically, the attitude estimation unit 16 acquires the attitude 44 of the pilot 100, including the orientation of the pilot's face 45 and the state of the pilot's arms 46, based on the state of the pilot's skeleton 42 in the skeleton state image 41 estimated by the skeleton estimation unit 15.

[0047] Here, as a method for directly acquiring a person's gaze, there is a gaze detection method that detects the direction of the pilot's gaze using a corneal reflection method with a near-infrared light source and eye tracking. However, it is difficult to apply the above gaze detection method to the pilot 100 of the aircraft 90. Therefore, the attitude estimation unit 16 acquires the orientation 45 of the pilot's face in the skeletal state image 41 based on the skeletal state image 41 estimated by the skeletal state unit 15 (see Figure 2) and the skeletal state image 41 (see Figure 5).

[0048] In other words, the attitude estimation unit 16 uses the state of the pilot's skeleton 42, which is skeletal data including the positions of feature points 43 including the pilot's eyes and nose or mouth, to obtain the orientation 45 of the pilot's face in the skeletal state image 41. The method by which the attitude estimation unit 16 obtains the orientation 45 of the pilot's face in the skeletal state image 41 is not particularly limited and can be obtained by known methods using the state of the pilot's skeleton 42, which is skeletal data including the positions of feature points 43 including the pilot's eyes and mouth.

[0049] Furthermore, the attitude estimation unit 16 uses the state of the pilot's skeleton 42, which is skeletal data including the positions of feature points 43, including each joint of the pilot's arm, to acquire the state of the pilot's arm 46 in the skeletal state image 41. The method by which the attitude estimation unit 16 acquires the state of the pilot's arm 46 in the skeletal state image 41 is not particularly limited and can be acquired by known methods using the state of the pilot's skeleton 42, which is skeletal data including the positions of feature points 43, including each joint of the pilot's arm.

[0050] (Output control unit) The control of the alarm output by the output control unit 17 (see Figure 2) will be explained with reference to Figures 2 and 7. The output control unit 17 is configured to control the output of an alarm to the pilot 100 of the aircraft 90.

[0051] Furthermore, in order to prevent the pilot 100 of the aircraft 90 from becoming accustomed to the warnings, the output control unit 17 is configured, as shown in Figure 7, to not output a warning when it estimates that the attitude 44 of the pilot 100 in the skeletal state image 41 acquired by the attitude estimation unit 16 (see Figure 2) is an attitude in which the pilot recognizes the warning target 51 included in the warning information 50. In other words, the output control unit 17 does not transmit a control signal to the warning device side communication unit 31 via the warning device side communication unit 11 that would cause the warning unit 30 of the warning device 3 to output a warning.

[0052] Specifically, the output control unit 17 is configured to perform control to refrain from outputting an alarm when it estimates that the pilot 100 is in a posture where he recognizes the alarm target 51, based on the state 46 of the pilot's arm in the skeletal state image 41 acquired by the attitude estimation unit 16, which shows the pilot with his hand on the control device 91 in response to the alarm target 51. In other words, the output control unit 17 performs control to refrain from outputting an alarm when it estimates that the pilot 100 is performing the necessary maneuvers or operations to respond to the alarm target 51, based on the state 46 of the pilot's arm included in the pilot's posture 44 in the skeletal state image 41 acquired by the attitude estimation unit 16.

[0053] The in-flight image capture unit 10 (see Figure 2) is permanently installed in the cockpit of the aircraft 90. Therefore, the arrangement of the control stick 91a, instruments, various switches, and various buttons in the pilot image 40 and the skeletal state image 41 is always in the same position and orientation. Furthermore, when the pilot 100 operates the control devices 91 located in the cockpit, the state 46 of the pilot's arms is in a specific state corresponding to the operation. Therefore, based on the state 46 of the pilot's arms included in the attitude 44 (see Figure 7) of the pilot 100 in the skeletal state image 41 acquired by the attitude estimation unit 16, and the arrangement of the control devices 91, the pilot 100's operating state of the control devices 91 can be estimated. Therefore, the output control unit 17 performs control to refrain from outputting an alarm when it estimates that the pilot 100's arm position 46 in the skeletal state image 41 indicates that the pilot is recognizing the alarm target 51, as evidenced by the pilot having their hands on the control stick 91a, instruments, various switches and buttons corresponding to the alarm target 51.

[0054] Furthermore, the output control unit 17 is configured to output an alarm when it estimates that the pilot's posture does not recognize the alarm target 51, based on the state 46 of the pilot's arm in the skeletal state image 41 acquired by the posture estimation unit 16, which shows that the pilot is not touching the control device 91 in order to correspond to the alarm target 51. In other words, the output control unit 17 transmits a control signal to the alarm device side communication unit 31 via the alarm control device side communication unit 11 that causes the alarm unit 30 of the alarm device 3 to output an alarm.

[0055] Furthermore, the output control unit 17 is configured to perform control to not output an alarm when it estimates that the pilot's attitude recognizes the alarm target 51 because the orientation 45 of the pilot's face in the skeletal state image 41 acquired by the attitude estimation unit 16 is the orientation 45 of the pilot's face corresponding to the alarm target 51. In other words, the output control unit 17 performs control to not output an alarm when it estimates that the direction of the pilot's gaze corresponds to the direction of the alarm target 51, based on the orientation 45 of the pilot's face included in the attitude 44 of the pilot's face in the skeletal state image 41 acquired by the attitude estimation unit 16.

[0056] Generally, it can be estimated that humans are looking straight ahead when their face is facing 45. Therefore, the forward direction of the pilot's face in the skeletal state image 41 acquired by the attitude estimation unit 16 can be considered the direction of the pilot's gaze. Accordingly, the output control unit 17 controls the system not to output an alarm when it estimates that the pilot's attitude is one in which the pilot's face is facing 45 in the skeletal state image 41, because the face is facing in the direction corresponding to the alarm target 51.

[0057] Furthermore, the output control unit 17 is configured to output an alarm if it estimates that the pilot's posture does not recognize the alarm target 51 because the orientation 45 of the pilot's face in the skeletal state image 41 acquired by the posture estimation unit 16 is not the orientation 45 corresponding to the alarm target 51. In other words, the output control unit 17 transmits a control signal to the alarm device side communication unit 31 via the alarm device side communication unit 11 that causes the alarm unit 30 of the alarm device 3 to output an alarm.

[0058] Furthermore, the output control unit 17 will not output an alarm if the importance level 52 of the alarm target 51 included in the alarm information 50 acquired by the alarm information acquisition unit 14 from the alarm information generation device 2 is low importance, and the attitude 44 of the pilot 100 in the skeletal state image 41 acquired by the attitude estimation unit 16 is estimated to be an attitude in which the pilot recognizes the alarm target 51. In other words, if the importance level 52 of the alarm target 51 is low importance, the output control unit 17 will not output an alarm if the attitude 46 of the pilot 100 in the skeletal state image 41 acquired by the attitude estimation unit 16 is estimated to be an attitude in which the pilot recognizes the alarm target 51 because the orientation of the pilot's face 45 corresponds to the orientation of the pilot's face 45 corresponding to the alarm target 51, and the state of the pilot's arms 46 is such that the pilot has their hands on the control equipment 91 corresponding to the alarm target 51.

[0059] Furthermore, the output control unit 17 will not output an alarm if the importance level 52 of the alarm target 51 included in the alarm information 50 acquired by the alarm information acquisition unit 14 from the alarm information generation device 2 is high importance, and if the visual information 53 included in the alarm information 50 indicates that the alarm target 51 is recognizable by visual inspection, and the attitude 44 of the pilot 100 in the skeletal state image 41 acquired by the attitude estimation unit 16 is estimated to be an attitude in which the alarm target 51 included in the alarm information 50 is recognized. In other words, the output control unit 17 will not send a control signal to the alarm device side communication unit 31 via the alarm control device side communication unit 11 to cause the alarm unit 30 of the alarm device 3 to output an alarm.

[0060] In response to this, the output control unit 17 is configured to output an alarm when the importance level 52 of the alarm target 51 included in the alarm information 50 acquired by the alarm information acquisition unit 14 from the alarm information generation device 2 is high importance, and the alarm target 51 is not recognizable by visual inspection in the visual inspection information 53 included in the alarm information 50. That is, the output control unit 17 transmits a control signal to the alarm device side communication unit 31 via the alarm device side communication unit 11 to cause the alarm unit 30 of the alarm device 3 to output an alarm.

[0061] Here, with reference to Figure 7, an example of control by the output control unit 17 (see Figure 2) that does not output an alarm will be explained in more detail. For example, the alarm target 51 is a mountain. Due to the long distance from the aircraft 90 to the mountain, the importance 52 of the alarm target 51 is low. Assume that the pilot 100 needs to fly while looking at the altimeter 91b in order to cross the mountaintop whose altitude is known. The output control unit 17 estimates that the pilot 100 is in an attitude of recognizing the mountain because the orientation 45 of the pilot 100's face in the skeletal state image 41 is facing the altimeter 91b in the cockpit, and also estimates that the pilot 100 is in an attitude of recognizing the mountain because the state of the pilot 100's arms 46 is in a state where they have their hands on the control stick 91a in order to cross the mountaintop, and therefore performs control that does not output an alarm.

[0062] Furthermore, with reference to Figure 8, another example of control by the output control unit 17 (see Figure 2) that does not output an alarm will be explained in more detail. For example, the alarm target 51 is the rescue target 80 (see Figure 1). The importance level 52 of the alarm target 51 is high because the confidence level (validity) of the rescue target 80 is above a predetermined standard. In the visual information 53, the rescue target 80 is recognizable by visual inspection. The output control unit 17 estimates that the pilot 100 is in a posture that recognizes the rescue target 80 because the orientation 45 of the pilot's face in the skeletal state image 41 is facing the direction where the rescue target 80 is located, and also estimates that the pilot 100 is in a posture that recognizes the rescue target 80 because the state 46 of the pilot's arms is in a state where they are reaching for the control stick 91a to move towards the head of the rescue target 80, and therefore performs control that does not output an alarm.

[0063] (Alarm output control processing by the control unit) Referring to Figure 9, the alarm output control process by the control unit 13 according to this embodiment will be explained. Note that the order of the processing steps can be reversed or executed simultaneously, as long as they do not contradict each other.

[0064] In step S1, if the alarm information acquisition unit 14 acquires alarm information 50 from the alarm information generation device 2 via the alarm control device side communication unit 11 (Yes in step S1), the process proceeds to step S2. If the alarm information 50 has not been acquired from the alarm information generation device 2 via the alarm control device side communication unit 11 (No in step S1), the process proceeds to step S1.

[0065] In step S2, the skeletal estimation unit 15 acquires the pilot image 40 obtained by the in-flight image acquisition unit 10. The process then proceeds to step S3.

[0066] In step S3, the skeleton estimation unit 15 uses the trained model 4 as input to the acquired pilot image 40 and outputs a skeleton state image 41 containing the state of the pilot's skeleton 42, thereby estimating the state of the pilot's skeleton 42. The process then proceeds to step S4.

[0067] In step S4, the attitude estimation unit 16 obtains the attitude 44 of the pilot 100, including the orientation of the pilot 100's face 45 and the state of the pilot 100's arms 46, based on the state of the pilot 100's skeleton 42 in the skeleton state image 41 estimated by the skeleton estimation unit 15. The process then proceeds to step S5.

[0068] In step S5, the output control unit 17 proceeds to step S6 if the importance level 52 of the alarm target 51 included in the alarm information 50 acquired from the alarm information generation device 2 is low importance (Yes in step S5), and proceeds to step S9 if the importance level 52 of the alarm target 51 included in the alarm information 50 acquired from the alarm information generation device 2 is high importance (No in step S5).

[0069] In step S6, if the output control unit 17 estimates that the pilot's attitude is one in which the pilot is recognizing the alarm target 51 because the orientation of the pilot's face 45 in the skeletal state image 41 acquired by the attitude estimation unit 16 is the orientation of the pilot's face 45 corresponding to the alarm target 51 (Yes in step S6), the process proceeds to step S7. If the output control unit 17 estimates that the pilot is not in an attitude that recognizes the alarm target 51 because the orientation of the pilot's face 45 is not the orientation of the pilot's face 45 corresponding to the alarm target 51 (No in step S6), the process proceeds to step S10.

[0070] In step S7, if the output control unit 17 estimates that the pilot 100 is in a posture that recognizes the alarm target 51 because the state of the pilot's arm 46 in the skeletal state image 41 acquired by the attitude estimation unit 16 shows that the pilot has their hand on the control device 91 in response to the alarm target 51 (Yes in step S7), the process proceeds to step S8. If the output control unit 17 estimates that the pilot 100 is not in a posture that recognizes the alarm target 51 because the state of the pilot's arm 46 shows that the pilot has their hand on the control device 91 in response to the alarm target 51 (No in step S7), the process proceeds to step S10.

[0071] In step S8, the output control unit 17 does not send a control signal to the alarm device side communication unit 31 via the alarm control device side communication unit 11 to cause the alarm unit 30 of the alarm device 3 to output an alarm. After that, the process ends.

[0072] In step S9, the output control unit 17 determines that if the alarm target 51 is recognizable by visual inspection in the visual inspection information 53 included in the alarm information 50 (Yes in step S9), the process proceeds to step S6. If the alarm target 51 is not recognizable by visual inspection in the visual inspection information 53 included in the alarm information 50 (No in step S9), the process proceeds to step S10.

[0073] In step S10, the output control unit 17 transmits a control signal to the alarm device side communication unit 31 via the alarm control device side communication unit 11, which causes the alarm unit 30 of the alarm device 3 to output an alarm. After that, the process ends.

[0074] [Differentiation] It should be noted that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims rather than by the description of the embodiments above, and further includes all modifications (exceptions) within the meaning and scope equivalent to the claims. For example, the alarm control device may be configured to include either an alarm information generating device or an alarm device, or the alarm control device may be configured to include either an alarm information generating device or an alarm device. Furthermore, for example, the warning control device may be configured to estimate the skeletal state of the flight engineer based on an image of the flight engineer including the flight engineer, and to suppress the warning output based on the estimated skeletal state of the flight engineer in the control of outputting a warning to the flight engineer. Furthermore, the control unit may be configured to suppress alarm output based on the estimated skeletal state of the pilot. That is, the control unit may be configured to suppress alarm output when it estimates that the pilot's attitude is such that it recognizes a warning target included in the alarm information. For example, the control unit may be configured to suppress alarm output by reducing the frequency of alarm output or by reducing the volume of audible or voice alarm output when it estimates that the pilot's attitude is such that it recognizes a warning target included in the alarm information. Furthermore, for example, the control unit may be configured to perform control that does not output an alarm based on the pilot's skeletal state, without acquiring alarm information including the alarm target and the importance of the alarm target. Furthermore, for example, the in-flight image capture unit may be included in the alarm information generation device, or it may be located outside the alarm control device in the cockpit. Furthermore, for example, the warning system may be configured to alert the pilot by illuminating a warning light located in the cockpit, in addition to or in addition to a headset worn on the pilot's head. Furthermore, for example, the pilot's feature points in the pilot image may further include patterns on the helmet, and the control unit may be configured to use a trained model in the skeletal estimation process to take the acquired pilot image as an input image and output a skeletal state image that includes the state of the pilot's skeleton. Furthermore, the method for estimating the state of the pilot's skeleton is not limited to using a pre-trained model, and may be configured to estimate the state of the pilot's skeleton, which is the pilot's skeleton data, by performing known skeleton estimation processing. For example, an identification label may be provided on a predetermined part of the flight suit worn by the pilot, and the state of the pilot's skeleton may be estimated by reading the provided identification label with a reader device. The identification label is not particularly limited, and for example, something like an AR marker could be considered. In this case, an identification label may also be provided on the pilot's helmet, and the state of the pilot's skeleton may be estimated by reading the identification label on the helmet with a reader device. Furthermore, for example, the control unit may be configured not to acquire the pilot's facial orientation based on the estimated state of the pilot's skeleton, and not to assume that the pilot's facial orientation is such that it recognizes the alarm target. Furthermore, for example, the control unit may be configured to output an alarm even if the alarm target is visually recognizable, if the alarm target is of high importance.

[0075] [Aspect] Those skilled in the art will understand that the exemplary embodiments described above are specific examples of the following embodiments.

[0076] (Item 1) Steps include obtaining crew images, including those of aircraft crew members, The steps include: estimating the skeletal state of the crew member based on the acquired crew member image; An aircraft alarm output method, comprising the step of performing control to suppress the alarm output based on the estimated skeletal state of the aircraft, in which an alarm is output to the crew of the aircraft. By estimating the skeletal state of the crew using acquired crew images, it is possible to determine whether the crew is currently in a state where they should be performing an action, based on the estimated skeletal state. Therefore, if it is determined that the crew is currently in a state where they should be performing an action, based on the estimated skeletal state, the alarm output can be suppressed. In other words, if the aircraft crew is aware of the cause of the alarm, the alarm output for that cause may be suppressed. As a result, it is possible to prevent the aircraft crew from becoming accustomed to the alarm. Furthermore, since the skeletal state of the crew is estimated using acquired crew images, the crew does not need to wear additional equipment such as inertial sensors to detect the position of each part of their body. Therefore, the burden on the crew does not increase compared to when additional equipment for detecting the position of each part of the body is worn. Thus, it is possible to prevent the crew from becoming accustomed to the alarm without increasing the burden on the aircraft crew. (Item 2) The step of acquiring alarm information is further enhanced. The step of performing control to suppress the alarm output is performed when the alarm information is acquired in the step of acquiring the alarm information, and is the alarm output method for an aircraft as described in item 1. In this case, since the control to suppress the alarm output is performed when alarm information is acquired, the processing load on the control unit in the control to suppress the alarm output can be reduced compared to when the control to suppress the alarm output is performed every control cycle. (Item 3) The aircraft alarm output method according to item 2, wherein the step of performing control to suppress the alarm output is to perform control to suppress the alarm output when it is estimated that the posture of the crew member is a posture in which the crew member recognizes the alarm target included in the alarm information, based on the estimated skeletal state of the crew member. In this case, based on the estimated skeletal state of the crew member, the system estimates that the crew member's posture is one in which they recognize the alarm target included in the alarm information. Therefore, it is possible to accurately determine whether the crew member's posture is in a state where they should be performing an action. As a result, based on the estimation that the crew member's posture is one in which they recognize the alarm target, appropriate control can be performed to suppress the alarm output. (Item 4) The process further includes the step of obtaining the posture of the crew member, including the state of the crew member's arms, based on the estimated skeletal state of the crew member. The aircraft alarm output method according to item 3, wherein the step of performing control to suppress the alarm output is performed when it is estimated that the crew member is in a posture that recognizes the alarm target because the state of the crew member's arm is such that they have their hand on an operating device in response to the alarm target. In this case, the acquired state of the crew member's arm indicates that they are in a position to recognize the alarm target, as they are touching the control device in order to respond to the alarm target. Therefore, it is possible to more accurately determine whether the crew member's posture is in a state where they should be performing the action they are currently in. As a result, based on the estimation that the crew member's arm state indicates they are in a position to recognize the alarm target, it is possible to perform control that suppresses the alarm output more appropriately. (Item 5) The process further includes the step of obtaining the posture of the crew member, including the orientation of the crew member's face, based on the estimated skeletal state of the crew member. The aircraft alarm output method according to item 4, further comprising the step of performing control to suppress the alarm output, wherein the control to suppress the alarm output is performed when it is estimated that the crew member is in a posture that recognizes the alarm target because the orientation of the crew member's face corresponds to the orientation of the face corresponding to the alarm target. Here, by obtaining the orientation of the crew member's face based on the state of the crew member's skeleton, it is possible to estimate whether the pilot's gaze direction is the direction they should be looking at at the moment, based on the obtained orientation of the crew member's face. Therefore, since it is estimated that the crew member's posture is one in which they recognize the warning target because the obtained orientation of the crew member's face is one in which they recognize the warning target, it is possible to determine with even greater accuracy whether the crew member's posture is in the state in which they should be performing the action. As a result, based on the estimation that the crew member's face orientation is one in which they recognize the warning target, it is possible to perform control that suppresses the warning output more appropriately. (Item 6) The step of acquiring the alarm information includes acquiring the alarm information, which includes the alarm target and the importance of the alarm target. The aircraft alarm output method according to any one of items 3 to 5, wherein the step of performing control to suppress the alarm output is performed when the importance of the alarm target is low and it is estimated that the crew's posture is in a posture in which they recognize the alarm target. Here, even when the importance of the warning target is low and the crew is aware of the warning target, the warning is still issued, which can lead to a kind of desensitization of the warning to the crew. Therefore, by implementing control to suppress the warning output when the importance of the warning target is low and the crew's posture is estimated to indicate awareness of the warning target, it is possible to effectively suppress desensitization of the warning to the aircraft crew. (Item 7) The step of acquiring the aforementioned alarm information further includes visual information regarding whether the alarm target is recognizable by the crew member's visual observation, The aircraft alarm output method according to item 6, wherein the step of performing control to suppress the alarm output is performed when the importance of the alarm target is of high importance, the alarm target is recognizable by visual inspection in the visual information, and the posture of the crew member is estimated to be in a posture in which the alarm target is recognized. Here, even if the importance of the alert target is high, if the alert target is recognizable by the crew visually and the crew is aware of the alert target, there is little need to issue an alert to the crew to draw their attention to the alert target. Therefore, even if the importance of the alert target is high, if the alert target is recognizable by visual information and the crew's posture is estimated to indicate that they are aware of the alert target, control is implemented to suppress the output of the alert. This makes it possible to more effectively prevent aircraft crews from becoming accustomed to the alerts. (Item 8) The aircraft alarm output method according to item 7, wherein the step of performing control to suppress the alarm output is to perform control to output the alarm when the importance of the alarm target is high importance and the alarm target is not recognizable by visual inspection in the visual information. In this case, if the importance of the alert target is high and the alert target is not recognizable by the crew's visual observation, there is a high need to output an alert to the crew to draw their attention to the alert target. Therefore, if the importance of the alert target is high and the alert target is not recognizable by visual observation, control is implemented to output an alert. This allows for appropriate control to output alerts for alert targets where there is a high need to output an alert. (Item 9) The aircraft alarm output method according to any one of items 1 to 8, wherein the step of performing control to suppress the alarm output is to perform control to not output the alarm based on the estimated skeletal state of the crew member. In this case, as a control measure to suppress alarm output, an alarm is not issued based on the skeletal condition of the crew member. Therefore, it is possible to effectively prevent aircraft crew members from becoming accustomed to the alarm. (Item 10) The step of estimating the skeletal state of the crew member involves using a trained model to obtain the skeletal state of the crew member as an output result, with the acquired image of the crew member as the input image. The aircraft alarm output method according to any one of items 1 to 9, wherein the step of performing control to suppress the alarm output is to perform control to suppress the alarm output based on the skeletal state of the crew member estimated using the trained model. In this case, the crew's skeletal state can be obtained using a pre-trained model, and control can be performed to suppress the alarm output based on the crew's skeletal state estimated using the pre-trained model. Therefore, the crew's skeletal state can be easily obtained using a pre-trained model, and control can be easily performed to suppress the alarm output based on the crew's skeletal state. (Item 11) The photography department is responsible for taking images of the crew, including the aircraft crew, A skeleton estimation unit that estimates the state of the crew member's skeleton based on the captured image of the crew member, An aircraft alarm control device comprising: an output control unit that performs control to suppress the alarm output based on the estimated skeletal state of the aircraft's crew member, in a control that outputs an alarm to the crew member of the aircraft. By estimating the skeletal state of the crew using acquired crew images, it is possible to determine whether the crew is currently in a state where they should be performing an action, based on the estimated skeletal state. Therefore, if it is determined that the crew is currently in a state where they should be performing an action, based on the estimated skeletal state, the alarm output can be suppressed. In other words, if the aircraft crew is aware of the cause of the alarm, the alarm output for that cause may be suppressed. As a result, it is possible to prevent the aircraft crew from becoming accustomed to the alarm. Furthermore, since the skeletal state of the crew is estimated using acquired crew images, the crew does not need to wear additional equipment such as inertial sensors to detect the position of each part of their body. Therefore, the burden on the crew does not increase compared to when additional equipment for detecting the position of each part of the body is worn. Thus, it is possible to prevent the crew from becoming accustomed to the alarm without increasing the burden on the aircraft crew. [Explanation of Symbols]

[0077] 1. Alarm control device 4. Pre-trained models 10. In-flight photography department (photography department) 15 Skeleton Estimation Section 17 Output Control Unit 40. Pilot images (crew images) 42 Skeletal condition 44 Posture 45. Direction of the face 46. ​​Condition of the arm 50 Alarm information 51. Items subject to warning 52. Importance of the alert target 53 Visual Information 90 aircraft 91 Operating equipment 100 Pilots (Crew)

Claims

1. Steps include obtaining crew images, including those of aircraft crew members, The steps include: estimating the skeletal state of the crew member based on the acquired crew member image; An aircraft alarm output method, comprising the step of performing control to suppress the alarm output based on the estimated skeletal state of the aircraft, in which an alarm is output to the crew of the aircraft.

2. The step of acquiring alarm information is further enhanced. The method for outputting an alarm for an aircraft according to claim 1, wherein the step of performing control to suppress the alarm output is performed when the alarm information is acquired in the step of acquiring the alarm information.

3. The aircraft alarm output method according to claim 2, wherein the step of performing control to suppress the alarm output is to perform control to suppress the alarm output when it is estimated that the posture of the crew member is a posture in which the crew member recognizes the alarm target included in the alarm information, based on the estimated state of the crew member's skeleton.

4. The process further includes the step of obtaining the posture of the crew member, including the state of the crew member's arms, based on the estimated skeletal state of the crew member. The step of performing control to suppress the alarm output is to perform control to suppress the alarm output when it is estimated that the crew member is in a posture that recognizes the alarm target because the state of the crew member's arm is such that they have their hand on an operating device to correspond to the alarm target, as described in claim 3.

5. The process further includes the step of obtaining the posture of the crew member, including the orientation of the crew member's face, based on the estimated skeletal state of the crew member. The aircraft alarm output method according to claim 4, further comprising the step of performing control to suppress the alarm output, wherein the control to suppress the alarm output is performed when it is estimated that the crew member is in a posture that recognizes the alarm target because the orientation of the crew member's face corresponds to the orientation of the face corresponding to the alarm target.

6. The step of acquiring the alarm information includes acquiring the alarm information, which includes the alarm target and the importance of the alarm target. The aircraft alarm output method according to claim 3, wherein the step of performing control to suppress the alarm output is performed when the importance of the alarm target is low and it is estimated that the crew's posture is in a posture in which the alarm target is recognized.

7. The step of acquiring the aforementioned alarm information further includes visual information regarding whether the alarm target is recognizable by the crew member's visual observation, The aircraft alarm output method according to claim 6, wherein the step of performing control to suppress the alarm output is performed when the importance of the alarm target is of high importance, the alarm target is recognizable by visual inspection in the visual information, and the posture of the crew member is estimated to be in a posture in which the alarm target is recognized.

8. The step of performing control to suppress the alarm output is to perform control to output the alarm when the importance of the alarm target is high importance and the alarm target is not recognizable by visual inspection in the visual information, as described in claim 7, an aircraft alarm output method.

9. The aircraft alarm output method according to claim 1, wherein the step of performing control to suppress the alarm output is to perform control not to output the alarm based on the estimated skeletal state of the crew member.

10. The step of estimating the skeletal state of the crew member involves using a trained model to obtain the skeletal state of the crew member as an output result, with the acquired image of the crew member as the input image. The method for an aircraft to output an alarm, according to claim 1, wherein the step of performing control to suppress the alarm output is performed based on the state of the crew member's skeleton estimated using the trained model.

11. The photography department is responsible for taking images of the crew, including the aircraft crew, A skeleton estimation unit that estimates the state of the crew member's skeleton based on the captured image of the crew member, An aircraft alarm control device comprising: an output control unit that performs control to suppress the alarm output based on the estimated skeletal state of the aircraft's crew member, in a control that outputs an alarm to the crew member of the aircraft.

Images