Pilot behavior detection method and device and aircraft

Abstract

The invention discloses a pilot behavior detection method and device and an aircraft, and belongs to the technical field of driving detection. The method comprises: acquiring image data; constructing a three-dimensional scene model based on a three-dimensional scene reconstruction algorithm according to the image data; constructing a human body posture model based on a human body posture estimation algorithm according to the image data; acquiring behavior data according to interaction of the human body posture model under the three-dimensional scene model; and when the behavior data exceed the preset behavior standard range, performing early warning on the behavior data. By combining the attitude of the pilot with the three-dimensional scene where the pilot is located, the behavior of the pilot is comprehensively and intuitively monitored and analyzed according to the interaction between the attitude of the pilot and the three-dimensional scene, so that the accuracy of pilot behavior detection is improved.

Description

Technical Field

[0001] This application relates to the field of pilot detection technology, specifically to a method, device, and aircraft for detecting pilot behavior. Background Technology

[0002] In aircraft aviation missions, the operational behavior of aircraft pilots involves complex interactions between the pilot, the aircraft's flight equipment, and the environment. For example, the pilot's reading of instruments, operation of the control stick, and coordination with the co-pilot are all embedded in specific spatial relationships and semantic environments.

[0003] In related technologies, pilot behavior is detected solely by the pilot's actions or state to determine whether it meets standards. However, since piloting operations are deeply intertwined with the cockpit environment, analyzing isolated behavioral or physiological data without considering the cockpit environment makes it difficult to accurately reconstruct the true context and safety hazards of the behavior, thus affecting the accuracy of pilot behavior detection. Summary of the Invention

[0004] This application provides a method, apparatus, and aircraft for detecting pilot behavior, aiming to improve the accuracy of pilot behavior detection.

[0005] In a first aspect, embodiments of this application provide a pilot behavior detection method, comprising the following steps: Acquire image data; A 3D scene model is constructed based on the image data and a 3D scene reconstruction algorithm. A human pose model is constructed based on image data and a human pose estimation algorithm. Behavioral data is obtained based on the interaction of the human posture model with the 3D scene model. When behavioral data exceeds the preset behavioral standard range, an early warning is issued for the behavioral data.

[0006] In some embodiments, acquiring image data includes: Image sequences are acquired using visual sensors in multiple different orientations; The image sequence is time-aligned to obtain image data.

[0007] In some embodiments, a 3D scene model is constructed based on image data using a 3D scene reconstruction algorithm, including: Construct sparse 3D point cloud and dense environment models based on image data; By registering and fusing sparse 3D point clouds with dense environment models, a 3D scene model is obtained.

[0008] In some embodiments, constructing a sparse 3D point cloud from image data includes: Extract feature points from image data; Estimating camera pose based on feature points; Construct a sparse 3D point cloud based on feature points and camera pose.

[0009] In some embodiments, sparse 3D point clouds are registered and fused with dense environment models to obtain a 3D scene model, including: Using a dense environment model as the environmental benchmark, the sparse 3D point cloud is registered with the dense environment model. The dense environment model is obtained in advance through a deep learning algorithm. Pose optimization is performed on sparse 3D point cloud and dense environment models based on the point cloud registration results. A scale consistency adjustment is performed on the sparse 3D point cloud and the dense environment model to obtain a 3D scene model.

[0010] In some embodiments, constructing a human pose model based on image data using a human pose estimation algorithm includes: Identify the coordinates of the pilot's skeletal joints based on image data; Project the skeleton joints onto the 3D scene model based on their coordinates; The pilot's posture is reconstructed based on the coordinates of the skeletal joints in the 3D scene model, and a human posture model is constructed.

[0011] In some embodiments, behavioral data is obtained based on the interaction of a human posture model within a 3D scene model, including: Based on the spatial distance and orientation relationship of the human posture model in the 3D scene model, the pilot's behavior is obtained; The pilot's behavior is stored as structured data to obtain behavioral data.

[0012] In some embodiments, issuing an alert when behavioral data exceeds behavioral standards includes: When behavioral data exceeds the behavioral standard range, warnings are issued through voice prompts, visual alerts, or event push notifications.

[0013] In some embodiments, the method further includes: The system displays 3D scene models and human posture models through a visual interface.

[0014] Secondly, embodiments of this application provide a pilot behavior detection device, comprising: The data acquisition module is used to acquire image data; The scene construction module is used to construct a 3D scene model based on image data and a 3D scene reconstruction algorithm. The pose construction module is used to construct a human pose model based on human pose estimation algorithms from image data. The behavior analysis module is used to obtain behavioral data based on the interaction of the human posture model in the 3D scene model; The early warning module is used to issue early warnings for behavioral data when the behavioral data exceeds the preset behavioral standard range.

[0015] Thirdly, embodiments of this application provide an aircraft that applies the method of the first aspect or includes the apparatus of the second aspect.

[0016] This application combines the pilot's posture with the three-dimensional scene in which the pilot is located, and comprehensively and intuitively monitors and analyzes the pilot's behavior based on the interaction between the pilot's posture and the three-dimensional scene, so as to improve the accuracy of pilot behavior detection. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a pilot behavior detection method provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating step S100 of a pilot behavior detection method provided in this embodiment of the present disclosure; Figure 3 This is a flowchart illustrating step S200 of a pilot behavior detection method provided in this embodiment of the present disclosure; Figure 4 This is a flowchart illustrating step S201 of a pilot behavior detection method provided in this embodiment of the present disclosure; Figure 5 This is a flowchart illustrating step S202 of a pilot behavior detection method provided in this embodiment of the present disclosure; Figure 6 This is a flowchart illustrating step S300 of a pilot behavior detection method provided in this embodiment of the present disclosure; Figure 7 This is a flowchart illustrating step S400 of a pilot behavior detection method provided in this embodiment of the present disclosure; Figure 8 This is a schematic diagram of the structure of a pilot behavior detection device provided in an embodiment of this disclosure.

[0019] Explanation of icon numbers: 100. Data Acquisition Module; 101. Visual Sensor; 102. Visual Calibration Module; 103. Time Synchronization Module; 200. Scene Construction Module; 201. Dense Scene Reconstruction Module; 202. Online Sparse Scene Reconstruction Module; 203. Scene Registration Module; 300. Pose Construction Module; 301. Skeleton Joint Recognition Module; 302. Coordinate Projection Module; 303. Pose Restoration Module; 400. Behavior Analysis Module; 500. Early Warning Module; 600. Visualization Module. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0022] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0023] The use of "applies to" or "configured to" in this application implies open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0024] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0025] Firstly, embodiments of this application provide a method for detecting pilot behavior, such as... Figure 1 As shown, it includes the following steps: S100, Acquire image data.

[0026] The image data consists of images collected synchronously from multiple perspectives. First, environmental image data is collected offline to provide a foundation for the subsequent construction of a 3D scene model. Then, images including the pilot and the environment in which the pilot is located are collected online in real time to provide a foundation for the subsequent construction of a 3D scene model and a human posture model.

[0027] like Figure 2 As shown, step S100 includes S101-S102.

[0028] S101, acquire image sequences through multiple visual sensors 101 in different directions.

[0029] The visual sensor 101 can be a depth camera, industrial camera, or binocular camera, used to acquire images of the cockpit from multiple different directions. For example, it can capture the pilot's upper body posture and environment from the upper front of the cockpit, the coordinated movements of the pilot and co-pilot from the side front of the cockpit, or the pilot's hand interactions with the joystick or buttons in the instrument panel area. Multiple visual sensors 101 positioned at different angles within the cockpit ensure that there are no spatial blind spots regarding the pilot's movements and the cockpit environment and interacting objects.

[0030] Each vision sensor 101 continuously acquires images or video streams to form an image sequence. Each frame of the image contains timestamp information, providing a time reference for the time alignment of subsequent image sequences.

[0031] Before deployment, the vision sensor 101 is calibrated to obtain its internal and external parameters. The internal parameters include focal length, principal point, and distortion coefficients, while the external parameters are the spatial relative positions and orientations of each vision sensor 101. This ensures that the images obtained by multiple vision sensors 101 in different orientations have a unified spatial reference frame. The calibration of the vision sensor 101 can be performed using the Zhang Zhengyou calibration method with a Zhang Zhengyou calibration board, or it can be assisted by structured light.

[0032] S102. Align the image sequence over time to obtain image data.

[0033] Because of the time difference caused by hardware response delay and transmission differences in the image sequences acquired by multiple visual sensors 101, it is necessary to time-align the image sequences. Based on the timestamp of each frame of different image sequences, hardware triggering or time synchronization protocols are used to align the image sequences from multiple visual sensors 101 to ensure that the pilot's actions and the environment in the cockpit corresponding to the image sequences obtained by different visual sensors 101 at the same time are consistent with the data in the subsequent steps.

[0034] The hardware triggering method involves controlling all visual sensors 101 to acquire images simultaneously via hardware signals, such as synchronization pulses, to avoid time deviations caused by the response delay of the visual sensors 101 themselves.

[0035] The time synchronization protocol can be PTP (Precision Time Protocol), which calibrates the timestamps of different sensors through the network to achieve microsecond-level time synchronization, adapting to scenarios that require data transmission over the network.

[0036] By aligning image sequences over time, modeling distortion caused by temporal inconsistencies can be avoided, thereby improving the modeling accuracy of subsequent 3D scene models and human pose models.

[0037] S200. Construct a 3D scene model based on the image data and a 3D scene reconstruction algorithm.

[0038] like Figure 3 As shown, step S200 includes S201-S202.

[0039] S201. Construct a sparse 3D point cloud based on image data.

[0040] Sparse 3D point clouds are used to preliminarily reconstruct the environment inside the cockpit through feature association and spatial calculation of 2D image data, providing reference coordinates for the subsequent construction of 3D scene models.

[0041] like Figure 4As shown, step S201 includes S2011-S2013.

[0042] S2011. Extract feature points from image data.

[0043] Feature points are pixels within each image of image data that possess uniqueness, stability, and matchability. In one embodiment, feature points of an image can be regions with distinct edges, such as instrument panel borders, joystick outlines, button edges, etc.; or they can be regions with geometrical abrupt changes, such as the boundary between the cockpit and the instrument panel.

[0044] Feature points can be extracted using feature extraction algorithms that satisfy scale invariance, rotation invariance, and illumination robustness. Examples include SIFT (Scale-Invariant Feature Transform) and SuperPoint. Different locations can be selected based on their characteristics. For instance, SIFT can be used for feature point extraction on multiple dashboard surfaces with complex textures; while SuperPoint can be used for low-texture or highly reflective scenes, such as the metal surface of a joystick or smooth areas of a cabin wall.

[0045] S2012. Estimate camera pose based on feature points.

[0046] The camera pose is the position and orientation of the vision sensor 101 in three-dimensional space. The position can be represented by translation parameters, and the orientation can be represented by rotation parameters, which are used to convert the pixel coordinates in the image into three-dimensional space coordinates.

[0047] The camera pose can be calculated using a pose estimation algorithm, such as the SfM (Structure from Motion) algorithm, which constructs the pose of all visual sensors 101 by matching feature points.

[0048] In one embodiment, since feature matching errors and camera calibration errors may cause pose deviations, the obtained camera pose can be optimized using the bundle adjustment method. The optimization variables are all camera poses and the 3D coordinates of feature points, and the feature point projection error is the objective function. The overall projection error is minimized through iterative optimization using a nonlinear least squares algorithm, ensuring the consistency of camera pose and providing a reliable camera spatial position reference for subsequent 3D point calculations.

[0049] S2013. Construct a sparse 3D point cloud based on feature points and camera pose.

[0050] Based on the camera pose, the 3D coordinates of each feature point are calculated using triangulation to construct a sparse 3D point cloud. Triangulation involves calculating the 3D coordinates of the corresponding point in the same coordinate system for any pair of matched feature points using the camera poses of the two vision sensors 101.

[0051] Sparse 3D point clouds have small data requirements and fast construction speed, which can meet the needs of real-time monitoring. At the same time, sparse 3D point clouds retain key structural information, providing a data foundation for subsequent registration and fusion.

[0052] S202. Construct a dense scene model based on the image data.

[0053] The dense environment model is a high-precision 3D model that includes all the environments inside the cockpit, including environmental details such as the instrument panel, buttons, joysticks, and bulkheads.

[0054] The dense environment model is obtained through deep learning algorithms. It is trained on 3D scan data to perform dense reconstruction of standard layout images inside the cockpit, such as models based on 3D Gaussian Splatting or NeRF (Neural Radiance Field), to obtain a dense environment model applicable to the same type of aircraft. Different aircraft can be adjusted according to the aircraft model.

[0055] S203. Register and fuse the sparse 3D point cloud with the dense environment model to obtain a 3D scene model.

[0056] like Figure 5 As shown, step S2032 includes S2031-S2033.

[0057] S2031. Using the dense environment model as the environmental benchmark, the sparse 3D point cloud is registered with the dense environment model.

[0058] Point cloud registration aims to find the spatial transformation relationship between sparse 3D point clouds and dense environment models, aligning them in the same coordinate system and ensuring that the feature points in the sparse point cloud model coincide with the corresponding points in the dense environment model in the same coordinate system.

[0059] S2032. Based on the point cloud registration results, perform pose optimization on sparse 3D point cloud and dense environment models.

[0060] After obtaining the initial transformation relationship through registration, a global optimization algorithm is used to further eliminate local registration errors, ensuring that the sparse point cloud and the dense model are completely consistent in the overall spatial pose, avoiding spatial contradictions caused by local registration errors, and ensuring that the 3D scene model can realistically reflect the cockpit spatial structure.

[0061] S2033. Scale consistency adjustment is performed on the sparse 3D point cloud and the dense environment model to obtain the 3D scene model.

[0062] By adjusting the scale consistency of sparse 3D point clouds and dense environment models, the proportional size of sparse 3D point clouds and dense environment models can be kept consistent, avoiding scaling deviations caused by differences in data sources.

[0063] S300. Construct a human pose model based on the human pose estimation algorithm using image data.

[0064] like Figure 6 As shown, step S300 includes S301-S303.

[0065] S301. Identify the coordinates of the pilot's skeletal joints based on image data.

[0066] The skeletal joints of a pilot are key points for human movement. Priority is given to the core key points related to the pilot's flight operations, such as the head, neck, shoulders, elbows, wrists, and hips. Changes in the position of these skeletal joints directly reflect operational behavior.

[0067] The coordinates of the skeletal joints can be identified from the image data using human pose estimation algorithms, such as ViTPose (visual pose estimation) or 3D Mesh Recovery (3D mesh reconstruction), to achieve accurate reconstruction of the overall 3D skeletal structure of the pilot.

[0068] S302. Project the skeleton joints onto the 3D scene model based on their coordinates.

[0069] Based on the intrinsic and extrinsic parameters and pose parameters of the vision sensor 101, a mapping relationship is established between the coordinates of the skeleton joints and the coordinate system of the 3D scene model, and the skeleton joints are projected into the 3D scene model.

[0070] S303. Reconstruct the pilot's posture based on the coordinates of the skeletal joints in the 3D scene model, and construct a human posture model.

[0071] The skeleton is formed by connecting the skeletal joints, and a human posture model is constructed. The human posture model includes the coordinates of the skeletal joints in the three-dimensional scene, bone length, joint angles and other parameters, which can dynamically reflect the pilot's real-time position on the platform.

[0072] Based on the constraints of human kinematics, the length of a pilot's skeleton is fixed, the range of motion of their joints is limited, and the coordination of limbs is adjusted to restore the pilot's posture.

[0073] By placing the human posture model and the 3D scene model in the same coordinate system, the interaction between the pilot's posture and the environment can be directly calculated, such as whether the head orientation corresponds to the instrument panel and whether the hands are in the control stick operation area, providing intuitive 3D dynamic data for flight training assessment and accident retrospective.

[0074] S400: Obtain behavioral data based on the interaction between the human posture model and the 3D scene model.

[0075] By analyzing the spatial interaction between the human posture model and the 3D scene model, we can extract pilot operational behaviors that have practical significance, and standardize and structure these behaviors to provide quantifiable and analyzable basic data for subsequent behavior assessment and risk warning.

[0076] like Figure 7 As shown, step S400 includes S401-S402.

[0077] S401. Based on the spatial distance and directional relationship between the human posture model and the three-dimensional scene model, obtain the pilot's behavior.

[0078] The spatial distance between a human posture model and a 3D scene model can be the 3D straight-line distance between key parts of the human posture model and interactive objects in the 3D scene model, such as the distance between a pilot's hand and a button, instrument, or joystick.

[0079] Directional relationships can be defined as the orientation of a human body part and the spatial orientation of an interactive object. This can be quantified by the angle between vectors. For example, the directional relationship between the head and the navigation instrument can be determined by the angle between the vector of the head's orientation and the vector of the normal direction of the navigation instrument plane.

[0080] Based on spatial distance and directional relationships, pilot behavior can be identified, such as operating equipment, acquiring information, or cooperating with others. Operating equipment can include pressing buttons, turning knobs, or pushing and pulling control sticks. Acquiring information can include viewing the main instrument panel, observing the navigation screen, and reading checklists. Cooperating with others can include communicating with the co-pilot through gestures and passing documents.

[0081] S402. Store the pilot's behavior as structured data to obtain behavioral data.

[0082] After identifying the pilot's behavior, the behavior is stored as structured data, which can be in JSON format and contains the core attributes of the behavior, such as type, time, location, object, and parameters.

[0083] Behavioral data is a collection of pilot behaviors, including not only individual behavioral information but also the temporal relationships between behaviors, such as continuous operations like checking instruments, pressing buttons, and adjusting the joystick, forming a complete behavioral sequence.

[0084] S500: Issue an early warning for behavioral data when the behavioral data exceeds the preset behavioral standard range.

[0085] The behavioral standards are a quantitative rule system built upon aviation safety regulations, aircraft operating manuals (SOPs), and historical safety data. Aviation safety regulations are the operational guidelines of civil aviation authorities, such as FAA (Federal Aviation Administration) AC 120-71B and EASA CS-25, as well as standard operating procedures provided by aircraft manufacturers, such as the pre-flight check procedures for the Boeing 737 and the control stick range limits for the Airbus A320. Historical safety data consists of behavioral data on maintaining safe flight history, analyzed through machine learning to determine the distribution range of behaviors that ensure safe flight.

[0086] The behavioral standard range is stored using structured data, and behavioral data is compared with the behavioral standard range to identify anomalous behavioral data. Identifying anomalous behavioral data can be done through parameter validation and timing logic validation.

[0087] Parameter validation compares the quantified parameters in the behavioral data with preset standard thresholds. If the parameters exceed the standard thresholds, they are judged as abnormal parameters. For example, if the standard threshold is that the left hand presses the button for more than 1.2 seconds, and the left hand presses for 0.5 seconds, it will be judged that the pressing time is less than the standard threshold, and thus a warning will be issued.

[0088] The timing logic check compares continuous actions with the standard operating procedure. If the sequence is reversed or key steps are missing, it is considered an anomaly. For example, the standard operating procedure is to check the instruments, press the button, and adjust the control stick. If the system detects that the pilot actually presses the button first and then checks the instruments, or presses the button directly without checking the instruments, it is considered an anomaly and an alert is issued.

[0089] When behavioral data exceeds the behavioral standard range, warnings are issued through voice prompts, visual alerts, or event push notifications.

[0090] In one embodiment, the pilot's physiological state can also be monitored through physiological state monitoring data, and real-time intelligent analysis can be performed in combination with behavioral data to identify behaviors that exceed the behavioral standard range. For example, if the fatigue index exceeds the fatigue threshold in the behavioral standard range, an abnormality can be detected and an early warning can be issued.

[0091] The method also includes displaying 3D scene models and human posture models through a visualization interface. The visualization interface can display static 3D scene models and dynamic human posture models in real time, transforming abstract data such as 3D spatial relationships and dynamic postures into intuitive visual information for easy observation.

[0092] Secondly, such as Figure 8 As shown in the figure, this application provides a pilot behavior detection device, including: a data acquisition module 100, a scene construction module 200, an attitude construction module 300, a behavior analysis module 400, an early warning module 500, and a visualization module 600.

[0093] The data acquisition module 100 is used to acquire image data and includes multiple vision sensors 101, a vision calibration module 102, and a time synchronization module 103.

[0094] The vision sensor 101 is used to acquire image sequences from multiple different orientations. The vision sensor 101 can be a depth camera, an industrial camera, or a binocular camera, acquiring images of the cockpit from multiple different orientations.

[0095] The vision calibration module 102 is used to calibrate the vision sensor 101 before deployment, and to obtain the internal and external parameters of the vision sensor 101.

[0096] The time synchronization module 103 is used to time-align the image sequence to obtain image data. Based on the timestamp of each frame of different image sequences, the image sequences from multiple visual sensors 101 are aligned using hardware triggering or a time synchronization protocol to ensure that the pilot's actions and the cockpit environment corresponding to the image sequences obtained by different visual sensors 101 at the same time are consistent with the data for subsequent steps.

[0097] The scene construction module 200 is used to construct a 3D scene model based on the 3D scene reconstruction algorithm according to the image data, including a dense scene reconstruction module 201, an online sparse scene reconstruction module 202, and a scene registration module 203.

[0098] The scene dense reconstruction module 201 is used to perform dense reconstruction of a standard layout image inside the cockpit by training with 3D scan data.

[0099] The online scene sparse reconstruction module 202 is used to construct a sparse 3D point cloud based on image data. The sparse 3D point cloud is used to initially reconstruct the environment inside the cockpit through feature association and spatial calculation of 2D image data, providing reference coordinates for the subsequent construction of a 3D scene model.

[0100] The scene registration module 203 is used to register and fuse sparse 3D point clouds with a preset dense environment model to obtain a 3D scene model. By finding the spatial transformation relationship between the sparse 3D point cloud and the dense environment model, the two are aligned in the same coordinate system, ensuring that the feature points in the sparse point cloud model coincide with the corresponding points in the dense environment model in the same coordinate system.

[0101] The attitude construction module 300 is used to construct a human attitude model based on image data and a human attitude estimation algorithm. It includes a skeletal joint recognition module 301, a coordinate projection module 302, and an attitude restoration module 303. The skeletal joint recognition module 301 identifies the coordinates of the pilot's skeletal joints based on image data. The coordinate projection module 302 projects the skeletal joints onto a 3D scene model. The attitude restoration module 303 restores the pilot's attitude using the coordinates of the skeletal joints in the 3D scene model, thus constructing the human attitude model. By having the human attitude model and the 3D scene model in the same coordinate system, the interaction between the pilot's attitude and the environment can be directly calculated, such as whether the head orientation corresponds to the instrument panel and whether the hands are in the joystick operation area, providing intuitive 3D dynamic data for flight training assessment and accident retrospective analysis.

[0102] The behavior analysis module 400 is used to acquire behavioral data based on the interaction between the human posture model and the 3D scene model. By analyzing the spatial interaction between the human posture model and the 3D scene model, it extracts meaningful pilot operation behaviors and standardizes and structures these behaviors to provide quantifiable and analyzable basic data for subsequent behavior assessment and risk warning.

[0103] The early warning module 500 is used to issue early warnings when behavioral data exceeds preset behavioral standard ranges. When behavioral data exceeds the standard range, early warnings are issued through voice prompts, visual alarms, or event push notifications.

[0104] The visualization module 600 is used to display 3D scene models and human posture models through a visualization interface. The visualization interface can display static 3D scene models and dynamic human posture models in real time, transforming abstract data such as 3D spatial relationships and dynamic postures into intuitive visual information for easy observation.

[0105] Thirdly, embodiments of this application provide an aircraft that applies the method of the first aspect or includes the apparatus of the second aspect.

[0106] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0107] The above provides a detailed description of a pilot behavior detection method, device, and aircraft provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting pilot behavior, characterized in that, Includes the following steps: Acquire image data; A 3D scene model is constructed based on the image data using a 3D scene reconstruction algorithm. A human pose model is constructed based on the image data using a human pose estimation algorithm; Behavioral data is obtained based on the interaction of the human posture model with the three-dimensional scene model. If the behavior data exceeds the preset behavior standard range, an early warning will be issued for the behavior data.

2. The method according to claim 1, characterized in that, The acquisition of image data includes: Image sequences are acquired using visual sensors in multiple different orientations; The image sequence is time-aligned to obtain the image data.

3. The method according to claim 1, characterized in that, Based on the image data, a 3D scene model is constructed using a 3D scene reconstruction algorithm, including: Construct sparse 3D point cloud and dense environment model based on the image data; The sparse 3D point cloud is registered and fused with the dense environment model to obtain the 3D scene model.

4. The method according to claim 3, characterized in that, Constructing a sparse 3D point cloud based on the image data includes: Extract feature points from the image data; The camera pose is estimated based on the feature points; The sparse 3D point cloud is constructed based on the feature points and the camera pose.

5. The method according to claim 3, characterized in that, The sparse 3D point cloud is registered and fused with the dense environment model to obtain a 3D scene model, including: Using the dense environment model as an environmental benchmark, the sparse 3D point cloud is registered with the dense environment model. The dense environment model is obtained in advance through a deep learning algorithm. Based on the point cloud registration results, the pose optimization is performed on the sparse 3D point cloud and the dense environment model. The sparse 3D point cloud and the dense environment model are scale-consistently adjusted to obtain the 3D scene model.

6. The method according to claim 1, characterized in that, Based on the image data, a human pose model is constructed using a human pose estimation algorithm, including: The coordinates of the pilot's skeletal joints are identified based on the image data; The skeleton joints are projected onto the three-dimensional scene model based on their coordinates. The pilot's posture is reconstructed based on the coordinates of the skeletal joints in the 3D scene model, thus constructing the human posture model.

7. The method according to claim 1, characterized in that, Behavioral data is obtained based on the interaction of the human posture model with the three-dimensional scene model, including: Based on the spatial distance and directional relationship of the human posture model in the three-dimensional scene model, the pilot's behavior is obtained; The pilot's behavior is stored as structured data to obtain the behavior data.

8. The method according to claim 1, characterized in that, Issuing an early warning when the behavioral data exceeds the behavioral standard range includes: If the behavioral data exceeds the behavioral standard range, an early warning will be issued through voice prompts, visual alarms, or event push notifications.

9. The method according to claim 1, characterized in that, The method further includes: The 3D scene model and the human posture model are displayed through a visual interface.

10. A pilot behavior detection device, characterized in that, include: The data acquisition module is used to acquire image data; The scene construction module is used to construct a 3D scene model based on the image data using a 3D scene reconstruction algorithm; The pose construction module is used to construct a human pose model based on the image data using a human pose estimation algorithm. The behavior analysis module is used to obtain behavior data based on the interaction of the human posture model in the three-dimensional scene model; The early warning module is used to issue an early warning for the behavioral data when the behavioral data exceeds the preset behavioral standard range.

11. An aircraft, characterized in that, The method described in any one of claims 1-9 or the apparatus described in claim 10 may be used.

Images