Automatic auxiliary driving method based on multi-modal information fusion

Abstract

The invention discloses an automatic auxiliary driving method based on multi-modal information fusion, and the method comprises the steps: carrying out the preprocessing of collected pilot operation data, pilot voice data and pilot image data, extracting the features of each mode, generating a sample set, and training a multi-modal Transform model. In the online stage, the trained multi-mode Transform model is adopted to evaluate the behavior state of a pilot in real time, and the evaluation result serves as input to perform mode selection and adjustment of the automatic auxiliary driving system. According to the method, pilot operation, voice and image data of different sensors are fused, more comprehensive and accurate pilot state information can be provided, the defects of a single data source in a specific scene are overcome, the robustness and accuracy of pilot behavior state recognition are improved, and the user experience is improved. And the level mode of the automatic auxiliary driving system can be adjusted in real time according to the state and operation of the pilot, and reliable automatic auxiliary driving support is provided for the pilot, so that the flight safety is ensured.

Description

Technical Field

[0001] The present invention relates to a technology in the field of civil aviation, specifically a pilot behavior status assessment and automated assisted driving method based on multimodal information fusion. Background Art

[0002] During the flight, pilots need to process a large amount of information and respond quickly, but factors such as excessive cognitive load, fatigue or emotional fluctuations may weaken their decision-making and behavior, leading to misjudgment, operational errors or unsafe behavior, thereby increasing flight risks. Therefore, accurately assessing the pilot's working status, timely identifying their behavioral changes, and providing intelligent assisted driving support have become the key to improving aviation safety and pilot work efficiency. However, most of the existing pilot behavior monitoring and assisted driving systems focus on analyzing pilot behavior through flight data recording and traditional flight control systems. These systems can often only analyze pilot operational behaviors (such as control inputs, flight trajectories, etc.), lack comprehensive evaluation of pilot behavior details, and are insufficient in real-time and the matching degree between pilots and assisted driving systems, and cannot provide sufficient support for pilots. Summary of the invention

[0003] In view of the deficiency of the prior art that the operation data, voice data and image data collected during the flight phase are not fully utilized, the present invention proposes an automated assisted driving method based on multimodal information fusion. By fusing the data of different sensors such as pilot operation, voice and image, more comprehensive and accurate pilot status information can be provided, which makes up for the deficiency of a single data source in specific scenarios, improves the robustness and accuracy of pilot behavior status recognition, and can adjust the level mode of the automated assisted driving system in real time according to the pilot status and operation, providing the pilot with reliable automated assisted driving support, thereby ensuring flight safety.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to an automated assisted driving method based on multimodal information fusion, which extracts the features of each modality and generates a sample set for training a multimodal Transformer model after preprocessing the collected pilot operation data, pilot voice data and pilot image data. In the online stage, the trained multimodal Transformer model is used to evaluate the pilot's behavior state in real time, and the evaluation result is used as input to select and adjust the mode of the automated assisted driving system.

[0006] The present invention relates to a pilot cognitive state recognition system for realizing the above method, comprising: a data acquisition module, a data processing module, a feature fusion module, a state recognition module and an automatic mode selection module, wherein: the data acquisition module collects pilot operation, voice and image data; the data processing module preprocesses the collected data and extracts key features from the preprocessed data; the feature fusion module receives each modal feature through a multimodal Transformer model, combines each modal feature by a feature splicing method, and forms a comprehensive feature vector; the state recognition module classifies the fused features through a fully connected layer and an output layer of the multimodal Transformer model, and outputs the behavior state of the pilot; the automatic mode selection module selects and adjusts the mode of the automatic assisted driving system according to the above output results. Technical Effects

[0007] The present invention uses a multimodal Transformer model to extract and fuse the pilot's operation data, voice data and image data, and uses feature splicing to generate a comprehensive feature vector to achieve deep interaction and information integration of different modal features; based on the automatic mode selection module, according to the pilot's behavior state output by the multimodal Transformer model, the working mode of the automatic assisted driving system is automatically adjusted to improve flight safety; based on the efficient calculation and real-time characteristics of the multimodal Transformer, the real-time recognition and dynamic adjustment of the cognitive state during flight are realized. Compared with the prior art, the present invention improves the robustness and accuracy of the recognition of the pilot's cognitive state by fusing operation, voice and image multimodal data; through the automatic mode selection module, the pilot's behavior state recognition results are matched with the assisted driving system, the system's adaptability to changes in the pilot's state is improved, the operating load is reduced, and the flight safety is enhanced; in complex dynamic mission scenarios, the system can respond to changes in the pilot's state in real time and adjust the system mode to achieve a more robust mission execution capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flow chart of the present invention;

[0009] Figure 2 The training model architecture diagram for the embodiment;

[0010] Figure 3 This is a system architecture diagram of an embodiment. DETAILED DESCRIPTION

[0011] like Figure 1As shown, this embodiment involves a pilot behavior status evaluation and automated assisted driving method based on multimodal information fusion, which generates a sample set by preprocessing the collected pilot operation data, pilot voice data and pilot image data and then extracting features for training the multimodal Transformer model. In the online stage, the trained multimodal Transformer model is used to evaluate the pilot behavior status, and the evaluation result is used as input to select and adjust the mode of the automated assisted driving system.

[0012] The pilot voice data is collected using a cockpit voice recorder.

[0013] The pilot image data is collected using a cockpit monitoring system.

[0014] The sample set is divided into a training set (70%), a validation set (15%) and a test set (15%) in proportion.

[0015] The preprocessing includes: for pilot voice data, removing background noise through a bandpass filter, normalizing the amplitude of the audio signal, and performing frame division and windowing; for pilot image data, denoising the collected image through a Gaussian filter, and performing image enhancement through histogram equalization;

[0016] The bandpass frequency of the bandpass filter is 300 Hz to 3400 Hz; the filter type is FIR; the frame length is 20 ms to 30 ms; the window function used is: n=1,2,…,N-1, where: w(n) is the value of the window function at time n; N is the length of the window; n is the index of the current sample, ranging from 0 to N-1.

[0017] The Gaussian kernel size of the Gaussian filter is 3x3; the block size of the histogram equalization is 8x8.

[0018] The feature extraction includes: operation time interval features, standard operating procedure compliance features, key command and alarm response time features, voice content features, intonation features, speech speed features, facial behavior features, head posture and body posture features, specifically: F operation =[t_dev mean ,t_dev std ,SOP degree ,t_res mean ,t_rea max ,t_res min ], F speech =[C,F0,ΔF0,R s ], F image =[Fface ,F head ,F body ],in: and are the mean difference and standard deviation between the standard operation time interval and the actual operation time interval, SOP degree The degree to which the operation performed matches the standard operating procedure, t_res mean , t_rea max and t_res min are the average response time, maximum response time and minimum response time of key instructions and alarms respectively, C is the Mel frequency cepstrum coefficient eigenvector, F0 is the fundamental frequency eigenvector, ΔF0 is the fundamental frequency change rate vector, R s For speaking speed.

[0019] The facial behavior features described above use a key point detection algorithm to detect facial key points, extract facial key points (including eye corners, nose tip, mouth corners and eyebrows), and extract features from the detected key points, specifically: face =[E close ,E blink_rate ,D smile ,D frown ], where: E close E is the degree of eye closure; blink_rate is the blinking frequency; D smile Is the degree of smile; D frown The degree of frowning.

[0020] The key point detection algorithm refers to: using the Dlib 68-point model, locating 68 key points of the face, including the corners of the eyes, the tip of the nose, the corners of the mouth and the eyebrow area, through a regression method based on the enhanced gradient boosting tree (ERT).

[0021] The head posture feature is obtained by using the posture estimation algorithm to obtain the Euler angle of the head, specifically: head =[θ pitch ,θ yaw ,θ roll ], where: pitch θ pitch , yaw θ yaw and roll θ roll .

[0022] The posture estimation algorithm refers to: using the PnP algorithm to calculate the position of the camera and the posture of the human body through known 3D key points and their 2D projections in the image, thereby inferring the Euler angles of the head.

[0023] The body posture features described above use a skeleton estimation model to obtain key points of the body (including the head, shoulders, and wrists), and calculate the relative positions and angles between the joints, specifically:body =[d 1,2 ,d 1,3 ,θ 1,2 ,…], where: d i,j is the distance between the i-th and j-th key points; θ i,j is the angle between the i-th and j-th key points.

[0024] The skeleton estimation model refers to: using the OpenPose skeleton key point detection and posture estimation model to generate a 2D skeleton image by detecting human joints (including head, shoulders and wrists) in the image.

[0025] The multimodal Transformer model includes: input layer, self-attention mechanism layer, fusion layer, fully connected layer and output layer, wherein: after receiving the features of each modality, the input layer maps these features to the same dimension through linear transformation to obtain the features of each modality after embedding; after the self-attention mechanism layer captures the relationship between the modalities, the fusion layer combines the features of each modality by feature splicing, and obtains the predicted probability distribution of the behavior state through the fully connected layer. The output layer converts the prediction results into the category of the pilot's behavior state.

[0026] The predicted probability distribution of the behavior state is P = {P normal ,P fatigue ,P distraction ,P incapacitation}, where P normal ,P fatigue ,P distraction ,P incapacitation They represent the probabilities of the four behavioral states of normal, fatigue, distraction, and disability. The behavioral state with the highest probability will be used as the final prediction result S=argmax(P), where S represents the predicted behavioral state (one of normal, fatigue, distraction, and disability).

[0027] The multimodal Transformer model is trained and optimized using a cross entropy loss function, specifically: Where: L is the loss value; N is the total number of samples; y i is the true label; is the predicted probability of the model for sample i.

[0028] The automated assisted driving system will dynamically adjust the system's automation level mode based on the pilot's behavior state changes. The automation level is divided into 5 modes, as shown in the following table:

[0029] The automation level mode adjustment is based on the following rules:

[0030] 1) If the behavior recognition result S = normal, the automation mode is set to mode 0;

[0031] 2) If the behavior recognition result S = distraction and P distraction ≤Td1, the automation mode is set to mode 1;

[0032] 3) If the behavior recognition result S = fatigue and P fatigue ≤Tf1 or S=distraction and P distraction ≤Td2, the automation mode is set to mode 2;

[0033] 4) If the behavior recognition result S = fatigue and P fatigue ≤Tf2 or S=disabled and P incapacitation ≤TI1, the automation mode is set to mode 3;

[0034] 5) If the behavior recognition result S = fatigue and P fatigue ≤Tf3 or S=disabled and P incapacitation ≤TI2, the automation mode is set to mode 4;

[0035] Among them, Td1 and Td2 are low and high distraction probability thresholds, respectively; Tf1, Tf2, and Tf3 are low, medium, and high fatigue probability thresholds, respectively; TI1 and TI2 are low and high disability probability thresholds, respectively.

[0036] The automation mode level adjustment is triggered by the change of pilot behavior state. Pilot state change rate i=4. If ΔS i >R p , it is considered as a sudden change in the pilot's state, triggering a rapid adjustment; if ΔS i ≤T p , only at the minimum adjustment time interval I min Then evaluate whether adjustments should be made.

[0037] like Figure 3As shown, a pilot behavior state assessment and automated assisted driving system based on multimodal information fusion for implementing the above method involved in this embodiment can execute the above pilot behavior state assessment and automated assisted driving method based on multimodal information fusion, including: a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module, a state recognition module and an automated mode selection module, wherein: the data preprocessing module preprocesses the collected data; the feature extraction module extracts key features from the preprocessed data; the feature fusion module receives each modal feature through a multimodal Transformer model, maps these features to the same dimension through linear transformation, obtains the embedded features of each modality, and inputs the self-attention mechanism to capture the relationship between the modalities, and the fusion layer combines the features of each modality by feature splicing to form a comprehensive feature vector; the state recognition module classifies the fused features through the fully connected layer and output layer of the multimodal Transformer model, and outputs the pilot's behavior state; the automated mode selection module selects and adjusts the mode of the automated assisted driving system according to the above output results.

[0038] The data acquisition module includes: an operation data acquisition unit, a voice data acquisition unit and an image data acquisition unit, wherein: the operation data acquisition unit obtains the pilot's operation data through flight system feedback, the voice data acquisition unit obtains the pilot's voice data through a voice acquisition device installed in the cockpit, and the image data acquisition unit obtains the pilot's image data through an image acquisition device installed in the cockpit.

[0039] The data preprocessing module includes: a voice data processing unit and an image data processing unit, wherein: the voice data processing unit performs denoising, normalization, framing and windowing processing according to the pilot's voice information to obtain preprocessed voice information suitable for subsequent feature extraction; the image data processing unit performs denoising and enhancement processing according to the pilot's image information to obtain preprocessed image information.

[0040] The feature extraction module includes: an operation feature extraction unit, a voice feature extraction unit and an image feature extraction unit, wherein: the operation feature extraction unit performs feature extraction based on the pilot's operation information to obtain operation features, the voice feature extraction unit performs feature extraction based on the pre-processed voice information to obtain voice features, and the image feature extraction unit performs feature extraction based on the pre-processed image information to obtain image features.

[0041] The feature fusion module includes: a feature mapping unit, a self-attention mechanism unit and a feature fusion unit, wherein: the feature mapping unit performs linear transformation according to the input features of each modality to obtain the embedded features of each modality, the self-attention mechanism unit captures the relationship between the modalities according to the embedded feature information, and the feature fusion unit performs feature splicing processing according to the relationship information between the modalities to obtain the fused comprehensive feature vector.

[0042] The state recognition module includes: a fully connected layer mapping unit and an output unit, wherein: the fully connected layer mapping unit performs nonlinear transformation according to the comprehensive feature vector to obtain the feature vector mapped to the classification space, and the output unit performs output layer processing according to the feature vector in the classification space to obtain the final output result of the classification.

[0043] The automation mode selection module includes: a mode adjustment trigger unit, a mode decision unit and a feedback adjustment unit, wherein: the mode adjustment trigger unit identifies the change of the pilot state according to the real-time pilot behavior state information, and obtains a trigger signal whether the mode adjustment is required; the mode decision unit performs reasoning and logical processing according to the trigger signal provided by the adjustment trigger unit and the preset decision rules to obtain the automation mode level and its adjustment strategy to be currently selected; the feedback adjustment unit monitors the system response and pilot reaction after the automation mode is switched according to the output result of the mode decision unit, and performs real-time feedback and adjustment to ensure the continuous matching of the system automation mode and the pilot behavior state.

[0044] Compared with the prior art, the present invention can capture the pilot's behavioral state more comprehensively and accurately by processing the multimodal fusion of pilot operation data, voice data and image data. Compared with the traditional state recognition technology that only relies on a single data source, this method improves the reliability and accuracy of pilot behavior state recognition; by training the multimodal Transformer model, it can realize online real-time evaluation of pilot behavior state, and dynamically adjust the automation mode according to the changes in pilot behavior state. When abnormalities such as fatigue, distraction, and disability are detected, the system quickly adjusts the automation level and provides auxiliary support, thereby reducing the risk of human error; it can more accurately select and adjust the mode level of the automated assisted driving system, so that the system can respond to changes in pilot behavior state more efficiently, ensure a high degree of coordination between the pilot and the system, and improve flight safety.

[0045] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for identifying pilot cognitive state based on multimodal feature fusion, characterized in that: After preprocessing the collected pilot operation data, pilot voice data and pilot image data, the features of each modality are extracted and a sample set is generated for training the multimodal Transformer model. In the online stage, the trained multimodal Transformer model is used to evaluate the pilot's behavioral status in real time, and the evaluation results are used as input to select and adjust the mode of the automated assisted driving system.

2. The method for identifying pilot cognitive state based on multimodal feature fusion according to claim 1, characterized in that: The preprocessing includes: for pilot speech data, removing background noise through a bandpass filter, normalizing the amplitude of the audio signal, and performing frame division and windowing; for pilot image data, denoising the collected image through a Gaussian filter, and performing image enhancement through histogram equalization.

3. The method for identifying pilot cognitive status based on multimodal feature fusion according to claim 1, characterized in that: The feature extraction includes: operation time interval features, standard operating procedure compliance features, key command and alarm response time features, voice content features, intonation features, speech speed features, facial behavior features, head posture and body posture features, specifically: F operation =[t_dev mean ,t_dev std ,SOP degree ,t_res mean ,t_res max ,t_res min ], F speech =[C,F0,ΔF0,R s ],F image =[F face ,F head ,F body ],in: and are the mean difference and standard deviation between the standard operation time interval and the actual operation time interval, SOP degree The degree to which the performed operation matches the standard operating procedure, t_res mean , t_res max and t_res min are the average response time, maximum response time and minimum response time of key instructions and alarms respectively, C is the Mel frequency cepstrum coefficient eigenvector, F0 is the fundamental frequency eigenvector, ΔF0 is the fundamental frequency change rate vector, R s For speaking speed.

4. The method for identifying pilot cognitive state based on multimodal feature fusion according to claim 2, characterized in that: The facial behavior features described above use a key point detection algorithm to detect facial key points, extract facial key points (including eye corners, nose tip, mouth corners and eyebrows), and extract features from the detected key points, specifically: face =[E close ,E blink_rate ,D smile ,D frown ], where: E close E is the degree of eye closure; blink_rate is the blinking frequency; D smile is the degree of smile; D frown for the degree of frowning; The key point detection algorithm refers to: using the Dlib68 point model, locating 68 key points of the face, including the corners of the eyes, the tip of the nose, the corners of the mouth and the eyebrow area, through a regression method based on the enhanced gradient boosting tree (ERT).

5. The method for identifying pilot cognitive status based on multimodal feature fusion according to claim 2, characterized in that: The head posture feature is obtained by using the posture estimation algorithm to obtain the Euler angle of the head, specifically: head =[θ pitch ,θ yaw ,θ roll ], where: pitch θ pitch , yaw θ yaw and roll θ roll ; The posture estimation algorithm refers to: using the PnP algorithm to calculate the position of the camera and the posture of the human body through known 3D key points and their 2D projections in the image, thereby inferring the Euler angles of the head.

6. The method for identifying pilot cognitive status based on multimodal feature fusion according to claim 2, characterized in that: The body posture features described above use a skeleton estimation model to obtain key points of the body (including the head, shoulders, and wrists), and calculate the relative positions and angles between the joints, specifically: body =[d 1,2 ,d 1,3 ,θ 1,2 ,…], where: d i,j is the distance between the i-th and j-th key points; θ i,j is the angle between the i-th and j-th key points; The skeleton estimation model refers to: using the OpenPose skeleton key point detection and posture estimation model to generate a 2D skeleton image by detecting human joints (including head, shoulders and wrists) in the image.

7. The method for identifying pilot cognitive status based on multimodal feature fusion according to claim 1, characterized in that: The multimodal Transformer model includes: an input layer, a self-attention mechanism layer, a fusion layer, a fully connected layer and an output layer, wherein: after receiving the features of each modality, the input layer maps these features to the same dimension through linear transformation to obtain the features of each modality after embedding; after the self-attention mechanism layer captures the relationship between the modalities, the fusion layer combines the features of each modality by feature splicing, and obtains the predicted probability distribution of the behavior state through the fully connected layer, and the output layer converts the predicted results into the category of the pilot's behavior state; The predicted probability distribution of the behavior state is P = {P normal ,P fatigue ,P distraction ,P incapacitation }, where P normal ,P fatigue ,P distraction ,P incapacitation They represent the probabilities of the four behavioral states of normal, fatigue, distraction and disability respectively. The behavioral state with the largest probability will be used as the final prediction result S=argmax(P), where S represents the predicted behavioral state.

8. The method for identifying pilot cognitive status based on multimodal feature fusion according to claim 1, characterized in that: The automated assisted driving system dynamically adjusts the automation level mode of the system based on the changes in the pilot's behavior state, where the automation level is divided into 5 modes, including: Mode 0: The auxiliary driving system is turned off, at the minimum automation level, and the pilot on board controls the driving of the aircraft as the direct executor; Mode 1: Based on Mode 0, the automated assisted pilot system needs to complete the attitude holding and track keeping functions in flight control and warn the pilot; Mode 2: Based on Mode 1, the automated assisted pilot system needs to complete the flight plan holding function in flight control and warn the pilot; Mode 3: Based on Mode 2, the automated assisted pilot system needs to complete the flight separation maintenance function in flight control and warn the pilot and ground personnel; Mode 4: Based on Mode 3, the automated assisted pilot system needs to complete the automatic landing function in the flight control and warn the pilot and ground personnel. At this time, the pilot is out of the aircraft control loop; The automation level mode adjustment is based on the following rules: 1) When the behavior recognition result S = normal, the automation mode is set to mode 0; 2) When the behavior recognition result S = distraction and P distraction ≤Td1, the automation mode is set to mode 1; 3) When the behavior recognition result S = fatigue and P fatigue ≤Tf1 or S=distraction and P distraction ≤Td2, the automation mode is set to mode 2; 4) When the behavior recognition result S = fatigue and P fatigue ≤Tf2 or S=disabled and P incapacitation ≤TI1, the automation mode is set to mode 3; 5) When the behavior recognition result S = fatigue and P fatigue ≤Tf3 or S=disabled and P incapacitation ≤TI2, the automation mode is set to mode 4; Among them, Td1 and Td2 are low and high distraction probability thresholds, respectively; Tf1, Tf2, and Tf3 are low, medium, and high fatigue probability thresholds, respectively; TI1 and TI2 are low and high disability probability thresholds, respectively.

9. A pilot cognitive state recognition system implementing the method described in any one of claims 1 to 8, characterized in that: include: A data acquisition module, a data processing module, a feature fusion module, a state recognition module and an automatic mode selection module, wherein: the data acquisition module collects pilot operation, voice and image data; the data processing module preprocesses the collected data and extracts key features from the preprocessed data; the feature fusion module receives each modal feature through a multimodal Transformer model, combines each modal feature by feature splicing, and forms a comprehensive feature vector; the state recognition module classifies the fused features through the fully connected layer and output layer of the multimodal Transformer model, and outputs the pilot's behavior state; the automatic mode selection module selects and adjusts the mode of the automatic assisted driving system according to the above output results.

10. The pilot cognitive status recognition system according to claim 9, characterized in that: The data acquisition module includes: an operation data acquisition unit, a voice data acquisition unit and an image data acquisition unit, wherein: the operation data acquisition unit obtains the pilot's operation data through the flight system feedback, the voice data acquisition unit obtains the pilot's voice data through the voice acquisition device installed in the cockpit, and the image data acquisition unit obtains the pilot's image data through the image acquisition device installed in the cockpit; The data preprocessing module includes: a voice data processing unit and an image data processing unit, wherein: the voice data processing unit performs denoising, normalization, framing and windowing processing according to the pilot's voice information to obtain preprocessed voice information suitable for subsequent feature extraction; the image data processing unit performs denoising and enhancement processing according to the pilot's image information to obtain preprocessed image information; The feature extraction module includes: an operation feature extraction unit, a voice feature extraction unit and an image feature extraction unit, wherein: the operation feature extraction unit performs feature extraction based on the pilot operation information to obtain the operation feature, the voice feature extraction unit performs feature extraction based on the pre-processed voice information to obtain the voice feature, and the image feature extraction unit performs feature extraction based on the pre-processed image information to obtain the image feature; The feature fusion module includes: a feature mapping unit, a self-attention mechanism unit and a feature fusion unit, wherein: the feature mapping unit performs linear transformation according to the input features of each modality to obtain the embedded features of each modality, the self-attention mechanism unit captures the relationship between the modalities according to the embedded feature information, and the feature fusion unit performs feature splicing processing according to the relationship information between the modalities to obtain the fused comprehensive feature vector; The state recognition module includes: a fully connected layer mapping unit and an output unit, wherein: the fully connected layer mapping unit performs nonlinear transformation according to the comprehensive feature vector to obtain a feature vector mapped to the classification space, and the output unit performs output layer processing according to the feature vector in the classification space to obtain the final output result of the classification; The automation mode selection module includes: a mode adjustment trigger unit, a mode decision unit and a feedback adjustment unit, wherein: the mode adjustment trigger unit identifies the change of the pilot state according to the real-time pilot behavior state information, and obtains a trigger signal whether the mode adjustment is required; the mode decision unit performs reasoning and logical processing according to the trigger signal provided by the adjustment trigger unit and the preset decision rules to obtain the automation mode level and its adjustment strategy to be currently selected; the feedback adjustment unit monitors the system response and pilot reaction after the automation mode is switched according to the output result of the mode decision unit, and performs real-time feedback and adjustment to ensure the continuous matching of the system automation mode and the pilot behavior state.

Images