Automated assisted driving method based on multi-modal information fusion

Abstract

The application discloses an automatic auxiliary driving method based on multi-modal information fusion, which comprises the following steps: collecting pilot operation data, pilot voice data and pilot image data, preprocessing the collected pilot operation data, pilot voice data and pilot image data, extracting features of each mode and generating a sample set for training a multi-modal Transformer model, adopting the trained multi-modal Transformer model in an online stage to evaluate the behavior state of the pilot in real time, and taking the evaluation result as input to select and adjust the mode of the automatic auxiliary driving system. The application can provide more comprehensive and accurate pilot state information by fusing the data of different sensors of the pilot operation, voice and image, can make up for the deficiency of a single data source in a specific scene, can improve the robustness and accuracy of pilot behavior state recognition, and can adjust the level mode of the automatic auxiliary driving system in real time according to the pilot state and operation, can provide reliable automatic auxiliary driving support for the pilot, and can ensure flight safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of civil aviation, and particularly relates to an automatic auxiliary driving method based on multi-modal information fusion. BACKGROUND

[0002] During flight, pilots need to process a large amount of information and respond quickly, but factors such as excessive cognitive load, fatigue or emotional fluctuations may weaken their decision-making and behavior, leading to incorrect judgments, operational errors or unsafe behavior, thereby increasing flight risk. Therefore, accurately assessing the working state of the pilot, timely identifying changes in their behavior, and providing intelligent auxiliary driving support have become the key to improving aviation safety and pilot work efficiency. However, existing pilot behavior monitoring and auxiliary driving systems mostly focus on analyzing pilot behavior through flight data records and traditional flight control systems, and these systems can only analyze the pilot's operational behavior (such as control input, flight trajectory, etc.), lack comprehensive assessment of pilot behavior details, and have deficiencies in real-time performance and pilot matching with the auxiliary driving system, which cannot provide sufficient support for pilots. SUMMARY

[0003] The present application proposes an automatic auxiliary driving method based on multi-modal information fusion to address the shortcomings of the prior art in not fully utilizing the operation data, voice data and image data collected during the flight phase. By fusing the data of different sensors of pilot operation, voice and image, more comprehensive and accurate pilot state information can be provided, making up for the shortcomings of a single data source in specific scenarios, improving the robustness and accuracy of pilot behavior state recognition, and enabling real-time adjustment of the level mode of the automatic auxiliary driving system according to the pilot state and operation, thereby providing reliable automatic auxiliary driving support for pilots and ensuring flight safety.

[0004] The present application is implemented through the following technical solutions:

[0005] The present application relates to an automatic auxiliary driving method based on multi-modal information fusion, which extracts the features of each modality and generates a sample set after preprocessing the collected pilot operation data, pilot voice data and pilot image data, for training a multi-modal Transformer model. In the online phase, the trained multi-modal Transformer model is used to evaluate the pilot's behavior state in real time, and the evaluation results are used as input for mode selection and adjustment of the automatic auxiliary driving system.

[0006] The application relates to a multi-modal information fusion-based automated auxiliary driving system for realizing the above method, comprising a data acquisition module, a data processing module, a feature fusion module, a state recognition module and an automated mode selection module, wherein: the data acquisition module acquires pilot operation, voice and image data; the data processing module pre-processes the acquired data and extracts key features from the pre-processed data; the feature fusion module receives various modal features through a multi-modal Transformer model, combines the various modal features in a feature splicing manner to form a comprehensive feature vector; the state recognition module classifies the fused features through the full connection layer and the output layer of the multi-modal Transformer model and outputs the behavior state of the pilot; and the automated mode selection module selects and adjusts the mode of the automated auxiliary driving system according to the output result.

[0007] Technical effects

[0008] The application extracts and fuses pilot operation data, voice data and image data through a multi-modal Transformer model, generates a comprehensive feature vector in a feature splicing manner, realizes deep interaction and information integration of different modal features, automatically adjusts the working mode of the automated auxiliary driving system according to the pilot behavior state output by the multi-modal Transformer model based on the automated mode selection module, and improves flight safety, realizes real-time recognition and dynamic adjustment of the cognitive state during flight based on the efficient calculation and real-time characteristics of the multi-modal Transformer. Compared with the prior art, the application improves the robustness and accuracy of the multi-modal information fusion-based automated auxiliary driving by fusing operation, voice and image multi-modal data, improves the adaptability of the system to the pilot state change, reduces the operation load and enhances the flight safety by matching the pilot behavior state recognition result with the auxiliary driving system through the automated mode selection module, and realizes more robust task execution capability in a complex dynamic task scene, the system can respond to the pilot state change in real time and adjust the system mode. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 The flowchart of the application is shown in the figure;

[0010] Figure 2 The model architecture diagram of the embodiment is shown in the figure;

[0011] Figure 3 The system architecture diagram of the embodiment is shown in the figure. DETAILED DESCRIPTION

[0012] As Figure 1As shown, the embodiment relates to a pilot behavior state evaluation and automatic auxiliary driving method based on multi-modal information fusion. After pre-processing the collected pilot operation data, pilot voice data and pilot image data, feature extraction is performed to generate a sample set, which is used to train a multi-modal Transformer model. In the online stage, the trained multi-modal Transformer model is used to evaluate the pilot behavior state, and the evaluation result is used as input to select and adjust the mode of the automatic auxiliary driving system.

[0013] The pilot voice data is collected by a cockpit voice recorder.

[0014] The pilot image data is collected by a cockpit monitoring system.

[0015] The sample set is divided into a training set (70%), a validation set (15%) and a test set (15%) in proportion.

[0016] The pre-processing includes: removing background noise from the pilot voice data by a band-pass filter, normalizing the audio signal amplitude, and performing frame division and windowing; for the pilot image data, denoising the collected image by a Gaussian filter, and performing image enhancement by histogram equalization.

[0017] The band-pass frequency of the band-pass filter is 300Hz to 3400Hz; the type of the filter is FIR; the frame length is 20ms to 30ms; and the window function used is: Where: w(n) is the value of the window function at time n; N is the length of the window; n is the index of the current sample, ranging from 0 to N-1.

[0018] The Gaussian kernel size of the Gaussian filter is 3x3; and the block size of the histogram equalization is 8x8.

[0019] The feature extraction includes: operation time interval feature, standard operation procedure compliance feature, key instruction and alarm response time feature, voice content feature, tone feature, speech rate feature, facial behavior feature, head posture and body posture feature, specifically: F operation =[t_dev mean ,t_dev std ,SOP degree ,t_res mean ,t_res max ,t_res min ], F speech =[C,F0,ΔF0,R s ], F image =[F face ,Fhead F body ], wherein: and are the mean difference and the standard deviation of the actual operation time interval and the standard operation time interval, respectively, SOP degree is the matching degree of the executed operation and the standard operation procedure, t_res mean , t_res max and t_res min are the mean response time, the maximum response time and the minimum response time of the critical instruction and the alarm, respectively, C is the Mel frequency cepstrum coefficient feature vector, F0 is the fundamental frequency feature vector, AF0 is the fundamental frequency change rate vector, R s is the speech speed.

[0020] The face behavior feature uses a key point detection algorithm to detect face key points to extract face key points (including eye corners, nose tips, mouth corners and eyebrows), and extracts features from the detected key points, specifically: F face = [E close , E blink_rate , D smile , D frown ], wherein: E close is the eye closure degree; E blink_rate is the blink frequency; D smile is the smile degree; and D frown is the frown degree.

[0021] The key point detection algorithm refers to using a Dlib 68-point model to locate 68 key points of the face, including eye corners, nose tips, mouth corners and eyebrow regions, through a regression method based on an enhanced gradient boosting tree (ERT).

[0022] The head pose feature uses a pose estimation algorithm to obtain the Euler angle of the head, specifically: F head = [theta pitch , theta yaw , theta roll ], wherein: pitch theta pitch , yaw theta yaw and roll theta roll .

[0023] The pose estimation algorithm refers to using a PnP algorithm to calculate the position of the camera and the pose of the human body through known 3D key points and their 2D projections in the image, so as to calculate the head Euler angle.

[0024] The body pose feature uses a skeleton estimation model to obtain body key points (including head, shoulder and wrist), and calculates the relative positions and angles between the joints, specifically: F body = [d1,2 ,d 1,3 ,θ 1,2 ,…], where: d i,j θ is the distance between the i-th and j-th keypoints; i,j Let be the angle between the i-th and j-th key points.

[0025] The skeleton estimation model refers to the use of the OpenPose skeleton keypoint detection and pose estimation model to generate a 2D skeleton map by detecting human joints (including head, shoulders and wrists) in the image.

[0026] The multimodal Transformer model comprises an input layer, a self-attention mechanism layer, a fusion layer, a fully connected layer, and an output layer. The input layer receives features from each modality and maps these features to the same dimension through a linear transformation, obtaining the embedded features for each modality. The self-attention mechanism layer captures the relationships between modalities, the fusion layer combines the features from each modality using feature concatenation, and the fully connected layer obtains the predicted probability distribution of the behavioral state. The output layer transforms the prediction results into categories of the pilot's behavioral state.

[0027] The predicted probability distribution of the behavioral state is P = {P} normal ,P fatigue ,P distraction ,P incapacitation}, where P normal ,P fatigue ,P distraction ,P incapacitation Let S represent the probabilities of four behavioral states: normal, fatigued, distracted, and disabled. The behavioral state with the highest probability will be used as the final prediction result S = argmax(P), where S represents the predicted behavioral state (one of normal, fatigued, distracted, or disabled).

[0028] The multimodal Transformer model is trained and optimized using the cross-entropy loss function, specifically as follows: Where: L is the loss value; N is the total number of samples; y i This is a real label; Let be the predicted probability of the model for sample i.

[0029] The aforementioned automated driver assistance system will dynamically adjust its automation level mode based on changes in the pilot's behavioral state. The automation level is divided into five modes, as shown in the table below:

[0030]

[0031] The aforementioned adjustment of the automation level mode is based on the following rules:

[0032] 1) If the behavior recognition result S = normal, the automation mode is set to mode 0;

[0033] 2) If the behavior recognition result S = distraction and P distraction ≤ Td1, the automation mode is set to mode 1;

[0034] 3) If the behavior recognition result S = fatigue and P fatigue ≤ Tf1 or S = distraction and P distraction ≤ Td2, the automation mode is set to mode 2;

[0035] 4) If the behavior recognition result S = fatigue and P fatigue ≤ Tf2 or S = incapacitation and P incapacitation ≤ TI1, the automation mode is set to mode 3;

[0036] 5) If the behavior recognition result S = fatigue and P fatigue ≤ Tf3 or S = incapacitation and P incapacitation ≤ TI2, the automation mode is set to mode 4;

[0037] where Td1, Td2 are low and high distraction probability thresholds respectively; Tf1, Tf2, Tf3 are low, medium and high fatigue probability thresholds respectively; TI1, TI2 are low and high incapacitation probability thresholds respectively.

[0038] The automation mode level adjustment is triggered by the pilot behavior state change. The pilot state change rate If ΔS i > R p , it is considered as a sudden change of pilot state, triggering a fast adjustment; if ΔS i ≤ T p , it is only evaluated whether to adjust after a minimum adjustment time interval I min .

[0039] For example Figure 3As shown, a pilot behavior state evaluation and automated auxiliary driving system based on multi-modal information fusion is provided, which can execute the pilot behavior state evaluation and automated auxiliary driving method based on multi-modal information fusion, including a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module, a state recognition module, and an automated mode selection module. The data preprocessing module preprocesses the collected data. The feature extraction module extracts key features from the preprocessed data. The feature fusion module receives each modal feature through a multi-modal Transformer model, maps these features to the same dimension through linear transformation, obtains the embedding features of each modal, and inputs the self-attention mechanism to capture the relationship between the modes. After that, the fusion layer combines each modal feature in the form of feature splicing to form a comprehensive feature vector. The state recognition module classifies the fused features through the full connection layer and the output layer of the multi-modal Transformer model, and outputs the pilot's behavior state. The automated mode selection module selects and adjusts the mode of the automated auxiliary driving system according to the output result.

[0040] The data acquisition module includes an operation data acquisition unit, a voice data acquisition unit, and an image data acquisition unit. The operation data acquisition unit obtains pilot operation data through flight system feedback. The voice data acquisition unit obtains pilot voice data through a voice acquisition device installed in the cockpit. The image data acquisition unit obtains pilot image data through an image acquisition device installed in the cockpit.

[0041] The data preprocessing module includes a voice data processing unit and an image data processing unit. The voice data processing unit performs denoising, normalization, framing, and windowing processing on pilot voice information to obtain preprocessed voice information suitable for subsequent feature extraction. The image data processing unit performs denoising and enhancement processing on pilot image information to obtain preprocessed image information.

[0042] The feature extraction module includes an operation feature extraction unit, a voice feature extraction unit, and an image feature extraction unit. The operation feature extraction unit extracts operation features from pilot operation information. The voice feature extraction unit extracts voice features from preprocessed voice information. The image feature extraction unit extracts image features from preprocessed image information.

[0043] The feature fusion module comprises a feature mapping unit, a self-attention mechanism unit and a feature fusion unit, wherein: the feature mapping unit performs linear transformation according to the input features of each modality to obtain embedded features of each modality, the self-attention mechanism unit captures the relationship between modalities according to the embedded feature information, and the feature fusion unit performs feature splicing processing according to the relationship information between modalities to obtain a comprehensive feature vector after fusion.

[0044] The state recognition module comprises a fully connected layer mapping unit and an output unit, wherein: the fully connected layer mapping unit performs nonlinear transformation according to the comprehensive feature vector to obtain a feature vector mapped to a classification space, and the output unit performs output layer processing according to the feature vector in the classification space to obtain a final output result of classification.

[0045] The automatic mode selection module comprises a mode adjustment trigger unit, a mode decision unit and a feedback adjustment unit, wherein: the mode adjustment trigger unit identifies the change of the pilot state according to the real-time pilot behavior state information to obtain a trigger signal of whether mode adjustment is needed, the mode decision unit performs reasoning and logic processing according to the trigger signal provided by the adjustment trigger unit in combination with a preset decision rule to obtain the current automatic mode level to be selected and its adjustment strategy, and the feedback adjustment unit monitors the system response after automatic mode switching and the pilot reaction according to the output result of the mode decision unit to perform real-time feedback and adjustment, thereby ensuring the continuous matching of the system automatic mode and the pilot behavior state.

[0046] Compared with the prior art, the pilot operation data, voice data and image data are processed through multi-modal fusion, so that the behavior state of the pilot can be more comprehensively and accurately captured. Compared with the traditional state recognition technology relying on only a single data source, the reliability and accuracy of the pilot behavior state recognition are improved. Through training of a multi-modal Transformer model, the pilot behavior state is evaluated online and in real time, and the automatic mode is dynamically adjusted according to the change of the pilot behavior state. When abnormal conditions such as fatigue, distraction and disability are detected, the system quickly adjusts the automatic level to provide auxiliary support, thereby reducing the risk of human error. The mode level of the automatic pilot system can be more accurately selected and adjusted, so that the system can more efficiently respond to the change of the pilot behavior state, ensure the high coordination between the pilot and the system, and improve the flight safety.

[0047] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme within the scope is subject to the constraints of the present application.

Claims

1. An automated assisted driving method based on multi-modal information fusion, characterized in that, The pilot operation data, the pilot voice data and the pilot image data collected are preprocessed, features of each modality are extracted and a sample set is generated, the sample set is used for training a multi-modal Transformer model, in an online stage, the trained multi-modal Transformer model is used for real-time evaluation of the behavior state of the pilot, and the evaluation result is used as input to select and adjust the mode of the automatic flight control system; The features of each modality include: operating time interval feature, standard operating procedure compliance feature, key instruction and alarm response time feature, voice content feature, tone feature, speed feature, facial behavior feature, head posture feature and body posture feature, specifically: , , Wherein: and are the average difference and standard deviation of the standard operating time interval and the actual operating time interval respectively, is the matching degree of the executed operation and the standard operating procedure, , and are the average response time, the maximum response time and the minimum response time of the key instruction and the alarm respectively, is the Mel frequency cepstral coefficient feature vector, is the fundamental frequency feature vector, is the fundamental frequency change rate vector, is the speed. 2.The automated assistant driving method based on multi-modal information fusion according to claim 1, characterized in that, The preprocessing includes: for the pilot voice data, removing background noise through a band-pass filter, normalizing the audio signal amplitude, and performing frame division and windowing; for the pilot image data, denoising the collected image through a Gaussian filter, and enhancing the image through histogram equalization. 3.The automated assistant driving method based on multi-modal information fusion according to claim 1, characterized in that, The face behavior feature uses a key point detection algorithm to detect face key points to extract face key points (including eye corners, nose tips, mouth corners, and eyebrows), and extracts features from the detected key points, specifically: Wherein: is the eye closure degree; is the blink frequency; is the smile degree; is the frown degree; The key point detection algorithm refers to: using a Dlib 68-point model, locating 68 key points of the face including the eye corners, nose tip, mouth corners and eyebrow regions through a regression method based on an enhanced gradient boosting tree (ERT).

4. The automated assisted driving method based on multi-modal information fusion according to claim 1, characterized in that, The head posture feature obtains Euler angles of the head by using a posture estimation algorithm, and specifically includes: wherein: pitch , yaw and roll ; The pose estimation algorithm refers to: using a PnP algorithm to calculate the position of the camera and the pose of the human body through the known 3D key points and their 2D projections in the image, and thus calculating the head Euler angle. 5.The automated assistant driving method based on multi-modal information fusion according to claim 1, characterized in that, The body posture feature uses a skeleton estimation model to obtain body key points (including a head, shoulders, and wrists) and calculate relative positions and angles between joints, specifically: wherein: is a distance between the th and the th key point; is an included angle between the th and the th key point; The skeleton estimation model refers to: using an OpenPose skeleton key point detection and pose estimation model to detect the human body joint points in the image and generate a 2D skeleton graph.

6. The automated assisted driving method based on multi-modal information fusion according to claim 1, characterized in that, The multi-modal Transformer model includes: an input layer, a self-attention mechanism layer, a fusion layer, a full connection layer and an output layer, wherein: after receiving the features of each modality, the input layer maps these features to the same dimension through linear transformation to obtain the embedded features of each modality; after capturing the relationship between the modalities, the fusion layer combines the features of each modality through feature splicing, and obtains the prediction probability distribution of the behavior state through the full connection layer, and the output layer converts the prediction result into the category of the pilot behavior state; The predicted probability distribution of the behavior state is wherein, respectively represent the probabilities of the four behavior states of normal, fatigue, distraction and disability, and the behavior state with the largest probability will be the final prediction result wherein, represents the predicted behavior state. 7.The automated assistant driving method based on multi-modal information fusion according to claim 1, characterized in that, The automatic flight control system dynamically adjusts the automation level mode of the system based on the change of the behavior state of the pilot, wherein the automation level is divided into five modes, wherein: Mode 0: the automatic flight control system is turned off, at the minimum automation level, the pilot on board controls the aircraft as the direct executor; Mode 1: based on mode 0, the automatic flight control system needs to complete the attitude and track keeping functions in flight control, and alarms the pilot; Mode 2: based on mode 1, the automatic flight control system needs to complete the flight plan keeping function in flight control, and alarms the pilot; Mode 3: based on mode 2, the automatic flight control system needs to complete the flight interval keeping function in flight control, and alarms the pilot and ground personnel; Mode 4: based on mode 3, the automatic flight control system needs to complete the automatic landing function in flight control, and alarms the pilot and ground personnel, at this time the pilot is outside the aircraft control loop; The adjustment of the automation level mode is based on the following rules: 1) when the behavior recognition result , the automation mode is set to mode 0; 2) when the behavior recognition result and , the automation mode is set to mode 1; 3) when the behavior recognition result and or and , the automation mode is set to mode 2; 4) when the behavior recognition result and or and , the automation mode is set to mode 3; 5) when the behavior recognition result and or and , the automation mode is set to mode 4; wherein, , are low, medium, and high distraction probability thresholds, respectively; , , are low, medium, and high fatigue probability thresholds, respectively; , are low and high incapacitation probability thresholds, respectively.

8. A multi-modal information fusion based automated assistant driving system implementing the method of any one of claims 1-7. including: The data acquisition module, the data preprocessing module, the feature extraction module, the feature fusion module, the state recognition module and the automatic mode selection module, wherein: the data preprocessing module preprocesses the collected data; the feature extraction module extracts key features from the preprocessed data; the feature fusion module receives features of each modality through a multi-modal Transformer model, maps these features to the same dimension through linear transformation, obtains embedding features of each modality, and inputs a self-attention mechanism to capture the relationship between modalities, and then a fusion layer combines the features of each modality in a feature splicing manner to form a comprehensive feature vector; the state recognition module classifies the fused features through the full connection layer and the output layer of the multi-modal Transformer model, and outputs the behavior state of the pilot; the automatic mode selection module selects and adjusts the mode of the automatic auxiliary driving system according to the output result.

9. The automated assisted driving system based on multi-modal information fusion according to claim 8, characterized in that, The data acquisition module includes an operation data acquisition unit, a voice data acquisition unit and an image data acquisition unit, wherein: the operation data acquisition unit obtains pilot operation data through feedback from the flight system, the voice data acquisition unit obtains pilot voice data through a voice acquisition device installed in the cockpit, and the image data acquisition unit obtains pilot image data through an image acquisition device installed in the cockpit; The data preprocessing module includes a voice data processing unit and an image data processing unit, wherein: the voice data processing unit performs denoising, normalization, framing and windowing processing on pilot voice information to obtain preprocessed voice information suitable for subsequent feature extraction, and the image data processing unit performs denoising and enhancement processing on pilot image information to obtain preprocessed image information; The feature extraction module includes an operation feature extraction unit, a voice feature extraction unit and an image feature extraction unit, wherein: the operation feature extraction unit extracts features from pilot operation information to obtain operation features, the voice feature extraction unit extracts features from preprocessed voice information to obtain voice features, and the image feature extraction unit extracts features from preprocessed image information to obtain image features; The feature fusion module includes a feature mapping unit, a self-attention mechanism unit and a feature fusion unit, wherein: the feature mapping unit performs linear transformation on each modality input feature to obtain embedding features of each modality, the self-attention mechanism unit captures the relationship between modalities according to the embedding feature information, and the feature fusion unit performs feature splicing processing according to the relationship information between modalities to obtain a comprehensive feature vector after fusion; The state recognition module includes a full connection layer mapping unit and an output unit, wherein: the full connection layer mapping unit performs nonlinear transformation on the comprehensive feature vector to obtain a feature vector mapped to a classification space, and the output unit performs output layer processing on the feature vector in the classification space to obtain the final output result of classification. The automatic mode selection module comprises a mode adjustment triggering unit, a mode decision unit and a feedback adjustment unit, wherein the mode adjustment triggering unit identifies the change of the pilot state according to the real-time pilot behavior state information, and obtains a triggering signal indicating whether mode adjustment is needed; the mode decision unit performs reasoning and logical processing according to the triggering signal provided by the adjustment triggering unit and in combination with a preset decision rule, and obtains the current automatic mode level to be selected and its adjustment strategy; and the feedback adjustment unit monitors the system response after the automatic mode switching and the pilot reaction according to the output result of the mode decision unit, and performs real-time feedback and adjustment to ensure the continuous matching between the system automatic mode and the pilot behavior state.

Images