Pilot airworthiness state sensing method based on multi-task rotation pairing learning

Abstract

The invention discloses a pilot airworthiness state sensing method based on multi-task alternate pairing learning, and relates to the technical field of aviation safety guarantee and monitoring, and the method comprises the steps: constructing a multi-task-based alternate pairing learning model, and inputting a pre-obtained video data set into the multi-task-based alternate pairing learning model for training; the multi-task-based alternate pairing learning model comprises a shared feature extraction network and three special task sub-networks, and each special task sub-network comprises an rPPG signal estimator, a continuous emotion predictor and an airworthiness state sequence predictor; a multi-task rotation pairing learning method is adopted in the training stage; and inputting a to-be-predicted video into the trained multi-task-based alternate pairing learning model for prediction, and outputting to obtain a multi-task prediction result. According to the invention, potential relations among the physiological state, the psychological state and the airworthiness state of the pilot can be learned, and the state of the pilot in the driving process can be accurately evaluated.

Description

Technical Field

[0001] The present invention relates to the field of aviation safety assurance and monitoring technology, and in particular to a pilot airworthiness status perception method based on multi-task rotation pairing learning. Background Art

[0002] Remote photoplethysmography (rPPG) technology is a non-invasive method used to monitor heart rate and other blood flow dynamics by capturing tiny color changes on the skin surface through visual devices, thereby reflecting a person's physiological state. The advantage of this technology is that it does not require direct contact with the body and is suitable for use in situations where it is inconvenient to use traditional monitoring equipment, such as pilots during flight.

[0003] Emotional continuity recognition is an important research in the intersection of psychology and artificial intelligence. It focuses on the two main dimensions of emotions: arousal and valence. Arousal refers to the degree of emotional activation, while valence describes the positive or negative tendency of emotions. Compared with discrete emotion recognition, continuous emotion recognition can better capture the dynamic changes of emotions, reflecting the gradual change process of emotions in different intensity and time dimensions, thereby more accurately reflecting people's emotions.

[0004] In the aviation field, flight safety is of paramount importance. As the core subject of flight operations, the pilot's physical and mental state directly affects the execution of the flight mission. During the flight, the pilot needs to deal with complex flight environments, operate numerous flight instruments and execute precise flight instructions. Any slight change in state may have a significant impact on flight safety. For example, negative states such as fatigue, stress, and emotional fluctuations may cause the pilot's reaction speed to slow down and his judgment errors to increase, thus endangering flight safety. Therefore, accurate and real-time perception of the pilot's state is extremely critical to ensuring flight safety. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a pilot airworthiness status perception method based on multi-task rotation pairing learning to accurately evaluate the pilot's status during driving.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] The pilot airworthiness status perception method based on multi-task rotation pairing learning includes:

[0008] Constructing a multi-task based rotation pairing learning model, and inputting the pre-acquired video data set into the multi-task based rotation pairing learning model for training; the multi-task based rotation pairing learning model includes a shared feature extraction network and three dedicated task sub-networks, and the dedicated task sub-networks include: rPPG signal estimator, continuous emotion predictor, and airworthiness state sequence predictor;

[0009] The shared feature extraction network is used to extract multi-dimensional underlying features from the video and output the shared features to the three dedicated task sub-networks; the rPPG signal estimator is used to complete the regression task of rPPG signal prediction based on the shared features; the continuous emotion predictor is used to obtain the arousal and valence prediction values ​​of the emotions of the characters in the video based on the shared features; the airworthiness state sequence predictor is used to output a binary classification result of the video prediction based on the shared features, i.e., airworthy or unairworthy;

[0010] In the training phase, the rPPG signal estimator and the continuous emotion predictor are first jointly trained, then the rPPG signal estimator and the airworthiness state sequence predictor are jointly trained, and finally the continuous emotion predictor and the airworthiness state sequence predictor are jointly trained;

[0011] The video to be predicted is input into the trained multi-task based rotation pairing learning model for prediction, and the multi-task prediction result is output.

[0012] Furthermore, the shared feature extraction network is used to extract multi-dimensional underlying features from the video and output the shared features to the three dedicated task sub-networks, specifically including:

[0013] Face video The 3D convolutional neural network is used to initially extract the spatiotemporal features of the video, where T is the total number of frames, (W, H) is the width and height of each frame of the video, and C is the number of channels. For each time step t, the process of extracting the spatiotemporal features of the video is expressed as:

[0014] F t =Conv3D(V t )

[0015] where F t is the feature after 3D convolution, V t is the video input of the tth frame;

[0016] The feature F after 3D convolution t The video is input into the long short-term memory network LSTM to obtain the temporal features of the video. The LSTM outputs the shared features H through recursive updating. For each frame t, the update process is expressed as:

[0017] H t =LSTM(Ht-1 ,F t )

[0018] Among them, H t is the output shared feature of the tth frame, which depends on the hidden state H at the previous moment t-1 And the input feature F of the current frame t ;

[0019] Long short-term memory network LSTM outputs shared features for subsequent task prediction C′ is the number of channels of shared features.

[0020] Furthermore, the rPPG signal estimator is used to complete the regression task of rPPG signal prediction according to the shared features, specifically including:

[0021] The shared feature sequence H is input into a 3D convolutional layer for spatiotemporal feature extraction to obtain a spatiotemporal feature sequence R. For each frame t, the spatiotemporal feature extraction process is expressed as:

[0022] R t =Conv3D(H t )

[0023] Among them, R t is the output spatiotemporal feature of the t-th frame;

[0024] Flatten the spatiotemporal feature sequence R after the convolution layer into a one-dimensional vector R flatten ;

[0025] The flattened feature R flatten Input into a fully connected layer, the output dimension of the fully connected layer is T, and the final rPPG signal is obtained, which is expressed as:

[0026]

[0027] in, is the extracted rPPG signal.

[0028] Furthermore, the continuous emotion predictor is used to obtain the predicted values ​​of arousal and valence of the emotions of the characters in the video according to the shared features, specifically including:

[0029] The pre-trained EfficientNet is used as the backbone network as the static expression extractor to extract the static expression features of the face from the shared features. For each frame t, the extraction process is expressed as:

[0030] S t =EfficientNet(H t )

[0031] Then the static expression feature vector St Input into TransformerBlock to extract the timing information between expressions and obtain dynamic expression features:

[0032] D t = TransformerBlock(S t )

[0033] Then the dynamic expression feature sequence D is passed through a fully connected layer to obtain the arousal degree and potency Two consecutive values:

[0034]

[0035] Furthermore, the airworthiness state sequence predictor is used to predict and output a binary classification result of the video according to the shared features, namely, airworthy or unairworthy, specifically including:

[0036] The first fully connected layer is used to map the shared features to a D-dimensional feature space, and the second fully connected layer is used to map the video features to a one-dimensional space, and the Sigmoid activation function is used to convert the output values Restricted to the range [0,1], it is expressed as:

[0037]

[0038] If the output value If it is greater than or equal to 0.5, the airworthiness status output is airworthy, otherwise it is unairworthy.

[0039] Furthermore, the loss function of the rPPG signal estimator adopts the mean square error MSE loss function, that is:

[0040]

[0041] Where N is the number of samples, is the true rPPG signal of the ith sample, The estimated rPPG signal for the ith sample.

[0042] Furthermore, the loss function of the continuous emotion predictor adopts the mean square error MSE loss function, that is:

[0043]

[0044] Where N is the number of samples, are the true arousal and valence labels of the i-th sample respectively.

[0045] Furthermore, the loss function of the airworthiness state sequence predictor adopts a binary cross entropy BCE loss function, namely:

[0046]

[0047] Where N is the number of samples, is the true airworthiness label of the i-th sample.

[0048] Furthermore, the training phase first jointly trains the rPPG signal estimator and the continuous emotion predictor, then jointly trains the rPPG signal estimator and the airworthiness state sequence predictor, and finally jointly trains the continuous emotion predictor and the airworthiness state sequence predictor. The specific process includes:

[0049] The first total loss function is calculated based on the loss functions of the rPPG signal estimator and the continuous emotion predictor The rPPG signal estimator and the continuous emotion predictor are jointly trained using the first total loss function;

[0050] The second total loss function is calculated based on the loss functions of the rPPG signal estimator and the airworthiness state sequence predictor. The rPPG signal estimator and the airworthiness state sequence predictor are jointly trained using the second total loss function;

[0051] The third total loss function is calculated based on the loss functions of the continuous emotion predictor and the airworthiness state sequence predictor. The continuous emotion predictor and the airworthiness status sequence predictor are jointly trained using the third total loss function.

[0052] The beneficial effects of the present invention are:

[0053] The pilot airworthiness status perception method based on multi-task rotation pairing learning provided by the present invention extracts rPPG signals and expression signals from the pilot's face video respectively, and predicts the pilot's airworthiness status at the same time. The present invention adopts the multi-task rotation pairing learning method for model training, which can better learn the potential connection between the pilot's physiological state, psychological state and the pilot's airworthiness status (whether it is conducive to flying), so as to more accurately evaluate the pilot's state during the driving process, and then achieve effective protection of flight safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The network architecture diagram of the pilot airworthiness status perception method based on multi-task rotation pairing learning;

[0055] Figure 2 This is a diagram of the shared feature extractor network architecture;

[0056] Figure 3 This is the network architecture diagram of the rPPG signal estimator;

[0057] Figure 4This is the network architecture diagram of the continuous emotion predictor;

[0058] Figure 5 This is the network architecture diagram of the airworthiness status predictor. DETAILED DESCRIPTION

[0059] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] See also Figure 1-Figure 5 , the present invention provides a technical solution:

[0061] The pilot airworthiness status perception method based on multi-task rotation pairing learning includes:

[0062] Construct a multi-task based rotation pairing learning model, and input the pre-acquired video data set into the multi-task based rotation pairing learning model for training; the overall architecture of the multi-task based rotation pairing learning model is as follows Figure 1 As shown, it includes a shared feature extraction network and three dedicated task sub-networks, and the dedicated task sub-networks include: rPPG signal estimator, continuous emotion predictor, and airworthiness state sequence predictor;

[0063] The shared feature extraction network is used to extract multi-dimensional underlying features from the video and output the shared features to the three dedicated task sub-networks; the rPPG signal estimator is used to complete the regression task of rPPG signal prediction based on the shared features; the continuous emotion predictor is used to obtain the arousal and valence prediction values ​​of the emotions of the characters in the video based on the shared features; the airworthiness state sequence predictor is used to output a binary classification result of the video prediction based on the shared features, i.e., airworthy or unairworthy;

[0064] The three dedicated task subnetworks are respectively responsible for completing the prediction of the physiological state task, the psychological state task and the airworthiness state task. By constructing a multi-task learning (MTL) model, the present invention can simultaneously predict the pilot's physiological state (rPPG signal), psychological state (continuous emotional changes), and the pilot's airworthiness state (determining whether the current pilot's state is conducive to flying), thereby improving flight safety.

[0065] In the training phase, the rPPG signal estimator and the continuous emotion predictor are first jointly trained, then the rPPG signal estimator and the airworthiness state sequence predictor are jointly trained, and finally the continuous emotion predictor and the airworthiness state sequence predictor are jointly trained;

[0066] The video to be predicted is input into the trained multi-task based rotation pairing learning model for prediction, and the multi-task prediction result is output.

[0067] In a specific embodiment, the shared feature extractor network architecture is shown in FIG. Figure 2 As shown, the shared feature extraction network is used to extract multi-dimensional underlying features from the video and output the shared features to the three dedicated task sub-networks, specifically including:

[0068] Face video The 3D convolutional neural network is used to initially extract the spatiotemporal features of the video, where T is the total number of frames, (W, H) is the width and height of each frame of the video, and C is the number of channels. For each time step t, the process of extracting the spatiotemporal features of the video is expressed as:

[0069] F t =Conv3D(V t )

[0070] where F t is the feature after 3D convolution, V t is the video input of the tth frame;

[0071] The feature F after 3D convolution t The video is input into the long short-term memory network LSTM to obtain the temporal features of the video. The LSTM outputs the shared features H through recursive updating. For each frame t, the update process is expressed as:

[0072] H t =LSTM(H t-1 ,F t )

[0073] Among them, H t is the output shared feature of the tth frame, which depends on the hidden state H at the previous moment t-1 And the input feature F of the current frame t ;

[0074] Long short-term memory network LSTM outputs shared features for subsequent task prediction C′ is the number of channels of shared features.

[0075] Although the three tasks of physiological state, psychological state and airworthiness state have different goals and outputs, they share some features. The shared feature extraction network extracts multi-dimensional underlying features from the video, which can be used by the three subsequent task sub-networks, allowing network parameters to be shared between different tasks, improving model efficiency and generalization ability.

[0076] In a specific embodiment, the rPPG signal estimator network architecture is shown in FIG. Figure 3 As shown, the rPPG signal estimator is used to complete the regression task of rPPG signal prediction according to the shared features, specifically including:

[0077] The shared feature sequence H is input into a 3D convolutional layer for spatiotemporal feature extraction to obtain a spatiotemporal feature sequence R. For each frame t, the spatiotemporal feature extraction process is expressed as:

[0078] R t =Conv3D(H t )

[0079] Among them, R t is the output spatiotemporal feature of the t-th frame;

[0080] Flatten the spatiotemporal feature sequence R after the convolution layer into a one-dimensional vector R flatten ;

[0081] The flattened feature R flatten Input into a fully connected layer, the output dimension of the fully connected layer is T, and the final rPPG signal is obtained, which is expressed as:

[0082]

[0083] in, is the extracted rPPG signal.

[0084] For the extraction of rPPG signals, the present invention uses a 3D convolutional neural network to further capture the spatial and temporal features of the shared features. In order to predict the low-dimensional rPPG signal from the high-dimensional temporal features, the present invention uses a fully connected layer to integrate the high-dimensional temporal features and perform nonlinear mapping, thereby completing the regression task of rPPG signal prediction.

[0085] In a specific embodiment, the continuous emotion predictor network architecture is shown in FIG. Figure 4 As shown, a static-dynamic emotion recognition system (SDER) is used to implement it, including a static expression extractor and a dynamic expression extractor.

[0086] The continuous emotion predictor is used to obtain the predicted values ​​of arousal and valence of the emotions of the characters in the video according to the shared features, specifically including:

[0087] The pre-trained EfficientNet is used as the backbone network as the static expression extractor to extract the static expression features of the face from the shared features. For each frame t, the extraction process is expressed as:

[0088] S t =EfficientNet(H t )

[0089] Then the static expression feature vector S t Input into TransformerBlock to extract the timing information between expressions and obtain dynamic expression features:

[0090] D t = TransformerBlock(S t )

[0091] Then the dynamic expression feature sequence D is passed through a fully connected layer to obtain the arousal degree and potency Two consecutive values:

[0092]

[0093] The static-dynamic emotion system is used to extract the static expression features and the temporal information of the features of the video sequence, which can increase the prediction accuracy of continuous emotions.

[0094] In a specific embodiment, the network architecture of the airworthiness state sequence predictor is as shown in FIG. Figure 5 As shown, the airworthiness state sequence predictor is used to predict the video according to the shared features and output a binary classification result, that is, airworthy or unairworthy, specifically including:

[0095] The first fully connected layer is used to map the shared features to a D-dimensional feature space, and the second fully connected layer is used to map the video features to a one-dimensional space, and the Sigmoid activation function is used to convert the output values Restricted to the range [0,1], it is expressed as:

[0096]

[0097] If the output value If it is greater than or equal to 0.5, the airworthiness status output is airworthy, otherwise it is unairworthy.

[0098] For the prediction of pilot's airworthiness status, the present invention regards it as a binary classification task, and the output is whether the pilot's status is conducive to the current flight. After the shared features are processed by layers of convolution, the output contains information about the pilot's behavior, physiological signals, emotions, etc., which can provide rich features for the binary classification task.

[0099] Furthermore, the loss function of the rPPG signal estimator adopts the mean square error MSE loss function, that is:

[0100]

[0101] Where N is the number of samples, is the true rPPG signal of the ith sample, The estimated rPPG signal for the ith sample.

[0102] Furthermore, the loss function of the continuous emotion predictor adopts the mean square error MSE loss function, that is:

[0103]

[0104] Where N is the number of samples, are the true arousal and valence labels of the i-th sample respectively.

[0105] Furthermore, the loss function of the airworthiness state sequence predictor adopts a binary cross entropy BCE loss function, namely:

[0106]

[0107] Where N is the number of samples, is the true airworthiness label of the i-th sample.

[0108] Furthermore, the training phase first jointly trains the rPPG signal estimator and the continuous emotion predictor, then jointly trains the rPPG signal estimator and the airworthiness state sequence predictor, and finally jointly trains the continuous emotion predictor and the airworthiness state sequence predictor. The specific process includes:

[0109] The first total loss function is calculated based on the loss functions of the rPPG signal estimator and the continuous emotion predictor The rPPG signal estimator and the continuous emotion predictor are jointly trained using the first total loss function;

[0110] The second total loss function is calculated based on the loss functions of the rPPG signal estimator and the airworthiness state sequence predictor. The rPPG signal estimator and the airworthiness state sequence predictor are jointly trained using the second total loss function;

[0111] The third total loss function is calculated based on the loss functions of the continuous emotion predictor and the airworthiness state sequence predictor. The continuous emotion predictor and the airworthiness status sequence predictor are jointly trained using the third total loss function.

[0112] Multi-task learning (MTL) aims to optimize three tasks: rPPG signal estimation, continuous emotion recognition, and pilot airworthiness status prediction. Unlike traditional MTL, which trains by adding the loss functions of the three tasks simultaneously, this paper innovatively proposes a multi-task rotation pairing learning method, which calculates the loss functions of the three tasks in pairs in the form of permutations and combinations. This training strategy aims to gradually explore and strengthen the shared features and potential interactions between tasks through interrelated task pairs.

[0113] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A pilot airworthiness status perception method based on multi-task rotation pairing learning, characterized in that: The method comprises: Constructing a multi-task based rotation pairing learning model, and inputting the pre-acquired video data set into the multi-task based rotation pairing learning model for training; the multi-task based rotation pairing learning model includes a shared feature extraction network and three dedicated task sub-networks, and the dedicated task sub-networks include: rPPG signal estimator, continuous emotion predictor, and airworthiness state sequence predictor; The shared feature extraction network is used to extract multi-dimensional underlying features from the video and output the shared features to the three dedicated task sub-networks; the rPPG signal estimator is used to complete the regression task of rPPG signal prediction based on the shared features; the continuous emotion predictor is used to obtain the arousal and valence prediction values ​​of the emotions of the characters in the video based on the shared features; the airworthiness state sequence predictor is used to output a binary classification result of the video prediction based on the shared features, i.e., airworthy or unairworthy; In the training phase, the rPPG signal estimator and the continuous emotion predictor are first jointly trained, then the rPPG signal estimator and the airworthiness state sequence predictor are jointly trained, and finally the continuous emotion predictor and the airworthiness state sequence predictor are jointly trained; The video to be predicted is input into the trained multi-task based rotation pairing learning model for prediction, and the multi-task prediction result is output.

2. The pilot airworthiness status perception method based on multi-task rotation pairing learning according to claim 1 is characterized by: The shared feature extraction network is used to extract multi-dimensional underlying features from the video and output the shared features to the three dedicated task sub-networks, specifically including: Face video The 3D convolutional neural network is input to initially extract the spatiotemporal features of the video, where T is the total number of frames, (W, H) is the width and height of each frame of the video, and C is the number of channels; for each time step t, the process of extracting the spatiotemporal features of the video is expressed as: F t =Conv3D(V t ) where F t is the feature after 3D convolution, V t is the video input of the tth frame; The feature F after 3D convolution t The video is input into the long short-term memory network LSTM to obtain the temporal features of the video. The LSTM outputs the shared features H through recursive updating. For each frame t, the update process is expressed as: H t =LSTM(H t-1 ,F t ) Among them, H t is the output shared feature of the tth frame, which depends on the hidden state H at the previous moment t-1 And the input feature F of the current frame t ; Long short-term memory network LSTM outputs shared features for subsequent task prediction C′ is the number of channels of shared features.

3. The pilot airworthiness status perception method based on multi-task rotation pairing learning according to claim 2 is characterized by: The rPPG signal estimator is used to complete the regression task of rPPG signal prediction according to the shared features, specifically including: The shared feature sequence H is input into a 3D convolutional layer for spatiotemporal feature extraction to obtain a spatiotemporal feature sequence R. For each frame t, the spatiotemporal feature extraction process is expressed as: R t =Conv3D(H t ) Among them, R t is the output spatiotemporal feature of the t-th frame; Flatten the spatiotemporal feature sequence R after the convolution layer into a one-dimensional vector R flatten ; The flattened feature R flatten Input into a fully connected layer, the output dimension of the fully connected layer is T, and the final rPPG signal is obtained, which is expressed as: in, is the extracted rPPG signal.

4. The pilot airworthiness status perception method based on multi-task rotation pairing learning according to claim 2 is characterized by: The continuous emotion predictor is used to obtain the predicted values ​​of arousal and valence of the emotions of the characters in the video according to the shared features, specifically including: The pre-trained EfficientNet is used as the backbone network as the static expression extractor to extract the static expression features of the face from the shared features. For each frame t, the extraction process is expressed as: S t =EfficientNet(H t ) Then the static expression feature vector S t Input into TransformerBlock to extract the timing information between expressions and obtain dynamic expression features: D t =TransformerBlock(S t ) Then the dynamic expression feature sequence D is passed through a fully connected layer to obtain the arousal degree and potency Two consecutive values:

5. The pilot airworthiness status perception method based on multi-task rotation pairing learning according to claim 2 is characterized by: The airworthiness state sequence predictor is used to predict and output a binary classification result of the video according to the shared features, namely, airworthy or unairworthy, specifically including: The first fully connected layer is used to map the shared features to a D-dimensional feature space, and the second fully connected layer is used to map the video features to a one-dimensional space, and the Sigmoid activation function is used to convert the output values Restricted to the range [0,1], it is expressed as: If the output value If it is greater than or equal to 0.5, the airworthiness status output is airworthy, otherwise it is unairworthy.

6. The pilot airworthiness status perception method based on multi-task rotation pairing learning according to claim 3 is characterized by: The loss function of the rPPG signal estimator adopts the mean square error MSE loss function, that is: Where N is the number of samples, is the true rPPG signal of the ith sample, The estimated rPPG signal for the ith sample.

7. The pilot airworthiness status perception method based on multi-task rotation pairing learning according to claim 4 is characterized by: The loss function of the continuous emotion predictor adopts the mean square error MSE loss function, that is: Where N is the number of samples, are the true arousal and valence labels of the i-th sample respectively.

8. The pilot airworthiness status perception method based on multi-task rotation pairing learning according to claim 5 is characterized by: The loss function of the airworthiness state sequence predictor adopts the binary cross entropy BCE loss function, that is: Where N is the number of samples, is the true airworthiness label of the i-th sample.

9. The pilot airworthiness status perception method based on multi-task rotation pairing learning according to claim 1, characterized in that: The training phase first jointly trains the rPPG signal estimator and the continuous emotion predictor, then jointly trains the rPPG signal estimator and the airworthiness state sequence predictor, and finally jointly trains the continuous emotion predictor and the airworthiness state sequence predictor. The specific process includes: The first total loss function is calculated based on the loss functions of the rPPG signal estimator and the continuous emotion predictor The rPPG signal estimator and the continuous emotion predictor are jointly trained using the first total loss function; The second total loss function is calculated based on the loss functions of the rPPG signal estimator and the airworthiness state sequence predictor. The rPPG signal estimator and the airworthiness state sequence predictor are jointly trained using the second total loss function; The third total loss function is calculated based on the loss functions of the continuous emotion predictor and the airworthiness state sequence predictor. The continuous emotion predictor and the airworthiness status sequence predictor are jointly trained using the third total loss function.

Images